As the AI hype-cycle has built, we've been treated to a plethora of claims about what sorts of improvements and breakthroughs the technology can deliver. One of the most fundamental -- and potentially important -- has been the idea that we can use AI to find new medicines and treatments for existing conditions where current options have come up short.
That promise has itself now come up short. IBM has announced that it will stop selling its Watson AI system as a tool for drug discovery. It's a high-profile retreat for the company, which has aggressively marketed AI as being useful for these purposes and which ran into problems last year when reports indicated its systems had made improper, dangerous recommendations for cancer patients (the system's recommendations were never put into effect).
While IBM cites sluggish sales as a reason for its withdrawal, deeper problems are potentially responsible. A recent deep dive(Opens in a new window) by IEEE Spectrum puts context around these issues. The upshot: After years of work and a number of moonshot projects, IBM has remarkably little to show for its efforts. And the company has created a certain amount of ill will towards itself, IEEE writes, because it took an aggressive, marketing-first approach to AI and Watson, promising grandiose achievements that didn't accurately portray what the system could actually reliably achieve.
Watson wowed the world with its performance on Jeopardy and an ability to analyze the relationships between words rather than treating them like search terms. In theory, Watson could use its engine to sort through reams of medical data in a similar fashion, finding the hidden signal within a system stuffed with noise. Reality has not cooperated. Of the small amount of research conducted on using AI to improve patient outcomes, none of it has involved IBM's Watson.
The IEEE piece takes pains to note that IBM faced huge challenges in attempting to bring its AI program online and use it effectively for human medicine. Nothing like Watson (or what Watson was intended to be) has ever existed before. No one knew how to build it. Yoshua Bengio, a leading AI researcher at the University of Montreal, summarized the efforts to help AI understand medical texts and terminology thusly: "We’re doing incredibly better with NLP than we were five years ago, yet we’re still incredibly worse than humans."A Vexing Problem
Watson's problem wasn't that it didn't work. The problem is, Watson doesn't do the right stuff. While it quickly learned to ingest and process vast quantities of data, it had a great deal of trouble identifying the bits of information within a study that might lead doctors to actually change their process of care. This is particularly true if the relevant information was incidental to the main point of the research.
Image by
Because patient data wasn't always properly formatted or even chronologically arranged, the software had trouble understanding patient histories. And the system was incapable of comparing new cancer patients against databases of previous patients to discover hidden treatment patterns, because such practices would not be considered evidence-based. Making a strong recommendation from evidence-based medicine requires double-blind studies, meta-analyses, and systemic evidence reviews, not an AI system claiming to have found a similarity between different types of patients.
It's not clear what's next for Watson, if anything. The tool has had some success in narrow, tailored applications with less ambiguity. But despite dozens of planned initiatives, oceans of hype, and a great deal of investment, IBM's Watson for Drug Discovery has clearly missed its own goals.
Update (4/22/019): IBM contests multiple aspects of this story. A spokesperson told us that the company is not discontinuing Watson for Drug Discovery, and that it is instead "focusing our resources within Watson Health to double down on the adjacent field of clinical development where we see an even greater market need for our data and AI capabilities."It's not a discontinuation. It's just a completely different focus in adjacent markets where IBM thinks it can earn more money.
Next, IBM disputes allegations that Watson Health has little to show for its efforts. It offers no particular proof for this claim, beyond noting (completely truthfully) that treating cancer is exceptionally difficult and medical progress is slow. This is a point we entirely agree with.
The question in play is not whether IBM is doing something worthwhile or important by focusing on cancer research, but whether or not its products are producing effective results. The consistent narrative from the companies and organizations that have used these products, thus far, is "No," or at the very least, "Not at the level of skill and capability the marketing department promised." IBM's largest successes, according to IEEE Spectrum, have come from the specific application of AI to narrow, well-understood problems.
Finally, IBM notes that in a follow-up cancer treatment recommendation survey, its accuracy rate rose from 73 percent in 2017(Opens in a new window) to 93 percent according to a January 2018 survey(Opens in a new window). It is not clear precisely what drove this improvement or whether the gains are attributable to an improvement in Watson's abilities or if other aspects of the evaluation process were changed to produce a superior result. The second test only focused on non-concordant cases from the first test rather than retesting the entire data set.
Now Read:Tagged In
Medical AI Biomedical Engineering Future Health Ibm AiMore from Extreme
