Is Abridge's tech a glimpse into the future of healthcare?

By Fred Vogelstein

When Shiv Rao was a cardiologist and investor in Pittsburgh, he saw a patient with a 10-year-history of breast cancer who seemed particularly tense during the appointment. “At the end of the encounter I asked her if there was something I did or said to make her so anxious. She told me that since her initial diagnosis of breast cancer her husband had come to every single encounter … and took notes. He couldn’t be there that day. She told me that him taking notes meant she could feel liberated to be present in the conversation knowing that they could go home and research all of his notes, Google them, and learn about all the different medical terminology that was discussed.”

Rao tells this story and others like it as a way of explaining why in 2018 he started Abridge. He wanted to use AI transcription software with a medical vocabulary to automate patient note taking during doctor appointments. Patients would use the app to record their visit, and it would automatically generate a transcript.

Today that simple quest has grown into one of the most talked about AI startups out there. In the past two years it’s replaced its patient app with a much more robust enterprise application for doctors. It’s raised $180 million, including a $150 million round at the beginning of this year. And it’s begun winning head to head competitions against even the most established companies in the space.  There are few companies that have benefited as handsomely from the AI frenzy as Abridge. 

It’s also why I was unsure what I'd discover when I reached out to Abridge recently to hear about their progress. Sure, its latest funding round valued the company at $850 million. Yes, it has a pitch that’s easy to grasp: Use AI transcription to speed up doctors' workflow. And, sure  it’s operating in the industry, healthcare, where AI can actually provide tangible, measurable benefits right away. 

But we’re also in the middle of a technology frenzy in AI not seen since the Great Internet Bubble of 1999. Anyone who's used ChatGPT can tell you that AI is going to change the world. But ask them how it’s going to do that and you mostly get a slew of unspecific superlatives filled with a mixture of hope and fear. Exhibit A: OpenAI, makers of ChatGPT. It’s losing about $5 billion a year. It’s become a drag on its biggest investor’s - Microsoft’s - earnings. Yet it just raised nearly $7 billion, valuing it at $157 billion. That ranks it about 125th in market cap, up there with Pfizer, Unilever and AT&T. Maybe OpenAI will grow into that valuation. But it’s hard to see that happening soon. 

I saw Rao demonstrate Abridge six months ago at a Stat conference and was impressed. But many companies now do AI transcription. Most hospitals already have versions of this technology. And for solo-practitioners the quality of off-the-shelf AI transcription has become both inexpensive and excellent.  

So was Abridge an AI success story or an AI bubble story? I asked myself. 

It sure seems like it’s on the right track for now. I spent nearly three hours talking to Zack Lipton, the executive in charge of building its technology as well as investors, advisors, doctors and critics. It’s a convincing vision with evidence to suggest not only a company that’s going to be around for a while, but one that could one day become a critical operating system of American healthcare.   

Why? Because healthcare is notorious for its silos of unstructured data - text-based patient summaries - that are difficult, even when anonymized, to analyze. But those silos contain a treasure trove of valuable clinical information. Abridge’s technology seems good enough to one day unlock access to it. Doctors now have to wait for formal studies or conversations at medical conferences to access information like this.

Sure, it’s anecdotal, not scientific data. But accessed and analyzed in a secure, anonymized way, real time across thousands of doctors nationwide,  it would still be super useful in the right hands. For example, picking the best medicine to treat epilepsy is often a guessing game. I’ve experienced this as a parent. But if a doctor could know in real time that 500 12-year-olds with similar brain wave tests responded better to one drug versus another, that would be an incredible advance. 

Lipton puts it this way: “Can we in the long run turn this (information) back towards doctors in the form of insights? That could help them not just practice medicine more efficiently, but actually better than they ever could have dreamed of.”.

Abridge has a long way to go before that happens. For now it’s aimed at outpatient doctors, many of whom do their charts and other paperwork at the end of their day instead of real time. But among that cohort, many report Abridge saves them five to 10 minutes per patient. Meanwhile, it’s won the backing of its biggest competitor’s long time CEO. Paul Ricci, who built and ran Nuance for nearly 20 years before he retired in 2018, is now an advisor to Lightspeed Ventures, which led Abridge’s $150 million funding round early this year. 

Just in the past year Abridge has won contracts with more than 60 hospital systems nationwide giving it access to tens of thousands of physicians. It operates very much like any smartphone recording app. But Lipton says some customers report penetration of roughly 50 percent among their primary care physicians.

Its biggest win was just two months ago: Kaiser Permanente, the nation's largest HMO, agreed to use Abridge throughout its network of 40 hospitals and 600 medical offices throughout the U.S. That alone gives Abridge access to nearly 25,000 physicians. “I have very paternal feelings about Nuance, so I’m reluctant to be (critical of them). But I think Abridge is outracing them right now,” Ricci told me. At the Stat conference I attended back in May, I saw one doctor during Q&A say, “Can I have that software now please?” 

Abridge has also convinced Epic, which controls the market for electronic medical records, to connect it into its systems. That's almost a requirement for any healthcare startup today. And Abridge is leveraging that relationship. Its technology is good enough that it can increasingly take a doctor/patient transcript and populate the correct fields in the patient’s EMR for doctor review. Turning unstructured data into structured data has been one of the Holy Grails of technologists everywhere.  

And Abridge is winning those contracts despite being a relative newcomer in the market. Its primary competitor, Nuance's DAX, has endless resources because Microsoft has owned it since 2021. It also has the best name recognition. Nuance’s Dragon software has been in the voice-to-text business for decades.  But as Rao will happily report, the roughly dozen times they’ve gone up against Nuance in the past year, they’ve won. 

"What makes them different and the reason I became an advisor to the company is that they are way more than a scribe," said Eric Topol, who as both a cardiologist and director/founder of Scripps Research Translational Institute in San Diego, has become one of the best known experts on the digitization of medicine. He said AI transcription has gotten so good in the past few years that any off the shelf offering can now record a patient interview, create a transcript and turn that into a chart note. What's hard, he says, is coming up with bespoke technology, like Abridge has, that actually saves doctors time and is well designed enough that they want to use it. "Their CTO Zach Lipton is one of the top AI minds in the country. It's because of him and that team that they are where they are."

Lipton is the first to acknowledge that part of Abridge’s success is also good timing. The AI innovations that enabled ChatGPT have helped create enormous leaps in computers’ abilities to  transcribe even multi-party conversations in two languages. And it can now  interpret those transcripts with minimal errors. Lipton said just off the shelf open source LLMs can transcribe and understand language in ways that few thought were possible even five years ago.

Meanwhile, he said the explosion in telehealth visits that grew out of the pandemic, got patients used to having their doctor visits recorded. He said the AI bubble has even helped. “I think with things like ChatGPT, it’s raised the level of awareness so that Abridge didn't sound like science fiction to business people at hospital systems,” Lipton said. 

And the demand from doctors for any solution like Abridge that will cut their paperwork burden is as high as it’s ever been. Their administrative load exacerbated by the pandemic has created doctor burnout that is now widespread enough to be among the most critical issues in American medicine. 

But good timing is irrelevant without good technology. Lipton is careful to explain that Abridge still requires doctors to review Abridge's drafts before submitting them. He said he and Rao don’t ever see Abridge allowing its system to play doctor. But they believe it doesn’t have to play doctor to be transformative for doctors. It just needs to be easy enough for doctors to use the first time without instructions, and to generate transcriptions accurate enough that doctors rarely have to edit its mistakes. Part of the reason systems like these have failed in the past is all the time saved using the technology was offset by the time doctors had to spend figuring out how to use it or correcting all the errors the system introduced into their records.

Off the shelf programs can do this in controlled settings using everyday English. It’s much tougher, Lipton says, to do it using up-to-date medical terminology that includes names of the latest medicines and therapies, in 14 - soon to be 28 - languages, and in suboptimal recording conditions.   Using Abridge, a doctor who speaks Chinese to his patients or one who is speaking English using the patient’s son as a translator, gets the same English transcription back. 

From there Abridge's system takes that transcription and instantly creates a physician's note for the chart, an explanation of the visit in layman’s terminology for the patient. If the conversation referred to specific ailments or diseases, it will attach the correct billing codes for hospital revenue administrators and insurance companies. If tests and/or medicines are discussed, it will draft orders for those as well. If data needs to be entered into specific fields in the patient’s electronic medical record, it will do that too. 

And the transcript follows the medical record everywhere. So when a doctor is reviewing his chart note at the end of the day he can highlight and immediately view where in the transcript the system pulled that information. He can listen to that portion of the recording at the same time. Lipton created a star system for doctors to provide feedback on Abridge’s transcriptions, another system for them to provide text feedback, and a third that analyzes what transcription errors doctors correct in every chart. 

Lipton has been thinking about how machine learning and AI might impact healthcare for more than a decade. It was the topic of his PhD thesis. He majored in math at Columbia University and  became a semi-successful jazz saxophonist after graduating. But in the middle of that he developed an autoimmune issue that sidelined him for a year.  With too much time to think  “I couldn’t help seeing how impoverished the epistemic state of medicine was and …. wanted some agency,” he said.

That led to stints as part of Amazon’s early AI efforts and at Microsoft Research, “building recommender systems for Amazon and reinforcement learning and dialogue systems for Microsoft (as far back as 2014) before anyone thought AI was the hottest thing in the world,” he said. He turned that into an offer to be an AI/machine learning  professor at Carnegie Mellon, where many believe the AI revolution started. For example, CMU is considered the birthplace of the self-driving car. 

He met Rao on his house hunting trip. “The person sitting next to me on the plane turns out to be (Steve Shapiro) the (then) chief of medicine at UPMC (University of Pittsburgh Medical Center).  I tell him I'm a new professor coming to Carnegie Mellon. I work in deep learning. I’m passionate about healthcare. And it's like, perfect timing.That's right at the moment (in 2017) when everyone's like, ‘Oh, this AI thing is real.’ He ends up giving me a ride home. The next day he emails asking if I can do happy hour tonight. So he rolls into the happy hour with the CTO of UPMC, the CFO of UPMC, and the executive president of UPMC Enterprises (Shiv Rao.)

Lipton eventually signed on with Abridge part time - as a founding advisor -  not wanting to resign from Carnegie Mellon before even starting. But he, Rao and the university found a way to get Abridge access to Lipton’s research labs. He joined Abridge full time as chief science and technical officer in early 2023.

He knows he’s unlikely to convince every doctor to use Abridge - maybe not even a majority of them. Every doctor works differently. Emergency room doctors I know who dictate a chart note in three minutes after seeing each patient, might not find Abridge as useful as an outpatient doctor who only has time to do his paperwork at the end of each day. Lipton also said it’s hard to imagine the ICU, or surgeons or radiologists using Abridge, for example. He said Abridge is currently working to convince emergency rooms and nurses to use it more. 

And some doctors might resist participating in any technology that exposes their actual patient conversation to scrutiny. Doctors are used to documenting everything about their interactions with patients. But practicing medicine in a world where hospital administrators can access a verbatim patient-redacted transcript of every doctor visit might feel intrusive to some practitioners. I know I’ve had conversations with doctors where the response I get to a question begins “Officially I can’t recommend that. Unofficially, I’ve seen a lot of anecdotal evidence that suggests it works well.” Will doctors want to use a system where they have to remember to turn off the recorder to speak candidly to their patients like this? 

Technology also has a way of creating unanticipated problems in hospital settings. I know of one hospital that uses technology to perform rapid medical evaluations to reduce overcrowding in its ER waiting room. But it’s come at a cost, according to Eric Snoey, an ER physician at Alameda Health System-Highland Hospital in Oakland. He’s worried it’s causing sloppy doctoring and is throwing gasoline on the problem of over-testing.

And who knows? Maybe long term AI does get so good that even companies like Abridge can’t stay far enough ahead of it to run a business.  Fawad Butt, a former top healthcare data executive at Kaiser and United Healthcare who just launched his own healthcare platform PenguinAI doesn’t think Lipton, or anyone can do it. “Do you remember (more than two decades ago)  when we thought email was going to be the killer app? It was, but it wasn’t the thing that sustained any business growth. That's exactly how I look at Abridge. While I'm enamored by their ability to raise a ton of money, I'm not impressed by either their technology or the business. I wouldn’t be surprised if there aren't many other players that will come in and … be better than these guys, especially if the environment is correct.” 

Lipton as you might expect doesn’t see it that way: “Yes we're living in an era where anyone can make something that kind of works if you're not looking too close. But there's a big difference between that and getting a doctor 95 percent, maybe 98 percent of the way to being done with their (paperwork) –  so that they have a complete, comprehensive billable note that … has all the problems described accurately … harmonizes with the billing ontology … and that … won’t be denied (by insurance companies) on account of insufficient documentation. ”

He said that part of Abridge’s secret is that they built their own system rather than mostly relying on what’s already out there.  Sure it’s leveraged off the scads of already available open source software, databases and pre-trained AI models.  But he said his system is customized enough now that “our speech costs are maybe like a tenth what they would be if we were relying on Google or AWS for speech recognition.”

“And it's not trivial work,“ he said, “because when you ramp up the size of the data set … you ramp up the size of the model to where it doesn’t work on a single GPU (graphics processing chip)  but needs to be shared across many different GPU's on a machine, or sometimes even across multiple machines, to get that all to work.”

Sure he has to keep racing to outpace the rising floor of what’s now considered commodity technology. But he says “There's a tremendous amount of sophistication required to run an organization that is serving (tens of thousands of doctors).  There's also an increasing vigilance on the part of hospital systems to get serious about how they're handling responsible deployment of AI systems. So who's the company that's in a position to provide a proper academic account of the quality of our systems …not just what the doctors see, but what the health system project teams, its AI governance staff, and its QA folks on the implementations team in the hospital see?

“With us they're able to see the dashboards, to monitor node quality, to perform their own QA to make sure that the system is performing as they expect, to monitor usage,  to handle all of the account provisioning, and then meet the really dense regulatory requirements about shipping software and healthcare enterprises. It’s going to be pretty hard for someone else to just go build a prototype.”