Editor’s note: This is part two of our interview with Craig Kwiatkowski. To read part one, click here.
Cedars-Sinai, California’s prominent health system, has various artificial intelligence programs deployed or in operation. It is at the forefront of healthcare AI.
Craig Kwiatkowski is chief information officer at Cedars-Sinai, leading teams building AI designed to improve care and help patients and providers.
In today’s interview, we talk with Kwiatkowski, who has a doctorate in pharmacy, about some of the AI tools being used in the health system. He describes how he measures the success of AI-enabled initiatives and how AI can help advance health equity. Specifically, he shows how Cedars Sinai Connect, an AI-powered primary care app, is addressing AI biases and training on data sets that reflect population groups. diverse population.
Q. What AI tools are you using or implementing at Cedars Sinai that seem particularly promising?
ONE. Our focus on tools is really looking at tools that can help reduce friction, improve efficiency and simplify things, frankly to help caregivers, clinicians our clinics and patients. There is no shortage of opportunities in the general AI category.
One thing I’m very excited about is ambient documentation, sometimes called ambient scribes or virtual scribes. The technology looks very promising and I feel quite optimistic about it. We have been testing these tools for some time. The response so far has been solid.
Many physicians find ambient tools help reduce the cognitive load and management of note writing, and we’re starting to see that as we put these tools out there. We also realize that this doesn’t always save time in all cases but it does make things easier for them and combats the burnout factor. And it allows them to focus more on the patient and less on the computer, which is obviously important.
One of the doctors I spoke with described Ambient as a really good medical student. It doesn’t make things perfect, but it’s pretty good and helps them almost have a scribe by their side, so to speak.
But we’re also starting to appreciate that it’s not for everyone. Some doctors have very efficient workflows using current tools, templates, phrases, and a lot of muscle memory to click through their notes and gather the information they need. And that’s more efficient than having them actually read through the prose and all the language that might exist in an AI-generated note.
We’re seeing those themes around some of the other tools we’re testing. Like the ability to draft in the basket. AI-generated content is really good and comprehensive, but it tends to be a bit more verbose.
The other technology we’re excited about and starting to use is virtual babysitters and virtual nurses, using some AI and visualization capabilities to provide alerts and more proactive management when those rates start to change. And that seems to have really big potential to improve efficiency and HR care and support.
Frankly, we’re also excited about the work planned and underway around patient outreach and further expanding our virtual tools, while also asking ourselves that, how can we make things easier, not only for caregivers and staff but also for patients to be able to schedule. yourself and receive care more easily?
Ask. How do you measure the success of AI-powered initiatives?
ONE. We’re handling this in a similar or consistent way to how we measure any new technology or solution. Perhaps it’s better to remind ourselves that we can continue to rely on the many time-tested ways we’ve deployed and used to measure technology over the years.
And that is we look to develop KPIs and metrics, and then we measure the performance of the initiative against those criteria. And those criteria are often tied to the problem we’re trying to solve, as well as the ROI we hope to get from the solution.
And so those outcome metrics should be pretty clear. In the example I mentioned in the access case, maybe we’d look at the next available appointment, or if we’re looking to expand digital scheduling capabilities, that’s a factor. number-simple denominator and percentage of where we are compared to where we are. want to become. So those are usually pretty obvious.
What sometimes becomes a little more difficult is that we don’t always have a baseline, we don’t always have a baseline metric, or maybe it’s something that’s a little more difficult to measure. Where possible, we will look to quickly gather those baselines or make some educated guesses that allow for extrapolation as a benchmark for the new tool.
In the case of ambient documentation, it is not always easy to quantify or measure a physician’s health status or burnout. Turnover is certainly one-way, but there is a level of burnout that may never be reported or result in turnover. And so it’s trying to measure if you haven’t done it yet.
Surveys are one way to do that, measuring happiness, intention to stay, etc. But then there are also other alternative metrics and notes that we can look at that are aspects of note-taking – time in pajamas, time outside of work, total time and documents. So there are ways to get information and measure that value, but in some cases it requires a little more intentionality and maybe some creativity that we don’t always have. proactively implement.
Q. How can AI help advance health equity?
ONE. There are a number of ways AI can help. It can analyze large amounts of health data to identify differences in access and outcomes, and it can help personalize care. AI automation can make systems more efficient to hopefully improve accessibility and availability.
A great example of that is something we’ve done at Ceder’s Sinai called CS Connect, a virtual health care option that has physicians available 24/7 for emergency care, inpatient care, and emergency care. same day and routine primary care only. That helps alleviate capacity challenges at our brick-and-mortar locations. And it can reach people whenever and wherever they need care.
There is a flexible guided response intake section with Q&A that the patient will experience during the intake process. They can see information about their potential diagnosis and then they have the choice of whether or not to see a doctor.
We recently expanded that offering to children, ages three and up, and to Spanish speakers, expanding the pool of people who can use these tools to get care.
Q. How does Cedars Sinai Connect address AI biases and train AI on datasets that reflect diverse populations?
ONE. We know the effectiveness of these large language models and AI tools depends heavily on the quality and diversity of the data it is trained on. We know that the more demographic and geographic categories we include, the more we can control for certain biases. If the population is not fully represented, we may have biased predictions.
So we know that’s important, as is the volume of data that goes into training and monitoring these tools for CS Connect. The AI technology was developed by a company called K Health in Israel, and we built the app experience with them. Again, back to the build vs. buy question.
We saw a gap in the market and decided to build. But the AI was initially trained on patient populations in Israel, and those populations were obviously very different from the people in our community in LA, and then across California, where the tool is.
So, recognizing that there are methods and mathematical approaches to tuning data sets as well as training to ensure that our population is accounted for to control for these types of biases There is also a growing appreciation that data and training is local and it should be.
And we need to take that into account when building these tools, along with ongoing training and monitoring of models as they are deployed. When we launched CS Connect, we had about 10,000 patients using the tool and about 15,000 visits. All of those patients and visits will help train and enhance the models continuously, hopefully continuing to improve accuracy and maintain the safety and correctness of the solution over time.
Editor’s note: This is the eighth in a series of articles featuring leading voices in health IT discussing the use of artificial intelligence in healthcare. To read the first article about Dr. John Halamka at the Mayo Clinic, click here. To read the second interview with Dr. Aalpen Patel at Geisinger, click here. To read the third part, with Meditech’s Helen Waters, click here. To read the fourth part, with Epic’s Sumit Rana, click here. To read part five, with Dr. Rebecca G. Mishuris of Mass General Brigham, click here. To read part six, with Dr. Melek Somai of Wisconsin’s Froedtert & Medical Network, click here. And to read part seven, with Dr. Brian Hasselfeld of Johns Hopkins Medicine, click here.
Follow Bill’s HIT news on LinkedIn: Bill Siwicki
Email him: [email protected]
Healthcare IT News is a publication of HIMSS Media.
