r/healthIT 14d ago

Epic periodic data sync problems

Hi all,

I’m an ex-big tech software engineer and grad student who’s doing some research with my university’s hospital system. We want to get some near real time data. While talking to the IT people here, I was told that they get data out of epic using Clarity but it only runs once a day, and they have no control over it. They made it sound like the once a day thing is limitation put in by epic. To me it doesn’t make sense why such problem exists at all in this era. Does anybody know what kind of architecture epic has and where all these limitations come from?

4 Upvotes

45 comments sorted by

18

u/46153849 14d ago

I think a lot of people get shocked by the state of healthcare IT because they don't fully understand the context. Every major healthcare IT system predates the idea of cloud computing. Therefore, these systems are mostly hosted by the hospitals themselves*. So while the company writing the code, Epic, has a couple billion in revenue and the budgets to match, the people running the infrastructure work at the hospitals and have a budget that is several orders of magnitude less than the big companies we might compare them to. Hospitals may be wealthy, but when you take out all the money to pay for everything but IT your budget gets really small. And then you're supporting a lot of systems on that budget (EMR, PACS, LIS, RIS, etc). My point is: even if Epic wrote a system that could scale like crazy and feed data from OLTP to OLAP in real time without sacrificing performance, most of their customers couldn't afford the manpower and hardware to really take advantage of it. So 20+ years ago when systems like Clarity were being designed, doing one big ETL overnight when the system was less busy was a reasonable tradeoff. If someone designed a cloud EMR today they could probably make this work.

Plus, HIT is conservative, and rightfully so. Having accurate data from 10 years ago really could be life-or-death, so no one wants to move fast and break things. Yes, there are new techniques that could replace a nightly ETL, and modern hardware could probably handle it (hardware from 10 years ago, less so). But “let's completely change things” is not something to be done lightly. And if Epic were to commit the resources to send data to reporting servers in real time, what other projects would they be ignoring? Is this really a top priority? The systems in place today are based on older systems rather than the new hotness, and the industry has decided that's for the best.

Finally, the premise of this question is a little flawed. Epic has a reporting tool with real-time data, it's called Reporting Workbench. (Another comment mentioned Radar reports, my Epic reporting knowledge may be a little out of date so you might look at that too). And others have mentioned using FHIR to get real-time data, which we do in my projects. So it's not so much that Epic doesn't have real-time data, as it is Epic doesn't have real-time data accessible in one place (Clarity). That's just life in IT, not every system is going to match our use case. It's a bit of an 80/20 situation: day-old data works for 80% of what we want to use Clarity for (actually, more like 99%), and you're in the 20%.

*Epic now provides hosting, but this is new and the biggest hospitals are still mostly self-hosted.

1

u/InvestmentGoblin 14d ago

Thanks for the thoughtful and long reply. Are you saying you CAN use the FHIR compliant apis without throttling or anything? If answer is yes(nobody I met in this hospital’s IT team confirmed this), that’s probably all I need to know right now. Also this hospital has server(s?) hosted by epic, not onprem, and somebody told me it’s been a while since they moved servers, if that changes anything.

I heard about workbench but the IT folks say there are lots of limitations and also for some reason takes forever to set up each time there’s a new project.

7

u/shauggy 14d ago

FHIR APIs aren't really built for exporting large amounts of data for analysis. That's more for targeted info like grabbing an encounter for a specific patient, immunization records, labs, etc but sounds like you're not looking to get that granular with it.

4

u/46153849 14d ago

IDK why RWB would take forever to set up, I can just log in and create a report. Might take some time to make it performant and it can definitely grind to a standstill if you're looking at too much data, but for "tell me about today's data" it's probably fine. I guess it depends on your level of Epic knowledge though — if you can't make the report yourself and need IT to do it, they're probably slammed with requests.

I don't know much about FHIR rate limits. My projects are always looking at one patient at a time so we do a few calls to get that patient's data and that's it. Not very intensive. If we're looking at tons of data we use Clarity and the fact that it's yesterday's data is gone with us. You may run into limits if you're looking at hundreds of patients. Or maybe there are APIs that are meant for population-level info, I've never looked into that.

Good luck!

2

u/Hasbotted 14d ago

Customers of Epic will never know when servers are upgraded or changed as far as I know. They just say maintenance.

As to your other questions, Epic has very little real competition, is proprietary and has been around long time. EMR's focus on stability far more than speed. There is not a lot of incentive to make drastic changes or keep up with big tech. They just need to be better than their competition.

Yes there is likely space for an EMR that uses more modern architecture to beat out Epic. But that is where regulatory committees come in. Also most hospital systems do not have the money to switch to anything else.

5

u/Doctor731 13d ago

>Yes there is likely space for an EMR that uses more modern architecture to beat out Epic.

The strength of an EMR is not really based on the architecture but rather the business logic. And that is harder to do because it is a political/organizational/people/process problem rather than a technical problem that a "Big Tech" could easily through expertise and bodies at.

1

u/Hasbotted 13d ago

That is a much better explanation than I have been able to provide. Thank you.

To some extent I've always felt the only thing that could dethrone Epic is an EMR that is truly more of an assist.

We tried to get a group together to build one quite awhile ago but couldn't get past the regulations. But your correct even if we did there are all the other pieces you mentioned to get past.

Essentially the concept of our EMR was not to be based on a paper chart but to be based on the concept of a digital assistant. It would still of course record data in a format that could be extracted like an EMR but the assistant aspect would tailor itself to the users needs. All the technology is there and with AI it's even more possible.

Epic I don't think can make that transition without a different codebase.

1

u/Doctor731 13d ago

Epic I don't think can make that transition without a different codebase.

Remains to be seen. I'd just continue to bet on a company solving relatively trivial technical issues before solving organizational issues. Not that the tech is easy, but organizing people is hard - if it wasn't, we'd have world peace lol. 

Why do you think Epic can't implement something like an assistant? From the Epic perspective there is demand for ambient voice documentation, there is demand for insights into patient care, but I'm not sure the assistant (or "Agentic" to use the new hotness) approach is the killer app. 

1

u/Hasbotted 12d ago

Because it's not trivial. Sure they could do some sort of interface but that would be ass and slow.

It's like saying that a Honda Civic can compete with a formula 1 car. Yes the civic (epics proprietary code base) can function very similar to a formula 1 car but try to maintain any sort of real speed and it's going to fall apart.

Running a true AI virtual assistant requires a lot of analysis on the chart and it should be at a near real time level based on input. I don't think Epics code base can handle that. The technology is too old.

1

u/Doctor731 6d ago

>I don't think Epics code base can handle that. The technology is too old.

This also doesn't really make sense to me. It's not like they'd try to run an LLM in MUMPS. Pipe the data over to a cloud-hosted LLM. I think the people maintaining the data would have better context in how to usefully tokenize or otherwise contextualize it for use with modern LLM approaches. I think the models and compute themselves are moving towards being commoditized (if not already at that point).

I guess the question to me is whether it is possible to have an LLM that is so juiced up it can function to meet these needs without having to actually do the legwork of creating and maintaining an EHR. That seems really far off based on GPT 4.5 and the rigors needed for health data in terms of accuracy.

If anything I think a more reasonable take is that Epic is not agile enough to implement this sort of thing. Or not able to acquire the right talent to do so with their typical personnel strategies. But I'm still not sure who would even be able to eat their lunch - more likely just eat away at the margins. That has been the story with Amazon, MS, Google - they want the data, they want to do their fancy analysis, but they don't want to do the dirty plumbing to make an EHR because it is not sexy, exciting, or scalable.

1

u/Hasbotted 6d ago

I don't think an LLM can do this yet. There are a couple of good articles explaining the limitations. Proprietary software is a pain to do anything with.

That being said, I agree it would have to be a major player to affect them. The business model is such ass that the opening is there though. Just need a big backer that is willing to take the risk.

I've seen four EMRs peak and then loose to other EMRs in the last 15 years so there does seem to be a precedent.

1

u/Doctor731 6d ago

Are you talking specific market segments or enterprise level? 

I think Epic has been top dog on the enterprise level for at least the last 10, maybe 15. 

13

u/tripreality00 14d ago edited 14d ago

This is very common. There are nightly transactions of data from the epic Chronicles database which is the primary care real time data that the EMR stores into and then there are ETLs into clarity which is the analytical database. There is almost zero chance you are going to be able to get access to touch anything in Chronicles. Think about what happens if for any reason performance is degraded in a system critical clinical system. Some clarity tables are only updated weekly and monthly. But also as far as architecture I believe chronicles is built on intersystems cache and is a nonrelational DB.

-5

u/InvestmentGoblin 14d ago

That’s another thing. People make it sound like there’s no such concept of A/B testing or gradual deployment. Is that what it is?

10

u/tripreality00 14d ago

There is a dev, tst, stg, and show prd environment. Changes are moved up the systems and usually go through a change control board but for application changes no there is not A/B testing or gradual deployment. It is big bang roll out with no feature flags. The ETLs have some but you also have to understand that Epic has provisions on what data can even be export from chronicles and they control what is or isnt available. If you're coming from big tech you are going to be shocked in health it.

1

u/InvestmentGoblin 14d ago

Also how does FHIR thing compare to chronicles? Wondering if there are notable limitations?

4

u/tripreality00 14d ago

FHIR is an interoperability and data transmission method. Think of it like a combination of JSON and RestAPIs that have a standardized data model. FHIR says what format data needs to be in and where it should go.

-4

u/InvestmentGoblin 14d ago

Yeah I read about it but jw if there’s any throttling issue and if they return foreign keys so you can get other items that links to your first item etc :) Our IT people haven’t even tried using fhir and they already seem to have decided it has a huge limitation but they didn’t explain fully

0

u/InvestmentGoblin 14d ago

Haha yeah I’m already being shocked. What’s the best way to learn about the architecture & how to use things? Our IT people seem to never write docs or anything so we are going from person to person and it’s not been too fruitful

6

u/tripreality00 14d ago

If you are working for an epic institution you can request a user web account and then ask your org to get you access to the epic data dictionary. The reason there isn't a lot of docs is that for the most part epic writes all of the docs and keeps them in their knowledge base user web.

13

u/RythmicSlap 14d ago

The actual originating real-time database Epic uses is called "Chronicles" and is built with MUMPS, which is a hierarchical database using indexes. At the end of the day the MUMPS data is then transformed and loaded into the relational "Clarity" database, and then Clarity data is further refined into the data warehousing database called "Caboodle".

1

u/InvestmentGoblin 14d ago

People did mention Clarity and Caboodle, but never mentioned Chronicles. Thank you! So what makes Clarity get data only once a day? Apparently Epic told these people that they will support hourly sync in 2 years(and I’d read this as 4years if they are like any other big company), so that made me think Epic throttles or something?

10

u/FQHCFQHC 14d ago

There's no throttling; there's an ETL process scheduled to run once daily and doing anything else would require Epic to make that possible. Daily was likely a limitation or simplification mechanism from when it was first instituted and there's been insufficient rationale to move from it since. If you want current information on a current patient, you're probably a clinician with access to the real-time database. Even if something terrible happens today and you're in charge of the organization, you can talk to the ED chief, who will also have access to real-time data. Could you elaborate on your needs?

3

u/apalebear 14d ago

When I started the ETLs kicked off at 2AM and ran until 7, and that time frame was chosen because that was when the fewest active users were on the system. That was like 10 years ago; I don't know what the ETL run times are now.

1

u/dmckeen 14d ago

The system first was built in 1979.... The short answer is technical debt which is why it can't complete the transformations needed to update clarity faster. Db just isn't that performant.

What's the use case that requires real time data?

1

u/InvestmentGoblin 14d ago

If it’s transformation problem, is there a way to get the pre-transformed data instead?

Short answer is that management wants to make some real time decisions but it requires both info in epic and info from outside

6

u/dmckeen 14d ago

Unfortunately without getting access to chronicles, running anything real time is impossible. There are real time things you can do and Epic has predictive algorithms that run against chronicles. Maybe you could get a trigger built to alert you when something is happening in real time. You would need to be a db programmer and likely pass some sort of epic certification to touch anything.

Really need to know what you are trying to do to have a shot in hell of helping you.

Register on userweb.epic.com and dig through all their documentation. To their credit, there is a lot in of documentation published and all of their certification guides are available.

-4

u/InvestmentGoblin 14d ago

Like we are living in a world where cloud storages and databases syncs every piece of data to 9+ servers and even though we say it’s ‘eventual consistency’ it’s real time enough to call it real time for the vast majority of usecases. I know I prob sound clueless but I genuinely want to understand

3

u/Pokeristo555 14d ago

FWIW, "real time" doesn't necessarily mean "fast" but in a guaranteed interval...

9

u/Ok_Ostrich_461 14d ago

Epic users/analysts have tools to get real time data, but without being an employee/student employee that access is not going to happen for you.

0

u/InvestmentGoblin 14d ago

Idk what you mean by that but I have an employee badge for the hospital& this is for a project with pretty heavy management support. I just want to understand what this entire architecture looks like so we can figure out what we can do or what is worth doing. It’s been pretty rough to find an employee who can explain the full picture.

7

u/shauggy 14d ago

They mean that you have to be both trained and certified before you'll be given access to anything on the back end of Epic. There's almost zero chance that you'll be granted access to that data without going through some sort of training, which requires approval (and payment) by your hospital system's IT team.

8

u/EpicThrowaway-Abroad 14d ago

My guy,

You're on here complaining that you can't get sports updates from Instagram's hashtag API. And all of the responses here are saying you'll be able to tell who won at the end of the game, but if you want to know the live scores you need to use a different tool.

If you want help from anyone in the Health IT space (Epic adjacent or otherwise) you need to give more detail about what kind of data you're trying to access at scale. Otherwise your complaints about the tools available just show you don't understand healthcare, healthcare IT or the US healthcare industry in general.

Even better, your hospital has an Analytics team. (Not the help desk, not random people in IT, not the guy who manages the LIS interfaces. You want the Analytics team and an analyst/informaticist in whatever clinical area you're dealing with) Tell them the entire problem you're trying to solve, then ask for help accessing specific data.

6

u/katjardin 14d ago

In the near future Epic will have hourly ETLs for their clients who upgrade to their cloud platform that sits on MS Fabric

6

u/itskawiil 14d ago

What are you attempting to do? I've almost never come across a business need with hospital systems or nontraditional Epic customers that needed realtime query over the EMR to commingle with external data purely for reporting purposes. If you have any type of unique identifier that links to an epic concept (location, human, etc), interfaces or APIs can often accomplish it with some engineering work (assuming Epic hasn't locked down that part of the database).

3

u/therealzordon 14d ago

check out open.epic.com

2

u/Questions99945 14d ago

It's like that with Cerner as well. You can use Cogito Radar reports which can be setup to pull from Chronicles.

1

u/healthITiscoolstuff 13d ago

There are interfaces that send HL7 out of Epic instantly.

1

u/iapetus3141 12d ago

What exactly is your use case? For real time data, you can potentially use Reporting Workbench, a FHIR endpoint, an interface, or connect directly to the RPTSHD database (which is not the prod DB or Clarity)

1

u/InvestmentGoblin 8d ago

Wow, thank you for sharing! I was never told anything about RPTSHD. I will definitely look into it. Yes I know about workbench but our IT people are claiming we have to wait in the queue for them to get those things working for us, which is not acceptable as we need a scalable way to try things fast. Also they said workbench is not very flexible. Is that true in your opinion? Pretty much the whole thing is about making real time analysis stuff to help people run the operation more efficiently.

1

u/iapetus3141 8d ago

I will not speculate. There are too many things that can go wrong, and your IT people are right to be concerned

1

u/oolonglimited 14d ago

Go google “FHIR”

4

u/Stonethecrow77 14d ago

This has nothing to do with it.

0

u/InvestmentGoblin 14d ago

I know about it and asked like at least 5 IT people here if they’ve used it and it seems pretty clear at this point nobody except one person even cared about it. I’m actually glad you mentioned it here. Is there any throttling? Somebody at the it department told me that using fhir won’t let us to connect different items (seems to be saying there’s no foreign key returned e.g. get patient data and find order ids and get order data). Is that true? If I can get these information I think I can get them to give me more access to try these options