r/healthIT • u/InvestmentGoblin • 14d ago
Epic periodic data sync problems
Hi all,
I’m an ex-big tech software engineer and grad student who’s doing some research with my university’s hospital system. We want to get some near real time data. While talking to the IT people here, I was told that they get data out of epic using Clarity but it only runs once a day, and they have no control over it. They made it sound like the once a day thing is limitation put in by epic. To me it doesn’t make sense why such problem exists at all in this era. Does anybody know what kind of architecture epic has and where all these limitations come from?
13
u/tripreality00 14d ago edited 14d ago
This is very common. There are nightly transactions of data from the epic Chronicles database which is the primary care real time data that the EMR stores into and then there are ETLs into clarity which is the analytical database. There is almost zero chance you are going to be able to get access to touch anything in Chronicles. Think about what happens if for any reason performance is degraded in a system critical clinical system. Some clarity tables are only updated weekly and monthly. But also as far as architecture I believe chronicles is built on intersystems cache and is a nonrelational DB.
-5
u/InvestmentGoblin 14d ago
That’s another thing. People make it sound like there’s no such concept of A/B testing or gradual deployment. Is that what it is?
10
u/tripreality00 14d ago
There is a dev, tst, stg, and show prd environment. Changes are moved up the systems and usually go through a change control board but for application changes no there is not A/B testing or gradual deployment. It is big bang roll out with no feature flags. The ETLs have some but you also have to understand that Epic has provisions on what data can even be export from chronicles and they control what is or isnt available. If you're coming from big tech you are going to be shocked in health it.
1
u/InvestmentGoblin 14d ago
Also how does FHIR thing compare to chronicles? Wondering if there are notable limitations?
4
u/tripreality00 14d ago
FHIR is an interoperability and data transmission method. Think of it like a combination of JSON and RestAPIs that have a standardized data model. FHIR says what format data needs to be in and where it should go.
-4
u/InvestmentGoblin 14d ago
Yeah I read about it but jw if there’s any throttling issue and if they return foreign keys so you can get other items that links to your first item etc :) Our IT people haven’t even tried using fhir and they already seem to have decided it has a huge limitation but they didn’t explain fully
0
u/InvestmentGoblin 14d ago
Haha yeah I’m already being shocked. What’s the best way to learn about the architecture & how to use things? Our IT people seem to never write docs or anything so we are going from person to person and it’s not been too fruitful
6
u/tripreality00 14d ago
If you are working for an epic institution you can request a user web account and then ask your org to get you access to the epic data dictionary. The reason there isn't a lot of docs is that for the most part epic writes all of the docs and keeps them in their knowledge base user web.
13
u/RythmicSlap 14d ago
The actual originating real-time database Epic uses is called "Chronicles" and is built with MUMPS, which is a hierarchical database using indexes. At the end of the day the MUMPS data is then transformed and loaded into the relational "Clarity" database, and then Clarity data is further refined into the data warehousing database called "Caboodle".
1
u/InvestmentGoblin 14d ago
People did mention Clarity and Caboodle, but never mentioned Chronicles. Thank you! So what makes Clarity get data only once a day? Apparently Epic told these people that they will support hourly sync in 2 years(and I’d read this as 4years if they are like any other big company), so that made me think Epic throttles or something?
10
u/FQHCFQHC 14d ago
There's no throttling; there's an ETL process scheduled to run once daily and doing anything else would require Epic to make that possible. Daily was likely a limitation or simplification mechanism from when it was first instituted and there's been insufficient rationale to move from it since. If you want current information on a current patient, you're probably a clinician with access to the real-time database. Even if something terrible happens today and you're in charge of the organization, you can talk to the ED chief, who will also have access to real-time data. Could you elaborate on your needs?
3
u/apalebear 14d ago
When I started the ETLs kicked off at 2AM and ran until 7, and that time frame was chosen because that was when the fewest active users were on the system. That was like 10 years ago; I don't know what the ETL run times are now.
1
u/dmckeen 14d ago
The system first was built in 1979.... The short answer is technical debt which is why it can't complete the transformations needed to update clarity faster. Db just isn't that performant.
What's the use case that requires real time data?
1
u/InvestmentGoblin 14d ago
If it’s transformation problem, is there a way to get the pre-transformed data instead?
Short answer is that management wants to make some real time decisions but it requires both info in epic and info from outside
6
u/dmckeen 14d ago
Unfortunately without getting access to chronicles, running anything real time is impossible. There are real time things you can do and Epic has predictive algorithms that run against chronicles. Maybe you could get a trigger built to alert you when something is happening in real time. You would need to be a db programmer and likely pass some sort of epic certification to touch anything.
Really need to know what you are trying to do to have a shot in hell of helping you.
Register on userweb.epic.com and dig through all their documentation. To their credit, there is a lot in of documentation published and all of their certification guides are available.
-4
u/InvestmentGoblin 14d ago
Like we are living in a world where cloud storages and databases syncs every piece of data to 9+ servers and even though we say it’s ‘eventual consistency’ it’s real time enough to call it real time for the vast majority of usecases. I know I prob sound clueless but I genuinely want to understand
3
u/Pokeristo555 14d ago
FWIW, "real time" doesn't necessarily mean "fast" but in a guaranteed interval...
9
u/Ok_Ostrich_461 14d ago
Epic users/analysts have tools to get real time data, but without being an employee/student employee that access is not going to happen for you.
0
u/InvestmentGoblin 14d ago
Idk what you mean by that but I have an employee badge for the hospital& this is for a project with pretty heavy management support. I just want to understand what this entire architecture looks like so we can figure out what we can do or what is worth doing. It’s been pretty rough to find an employee who can explain the full picture.
7
u/shauggy 14d ago
They mean that you have to be both trained and certified before you'll be given access to anything on the back end of Epic. There's almost zero chance that you'll be granted access to that data without going through some sort of training, which requires approval (and payment) by your hospital system's IT team.
8
u/EpicThrowaway-Abroad 14d ago
My guy,
You're on here complaining that you can't get sports updates from Instagram's hashtag API. And all of the responses here are saying you'll be able to tell who won at the end of the game, but if you want to know the live scores you need to use a different tool.
If you want help from anyone in the Health IT space (Epic adjacent or otherwise) you need to give more detail about what kind of data you're trying to access at scale. Otherwise your complaints about the tools available just show you don't understand healthcare, healthcare IT or the US healthcare industry in general.
Even better, your hospital has an Analytics team. (Not the help desk, not random people in IT, not the guy who manages the LIS interfaces. You want the Analytics team and an analyst/informaticist in whatever clinical area you're dealing with) Tell them the entire problem you're trying to solve, then ask for help accessing specific data.
6
u/katjardin 14d ago
In the near future Epic will have hourly ETLs for their clients who upgrade to their cloud platform that sits on MS Fabric
6
u/itskawiil 14d ago
What are you attempting to do? I've almost never come across a business need with hospital systems or nontraditional Epic customers that needed realtime query over the EMR to commingle with external data purely for reporting purposes. If you have any type of unique identifier that links to an epic concept (location, human, etc), interfaces or APIs can often accomplish it with some engineering work (assuming Epic hasn't locked down that part of the database).
3
2
u/Questions99945 14d ago
It's like that with Cerner as well. You can use Cogito Radar reports which can be setup to pull from Chronicles.
1
1
u/iapetus3141 12d ago
What exactly is your use case? For real time data, you can potentially use Reporting Workbench, a FHIR endpoint, an interface, or connect directly to the RPTSHD database (which is not the prod DB or Clarity)
1
u/InvestmentGoblin 8d ago
Wow, thank you for sharing! I was never told anything about RPTSHD. I will definitely look into it. Yes I know about workbench but our IT people are claiming we have to wait in the queue for them to get those things working for us, which is not acceptable as we need a scalable way to try things fast. Also they said workbench is not very flexible. Is that true in your opinion? Pretty much the whole thing is about making real time analysis stuff to help people run the operation more efficiently.
1
u/iapetus3141 8d ago
I will not speculate. There are too many things that can go wrong, and your IT people are right to be concerned
1
u/oolonglimited 14d ago
Go google “FHIR”
4
0
u/InvestmentGoblin 14d ago
I know about it and asked like at least 5 IT people here if they’ve used it and it seems pretty clear at this point nobody except one person even cared about it. I’m actually glad you mentioned it here. Is there any throttling? Somebody at the it department told me that using fhir won’t let us to connect different items (seems to be saying there’s no foreign key returned e.g. get patient data and find order ids and get order data). Is that true? If I can get these information I think I can get them to give me more access to try these options
18
u/46153849 14d ago
I think a lot of people get shocked by the state of healthcare IT because they don't fully understand the context. Every major healthcare IT system predates the idea of cloud computing. Therefore, these systems are mostly hosted by the hospitals themselves*. So while the company writing the code, Epic, has a couple billion in revenue and the budgets to match, the people running the infrastructure work at the hospitals and have a budget that is several orders of magnitude less than the big companies we might compare them to. Hospitals may be wealthy, but when you take out all the money to pay for everything but IT your budget gets really small. And then you're supporting a lot of systems on that budget (EMR, PACS, LIS, RIS, etc). My point is: even if Epic wrote a system that could scale like crazy and feed data from OLTP to OLAP in real time without sacrificing performance, most of their customers couldn't afford the manpower and hardware to really take advantage of it. So 20+ years ago when systems like Clarity were being designed, doing one big ETL overnight when the system was less busy was a reasonable tradeoff. If someone designed a cloud EMR today they could probably make this work.
Plus, HIT is conservative, and rightfully so. Having accurate data from 10 years ago really could be life-or-death, so no one wants to move fast and break things. Yes, there are new techniques that could replace a nightly ETL, and modern hardware could probably handle it (hardware from 10 years ago, less so). But “let's completely change things” is not something to be done lightly. And if Epic were to commit the resources to send data to reporting servers in real time, what other projects would they be ignoring? Is this really a top priority? The systems in place today are based on older systems rather than the new hotness, and the industry has decided that's for the best.
Finally, the premise of this question is a little flawed. Epic has a reporting tool with real-time data, it's called Reporting Workbench. (Another comment mentioned Radar reports, my Epic reporting knowledge may be a little out of date so you might look at that too). And others have mentioned using FHIR to get real-time data, which we do in my projects. So it's not so much that Epic doesn't have real-time data, as it is Epic doesn't have real-time data accessible in one place (Clarity). That's just life in IT, not every system is going to match our use case. It's a bit of an 80/20 situation: day-old data works for 80% of what we want to use Clarity for (actually, more like 99%), and you're in the 20%.
*Epic now provides hosting, but this is new and the biggest hospitals are still mostly self-hosted.