Here’s the latest event in the saga of VA/DoD health information sharing from Bob Brewin: Military Health System and TRICARE Lose Control Over IT Budget

One official said the move reflects frustration among senior Pentagon leaders with MHS efforts to procure new health IT systems, both independently and in partnership with the Veterans Affairs Department to develop the integrated electronic heath record. The departments have spent at least $1 billion over the past five years pursuing an integrated system.

This follows Chuck Hagel’s testimony to Congress that We Don’t Know What the Hell We Are Doing and former DoD Undersecretary for Health Affairs Ward Cascell’s that 2009 revelation that AHLTA is Intolerable.   I got private emails from DoD docs that were even more explicit:

AHLTA is far worse that you even alluded. It has virtually sucked the life out of our Providers and our MTFs. Yes, there may be some benefits but the pain is worse than the gain. I can’t believe that there will ever be a system that could successfully create a bi-directional interface with AHLTA. Any discussions that CHCS Ancillary functions will be replaced by the AHTLA as an architecture are just smoke screens for the embarrassment that AHLTA really is.

The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.

AHLTA is more than Intolerable…It’s the 3rd highest reason listed by the Army at the June 08 AUSA Conference Providers are leaving the military…

The first time I saw the AHLTA design, I thought that this was a reversal of all the successes the federal government had seen in health IT.  I remember thinking, “This is just one giant single-point-of-failure.”

30 years ago, we had two operational VA/DoD sharing sites. Here is March AFB’s Beth  Teeple’s oral history of the March AFB/Loma Linda test.

Thanks in part to Chuck Hagel’s early support of VistA, Rep. Sonny Montgomery, chair of the House Veterans Affairs Committee, noted that while VA had deployed a Core VistA system in all 172 hospitals for $82m, DoD had only produced prototypes of 4 stand-alone modules – for $250m (prices in 1985 dollars).  DoD called this IOCs – Interim Operating Capabilities, but we called them “Incompatible Operating Capabilities.” Each was  completely independent of the others, using incompatible coding systems, hardware, user interfaces, and communications protocols.  “Integration” was intended to come later.

This was classic DoD “Humpty Dumpty” development.  Break the system into pieces, then hire systems integrators to put it all back together again.  This is a wonderful business opportunity for the beltway systems integrators, but after 30 years of broken systems, its time to reevaluate the whole approach.

VistA never broke into pieces, but was based on common metadata and a shared set of tools.  It was “integrated” by virtue of never having been “disintegrated.”  Over the years, I learned that when someone speaks of “integrating” a system, we have to ask, “what disintegrated it in the first place?”  Until those forces are addressed, there is little chance of success.

Here is an excerpt of a letter  Sonny Montgomery sent me in 1984 Sonny Montgomery sent me in 1984:

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!

The VA and DoD forked into two paths: DCHP became VistA, and has won many awards and distinctions.  DoD reluctantly accepted CHCS, but under its management, has spiraled down into the mess we see today.

Here is 1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS, my 1985 overview of DHCP to TRIMIS Program Office, 

And here is a 1984 oct 4 montgomery letter to Sec Def Casper Weinberger re DoD use of VA software:

Mr. Secretary, I cannot understand the DOD reluctance to try the VA system, which will provide on a timely basis the mandatory system compatibility between the two agencies.

It’s amazing that we are having the same conversation 29 years later.  Not a whole lot has changed, except that we’ve spent billions of dollars and decades delivering “intolerable” health care to those who most deserve it.

I’m getting tired of rehashing 30 year old events, but it seems necessary.  DoD has been relentlessly trying to do the same thing – and failing.  It’s time we break out of the “More Expensive Failure” mode and move to an approach that works.

In my next post, I’ll present a proposal for some solutions.

Secretary of Defense Chuck Hagel testified before a Congressional hearing yesterday about the Integrated Electronic Health Record project: “I didn’t think we knew what the hell we were doing.” I’m glad that he put the stop to the effort after only $1 billion, the UK National Health Service blew an incredible $17 billion before pulling the plug.

This ratchet ups the rhetoric of Assistant Secretary of Defense for Health Affairs’ Ward Cascells’ 2009 assesment that DoD’s AHLTA system is “Intolerable”

Not to kick a dead horse, but this has been going on for nearly 40 years now.  The DoD had spent $250 million prototyping the TRIMIS system, a collection of incompatible demonstration systems, while we at the VA were delivering a working, integrated hospital information system (DHCP) for 172 hospitals for $82 million.  House Veterans Affairs Committee Chairman Sonny Montgomery hit the roof over this, and partially triggered by the successful VA/DoD integration prototype declared that one of the competitors for the CHCS system would propose an adaptation of DHCP (now called VistA).  I went to SAIC to help with the effort, and we ended up delivering a very successful CHCS system, which is still the core of the DoD EHR system today.

The fundamental problem with the DoD is that they do not understand how to deal with systems of the complexity of a modern health care system.  They understand how to build an aircraft carrier, make sure that the troops have the supplies they need, and other activities from a linear perspective: the whole is equal to the sum of the parts.  Break the carrier into pieces, design all the pieces, and put them back together again to make a whole carrier.  This is (kindof) well and good for things that have this whole-equal-sum-of-parts quality.  Toasters can be taken apart and put back together again, and will still be the same toaster.

But health care is far more complex and dynamic than an aircraft carrier.  The hospital, Peter Drucker said, is the most complex organization in modern society.  Like a cat, we cannot dissect a hospital and put it back together again.

In a lesson straight from Humpty Dumpty, DoD wanted to break the electronic health record system into 17 “best of breed” applications, then hire a “systems integrator” to put Humpty together again.  This is like trying to build the world’s best car by trying to integrate the engine from a Corvette with the seats from a Rolls Royce and the chassis from a Porsche.  But despite how lucrative it is to be one of “all the king’s men,” it is simply not going to happen.

Yesterday, a friend of mine with many decades in the health IT industry called to tell me that he had just signed up with the VA in San Diego.  He said he was amazed at how well coordinated his care was – and this is from someone trying to do this in the private sector for 30 years.  This is the result of a fundamental approach taken from the earliest days of VistA – we were “integrated” by virtue of the fact that we never “disintegrated” into pieces.  We build a tool kit from which we composed the system over time, instead of the DoD’s approach of decomposing the system into pieces and then trying to put them back together again.  In that sense, VistA is remarkably similar to Wikipedia in this sense.  (See my discussion with Wiki inventor Ward Cunningham on this topic.)

So the fundamental issue is that we are dealing with a cat-like problem with toaster-like thinking.

Here’s my proposal: Fund a Skunkworks to get us out of this mess:

I’ve been developing VA/DoD interfaces since 1985.  They were technically correct, but politically incorrect.  I would hope that in the future, we can get past all the political nonsense of the past few decades, and just settle in to getting the technology working.

1.  Give me a contract to form a skunkworks.  I’ll collect 8-10 of the smartest people I know to develop the simplest solution that is “good enough” to get started.  I’ll also define an approach for “making it better.”

2.  I would like a couple of hospitals to work with (preferably in the San Diego area), at least one VA and one DoD.

3.  Connect me with teams of folks (both in the VA and DoD) who are passionate about making this thing work.  Create a bonus pool (for DoD as well as VA) against which I can make recommendations for bonuses for their contributions to the success of the skunkworks.

4.  Run interference for me to keep the bureaucracy out of the way.  We’ll be doing this in San Diego, far outside the beltway.

This can be a parallel activity to whatever the inside-the-beltway thinkers want to mull over.  Just ramp up a small, talented team working on the problem, independent of what is formally happening.  Think of it as portfolio diversification. The skunkworks funding would be just a fraction of what the IEHR-style aproach has been.

After reading Bob Brewin’s piece Did Jon Stewart Foil the Pentagon’s Health Records Plan? I have decided that Jon Stewart is a worthy recipient of the Unlimited Free Passage on the Underground Railroad certificate.  He understands the problems that the hardhats have been facing in the development of VistA over the years, and also seems to understand the success that it has enjoyed despite the hardships over the years.   I hope that this recognition will help him see some opportunities for improving government, health care, and service to our Veterans.  It is also an amazing story of how a bottom-up, decentralized approach to innovation can work, even in the most hardened bureaucracy.

The VA MUMPS Underground Railroad was formed in the early days of the VistA development in response to the attempts of the centralists to shut down a field-based decentralized approach.  The Hardhats were the technical folk who wrote the code to make it happen, but VistA was always more than just source code, so we needed recognize the many others who were involved in making it a success as described in Phillip Longman’s book Best Care Anywhere and this video. US Medicine editor Nancy Tomich describes the situation.  Nancy and I are now working on the New Health Project to carry things to the next generation.

The Underground Railroad has been struggling to build a common vision of VA/DoD health sharing for decades,   and not without its casualties,  so it is good to finally see some media attention to the issue.

This is the most prestigious award offered by the Underground Railroad, having previously been given in 1982 to Chuck Hagel:

In keeping with the Underground Railroad’s history, his certificate can only be given in person, with appropriate ceremonial presence.

I just watched Jon Stewart’s Daily Show for March 27 and was amazed to see VistA and ALHTA mentioned (starting around minute 7), castigating both the VA and the DoD for their “incompatible” medical record systems.  Since this has been my briar patch for 35 years now, it is amazing to see this play out on national TV.

I had a working VA/DoD medical record system working in 1985:

It was thoroughly studied by Congress, GAO, VA, and Arthur D. Little consultants, and passed with flying colors (well, except for some of the dirty tricks that DoD pulled, trying to make it look bad).  Here is an oral history interview about the system.

Continue Reading »

The details for the Integrated Electronic Health Record (IEHR) are just now beginning to roll out.   It’s pretty much a replay of the “Best of Breed” marketing approach that I’ve seen been pitched for decades.  Basically, collect all the parts and the whole will fall together with “just a bit of integration.”  They are issuing dozens of RFPs and RFIs describing all the pieces they seek; how it all fits together is a lot of hand-waving at the moment, with an inordinate faith in the role of an Enterprise Service Bus to magically make everything work together.

This is a little like someone selling you the world’s best car, based on “best of breed” components from all the best manufactures.   An engine from a Corvette, seats from a Rolls Royce, transmission from a Ferrari, etc. To make it all fit together, they would say it is all “standardized” – every part would use metric nuts and bolts, 12 volts electricity, and a common electrical harness (equivalent of the Enterprise Service Bus), to make sure that all the parts could work together.

We took an opposite approach in VistA, starting with a conceptual model of a system driven by active metadata – a common roadmap to patient database.  We were “integrated” by virtue of not “disintegrating” into pieces in the first place.  We had an overarching Conceptual Integrity to the design that gave us a common foundation from which to grow, climbing up the ladder of abstraction rather than falling down it.

This turns out to have been a good decision.  The President’s Council of Advisors for Science and Technology (PCAST) issued a report the report—Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward— that specifically pointed out VistA as one of the success stories for large scale EHR systems.  It also advocated greater use of metadata: tagging data dynamically through meta rather than “pigeonholing” it into pre-defined, previously synchronized slots.

This argument can be very abstract to some people, particularly to bureaucrats whose entire life has been engulfed by hierarchies and “pigeonhole” systems thinking.  The notion that systems can associate dynamically, spanning hierarchies (or not even being associated with a hierarchy) casts them into unfamiliar uncharted territory.  To them, complex systems require complex organizations to manage and regulate them. They seek efficiency, rather than resilience, adaptability, or robustness.

In order to make this a bit more understandable, I am proposing a theme song for the IEHR architecture: Johnny Cash’s One Piece at a Time.  Think of the IEHR vision as Johnny Cash’s Cadillac.  I’ll let the viewer draw their own analogies.  Of course, Johnny didn’t have billions of taxpayers dollars to spend, nor did he have a gaggle of consultants ready to reap huge contracts to do the “little bit of integration.”

From the initial 1978 Oklahoma City VA/DoD/IHS meeting, we had envisioned a government-wide health information system, based on shared, open source software.  Things went well for the VA, and we deployed a nation-wide system starting in 1983.  Things looked bright for a while for DoD use, as we installed a shared system at March AFB in Riverside, CA. connected to Loma Linda VA hospital.  Congressman Sonny Montgomery got wind of the effort, and supported it to the hilt.  DoD didn’t like the idea – spending more money on consultants to make it look bad than they did to try out the system to see if it worked.

Here are some papers I’ve scanned in that relate to some of the early history of VA/DoD sharing.

Despite these being nearly 30 years old, the issues they talk about are pretty much current – just part of a never-ending story about VA-DoD integration efforts.

1985 munnecke overview of DHCP to TRIMIS Program Office

1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS

1984 oct 4 montgomery letter to weinberger re DoD use of VA software

1984 nov 5 montgomery letter to Underground Railroad

1986 Anon letter to DOD Inspector General re alleged conflicts of interest in CHCS

1985 first VA DoD email message exchange at March AFB

1984 MITRE report on Utilization of VA software in the TRIMIS program

1984 Octo Barnett responds to MITRE report on DoD methodology

1985 munnecke email re ADL dirty tricks

1997 US Medicine article by tom From DHCP to Vision for Change

1978 Tom Munnecke’s Original DHCP FileMan and Kernel design notes at OK City kickoff meeting

Today is the 33rd anniversary of the 1978 Oklahoma City kickoff meeting for what was then called the CASS (Computer Assisted Systems Support Staff), later to be called Decentralized Hospital Computer Program, which today is called the VistA EHR.  We had groups from the VA, DoD, Indian Health Service, and other academia.  This is where we laid out the basic structure of the metadata (Data Dictionary), File Manager, the basic utilities (a common date routine, for example, that was Y2K compatible.).

I have lots of stuff to scan and post, but I’ll start with my my original notes for the design of the security system.  This explained the process by which we would control access to patient data through a system of privilege codes.  Our first two files were Patient (#2), and User (#3).  File #1 was supposed to be the Data Dictionary, as a meta-level description of the other files, but that idea (which today might be called Meta-Circular Evaluation of a Homoiconic Language but that was a bridge too far for the times.   So, we spun the data dictionary spun off as it’s own metadata world.  The lesson to learned is that VistA at its core is closer to an LISP-like, artificial intelligence approach than the standard COBOL/SQL “modern model” of the times.  This is the hardest thing for me to communicate to newcomers to the architecture – they look only at the code, not the metadata.

We programmed this security model into the VA’s system, and it was reviewed by the federal Computer Security Center (using what was called the Rainbow security guidelines.)  I remember flying to San Francisco to spend two days with men in black suits who wouldn’t identify the agency they worked for… The system passed with flying colors, and they were very complimentary about the design.   Then, about 7 years later, as we were porting the system to the DoD as the CHCS system, I repeated the validation process, only with a larger group of DoD, GAO, and more mysterious men.  They didn’t bring thumbscrews to test me with, but it was quite a nerve-wracking experience.  We passed that  inquisition, too.

So, these scribbles ended up controlling the access to nearly all federal electronic health record access for the past 3 decades.  (The Indian Health Service also used it for their RPMS system, as well).  As far as I know, they have never been breached technically.  The breaches have always been authorized users doing bad things with the data, such as a VA employee with access to doctors’s records selling a list of their SSNs and home information to some nefarious buyers.

It’s fun to go back to retrace my design steps in the early days.  I drew the original onion diagram on a placemat in June, 1978 at Coffee Dan’s restaurant in Loma Linda, CA. over dinner with George Timson, which delineated the initial core of 19 commands, 22 functions and one data type.  This was before I was hired by the VA; I started in September, 1978.  I think we started with the Patient and User file right off the bat, and this security logic linking the two was probably the first code I wrote.  Then I moved on to writing some of the early data dictionary logic.

I know that Richard Davis from Lexington was also quite active in this area, and one of my primary intellectual sparring partners for wrangling with some of the more abstract issues of the architecture.

This idea has certainly stood the test of time, and I think it could be updated to carry through to an updated approach – coupling the access metadata to the data as it is queried and shared.  I suspect that it could be integrated into an RDF Schema which could be used for generalized connection the Linked Data model.

I had thought that DoD’s AHLTA held the record for being the world’s greatest health IT fiasco, spending $4-5 billion of taxpayer’s money on a system that was deemed Intolerable by the GAO and an Assistant Secretary of Defense.  It was so bad that it is cited as one of the leading reasons physicians are leaving military service.  One user told me, “The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.”

I knew that AHLTA would be a failure the instant I saw a diagram of it – a giant, centralized single point of failure that ignored everything that I had found successful in doing the VistA architecture.

This is not just dollars we are talking about, or missed opportunity costs.  Bad software kills people.  I don’t know if we’ll ever know how many patients AHLTA has killed, but it has to be significant.

But enough about AHLTA: I just read this article NHS IT system condemned about the UK National Health Service is in the midst of a far greater fiasco:

In a jaw-dropping condemnation of the NHS National Programme for IT, the National Audit Office has exposed a white elephant in the final stages of collapse.

In what read as a final pronouncement, the NAO reported that after nine years and £2.7bn, (US $4.5 billion)  the NHS has failed to deliver its primary aim of an electronic care record for everyone in the country.

The situation looks so dire the system might continue leeching money from the NHS for another decade if the whole scheme and all its software is not seized by state liquidators.

They go on to say that they don’t think its worth going forward with the remaining US$7.7 billion:

The NAO (National Audit Agency) said CSC would likely fail to deliver the rest before its contract runs out in 2016. DoH had been in dispute over its contract with CSC for 18 months, trying to claw back some of the £5bn (US $8.2b) it had promised the supplier.

What is particularly galling about this situation is that both the US DoD and the UK NHS had access to an award-winning, open source hospital information system that has been running at this scale for 25 years… the VA’s VistA system.  As one of the original software architects of this system, I have seen an endless stream of novice health IT folks appear on the scene, thinking that because they know some other aspect of IT that they can apply this to health IT.  In the past, these folks would slink away after losing $100m or so.  But now, it seems the stakes have been raised to the tens of billions.

We took an innovative approach in designing VistA, and it worked…  So, why are people continuing to pour money down the same failed approaches?

The answer is to follow the money:  A $5 billion fiasco is extremely lucrative to the beltway bandits.  Sure, they have to weather a few editorials and a couple of roastings before the various legislators.  But this can all be fixed with a little lobbying effort.  They’ll apologize, promise never to do it again, change the name of the project, and double the price tag of the next round of funding.  And laugh all the way to the bank with the profits they made on the fiasco.  It’s likely that a successful implementation would have probably earned them less profit.

Tthis isn’t just about an atrocious waste of taxpayer’s money.  People die from bad software.

Tom Munnecke at Senate Veterans Affairs Hearing Oct 6, 2010

I presented at the Senate Committee on Veterans Affairs: Hearingsyesterday.  My presentation starts at about minute 60 on the video.

Here is my written testimony and here is my invitation letter from the Senate and here is a written copy of my oral testimony .

It was an interesting experience for me, getting to spend some quality conversation time with 4 senators.  Senator Akaka in particular impressed me as someone who was really interested in the democratic process.

I supported the move to Open Source Vista, but also warned that this was a medical informatics problem, not just a general IT problem to be thrown in the basket with all the other IT problems of the VA.

I also suggested moving to a personalized model for future care, thinking that this would simply much of the complexity around trying to integrate VA and DoD records.  I like to think of getting VA and DoD to work together is like mating elephants, only they are both alpha males.

One of the overarching themes of the hearing was the notion of doing small, incremental work, rather than “big bang” gold-plated requirements.  I was quite pleased with VA CIO Roger Baker’s perspective on this issue, although I worry what will happen when he leaves, and the passive/aggressive bureaucrats return to their old ways.  They are just hiding out in the bushes, waiting to pop up again when things are clear.

I’m researching a op-ed piece on federal health care software, and am browsing through my old proceedings from the early days of the MUMPS Users Group meetings.  These meetings were quite an entrepreneurial incubator: Judy Faulkner was starting Epic Systems,  Paul Egerman, founded Interpretive Data Systems, which then became  IDX and sold to GE HealthCare systems for $1.2 billion, Terry Ragon started Intersystems, and I had just started working for the VA to work on what was to become VistA, and spread to the Department of Defense as CHCS and the Indian Health Service as RPMS.

Here is scan of Judy Faulkner’s 1979 paper called PISAR: A Time-Oriented Data Management System.  Judy’s Epic Systems went on to become a powerhouse in the Electronic Medical Record space.  It is remarkably similar to the architecture I was developing for the VA VistA’s system with George Timson and others called the File Manager.

It’s interesting to look at these systems 30 years later.  Judy’s company, Epic, has become a giant in the electronic medical record world.  VistA has gone on to power the transformation of the VA documented in Phil Longman’s Best Care Anywhere.

Rumors abound that EPIC is the lead contender for replacing AHLTA.  An alternative is the Open Vista Initiative.

I have great respect for what Judy has accomplished over the years, but at the same time, I think it would be a travesty for the government to turn to a closed, proprietary system instead of a roughly equivalent open source, publicly available one.

Isaac Newton said, “If I have seen farther than others, it is because I have stood on the shoulders of giants.”  In today’s Federal health care IT world, it’s getting hard to see anywhere, because everyone is standing on each other’s toes.  The VA’s VistA system has been in constant evolution since 1978 – and continues to thrive today.  The DoD’s uses a “break and replace” model, running one system into the ground until it breaks, only to replace it wholesale with another, more expensive system.  (It spent $250 million on TRIMIS until it threw it away to spend $1.6 billion for CHCS, which it tried to replace for $5 billion with AHLTA, which it is now throwing away to be replaced by something else.)

How will DOD pull itself out of this bureaucratic disaster area?  Will it spend billions more on a closed, proprietary system, or will it join the VA to create an evolutionary open source approach that benefits all?

Stay tuned.

P.S. I’m digging out all my old papers from the era and will be posting them in an archive Real Soon Now.