Here’s the latest event in the saga of VA/DoD health information sharing from Bob Brewin: Military Health System and TRICARE Lose Control Over IT Budget

One official said the move reflects frustration among senior Pentagon leaders with MHS efforts to procure new health IT systems, both independently and in partnership with the Veterans Affairs Department to develop the integrated electronic heath record. The departments have spent at least $1 billion over the past five years pursuing an integrated system.

This follows Chuck Hagel’s testimony to Congress that We Don’t Know What the Hell We Are Doing and former DoD Undersecretary for Health Affairs Ward Cascell’s that 2009 revelation that AHLTA is Intolerable.   I got private emails from DoD docs that were even more explicit:

AHLTA is far worse that you even alluded. It has virtually sucked the life out of our Providers and our MTFs. Yes, there may be some benefits but the pain is worse than the gain. I can’t believe that there will ever be a system that could successfully create a bi-directional interface with AHLTA. Any discussions that CHCS Ancillary functions will be replaced by the AHTLA as an architecture are just smoke screens for the embarrassment that AHLTA really is.

The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.

AHLTA is more than Intolerable…It’s the 3rd highest reason listed by the Army at the June 08 AUSA Conference Providers are leaving the military…

The first time I saw the AHLTA design, I thought that this was a reversal of all the successes the federal government had seen in health IT.  I remember thinking, “This is just one giant single-point-of-failure.”

30 years ago, we had two operational VA/DoD sharing sites. Here is March AFB’s Beth  Teeple’s oral history of the March AFB/Loma Linda test.

Thanks in part to Chuck Hagel’s early support of VistA, Rep. Sonny Montgomery, chair of the House Veterans Affairs Committee, noted that while VA had deployed a Core VistA system in all 172 hospitals for $82m, DoD had only produced prototypes of 4 stand-alone modules – for $250m (prices in 1985 dollars).  DoD called this IOCs – Interim Operating Capabilities, but we called them “Incompatible Operating Capabilities.” Each was  completely independent of the others, using incompatible coding systems, hardware, user interfaces, and communications protocols.  “Integration” was intended to come later.

This was classic DoD “Humpty Dumpty” development.  Break the system into pieces, then hire systems integrators to put it all back together again.  This is a wonderful business opportunity for the beltway systems integrators, but after 30 years of broken systems, its time to reevaluate the whole approach.

VistA never broke into pieces, but was based on common metadata and a shared set of tools.  It was “integrated” by virtue of never having been “disintegrated.”  Over the years, I learned that when someone speaks of “integrating” a system, we have to ask, “what disintegrated it in the first place?”  Until those forces are addressed, there is little chance of success.

Here is an excerpt of a letter  Sonny Montgomery sent me in 1984 Sonny Montgomery sent me in 1984:

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!

The VA and DoD forked into two paths: DCHP became VistA, and has won many awards and distinctions.  DoD reluctantly accepted CHCS, but under its management, has spiraled down into the mess we see today.

Here is 1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS, my 1985 overview of DHCP to TRIMIS Program Office, 

And here is a 1984 oct 4 montgomery letter to Sec Def Casper Weinberger re DoD use of VA software:

Mr. Secretary, I cannot understand the DOD reluctance to try the VA system, which will provide on a timely basis the mandatory system compatibility between the two agencies.

It’s amazing that we are having the same conversation 29 years later.  Not a whole lot has changed, except that we’ve spent billions of dollars and decades delivering “intolerable” health care to those who most deserve it.

I’m getting tired of rehashing 30 year old events, but it seems necessary.  DoD has been relentlessly trying to do the same thing – and failing.  It’s time we break out of the “More Expensive Failure” mode and move to an approach that works.

In my next post, I’ll present a proposal for some solutions.

Secretary of Defense Chuck Hagel testified before a Congressional hearing yesterday about the Integrated Electronic Health Record project: “I didn’t think we knew what the hell we were doing.” I’m glad that he put the stop to the effort after only $1 billion, the UK National Health Service blew an incredible $17 billion before pulling the plug.

This ratchet ups the rhetoric of Assistant Secretary of Defense for Health Affairs’ Ward Cascells’ 2009 assesment that DoD’s AHLTA system is “Intolerable”

Not to kick a dead horse, but this has been going on for nearly 40 years now.  The DoD had spent $250 million prototyping the TRIMIS system, a collection of incompatible demonstration systems, while we at the VA were delivering a working, integrated hospital information system (DHCP) for 172 hospitals for $82 million.  House Veterans Affairs Committee Chairman Sonny Montgomery hit the roof over this, and partially triggered by the successful VA/DoD integration prototype declared that one of the competitors for the CHCS system would propose an adaptation of DHCP (now called VistA).  I went to SAIC to help with the effort, and we ended up delivering a very successful CHCS system, which is still the core of the DoD EHR system today.

The fundamental problem with the DoD is that they do not understand how to deal with systems of the complexity of a modern health care system.  They understand how to build an aircraft carrier, make sure that the troops have the supplies they need, and other activities from a linear perspective: the whole is equal to the sum of the parts.  Break the carrier into pieces, design all the pieces, and put them back together again to make a whole carrier.  This is (kindof) well and good for things that have this whole-equal-sum-of-parts quality.  Toasters can be taken apart and put back together again, and will still be the same toaster.

But health care is far more complex and dynamic than an aircraft carrier.  The hospital, Peter Drucker said, is the most complex organization in modern society.  Like a cat, we cannot dissect a hospital and put it back together again.

In a lesson straight from Humpty Dumpty, DoD wanted to break the electronic health record system into 17 “best of breed” applications, then hire a “systems integrator” to put Humpty together again.  This is like trying to build the world’s best car by trying to integrate the engine from a Corvette with the seats from a Rolls Royce and the chassis from a Porsche.  But despite how lucrative it is to be one of “all the king’s men,” it is simply not going to happen.

Yesterday, a friend of mine with many decades in the health IT industry called to tell me that he had just signed up with the VA in San Diego.  He said he was amazed at how well coordinated his care was – and this is from someone trying to do this in the private sector for 30 years.  This is the result of a fundamental approach taken from the earliest days of VistA – we were “integrated” by virtue of the fact that we never “disintegrated” into pieces.  We build a tool kit from which we composed the system over time, instead of the DoD’s approach of decomposing the system into pieces and then trying to put them back together again.  In that sense, VistA is remarkably similar to Wikipedia in this sense.  (See my discussion with Wiki inventor Ward Cunningham on this topic.)

So the fundamental issue is that we are dealing with a cat-like problem with toaster-like thinking.

Here’s my proposal: Fund a Skunkworks to get us out of this mess:

I’ve been developing VA/DoD interfaces since 1985.  They were technically correct, but politically incorrect.  I would hope that in the future, we can get past all the political nonsense of the past few decades, and just settle in to getting the technology working.

1.  Give me a contract to form a skunkworks.  I’ll collect 8-10 of the smartest people I know to develop the simplest solution that is “good enough” to get started.  I’ll also define an approach for “making it better.”

2.  I would like a couple of hospitals to work with (preferably in the San Diego area), at least one VA and one DoD.

3.  Connect me with teams of folks (both in the VA and DoD) who are passionate about making this thing work.  Create a bonus pool (for DoD as well as VA) against which I can make recommendations for bonuses for their contributions to the success of the skunkworks.

4.  Run interference for me to keep the bureaucracy out of the way.  We’ll be doing this in San Diego, far outside the beltway.

This can be a parallel activity to whatever the inside-the-beltway thinkers want to mull over.  Just ramp up a small, talented team working on the problem, independent of what is formally happening.  Think of it as portfolio diversification. The skunkworks funding would be just a fraction of what the IEHR-style aproach has been.

After reading Bob Brewin’s piece Did Jon Stewart Foil the Pentagon’s Health Records Plan? I have decided that Jon Stewart is a worthy recipient of the Unlimited Free Passage on the Underground Railroad certificate.  He understands the problems that the hardhats have been facing in the development of VistA over the years, and also seems to understand the success that it has enjoyed despite the hardships over the years.   I hope that this recognition will help him see some opportunities for improving government, health care, and service to our Veterans.  It is also an amazing story of how a bottom-up, decentralized approach to innovation can work, even in the most hardened bureaucracy.

The VA MUMPS Underground Railroad was formed in the early days of the VistA development in response to the attempts of the centralists to shut down a field-based decentralized approach.  The Hardhats were the technical folk who wrote the code to make it happen, but VistA was always more than just source code, so we needed recognize the many others who were involved in making it a success as described in Phillip Longman’s book Best Care Anywhere and this video. US Medicine editor Nancy Tomich describes the situation.  Nancy and I are now working on the New Health Project to carry things to the next generation.

The Underground Railroad has been struggling to build a common vision of VA/DoD health sharing for decades,   and not without its casualties,  so it is good to finally see some media attention to the issue.

This is the most prestigious award offered by the Underground Railroad, having previously been given in 1982 to Chuck Hagel:

In keeping with the Underground Railroad’s history, his certificate can only be given in person, with appropriate ceremonial presence.

Jon Stewart talked about VistA again on the Daily Show on Apr 4.  He also talked about Chuck Hagel and the original VistA system, which I first noted in this blog entry.

Unfortunately, he missed the whole part about the success of the Underground Railroad, and how it lead to software innovations that triggered improved health care throughout the VA.

Wake Up, Jon….. there’s lot’s of rocks to be thrown at bureaucracies, but here is a great story of something that worked, but it’s being hidden behind what’s failing.  We have a success story to talk about – let’s use that to improve the situation.

I just watched Jon Stewart’s Daily Show for March 27 and was amazed to see VistA and ALHTA mentioned (starting around minute 7), castigating both the VA and the DoD for their “incompatible” medical record systems.  Since this has been my briar patch for 35 years now, it is amazing to see this play out on national TV.

I had a working VA/DoD medical record system working in 1985:

It was thoroughly studied by Congress, GAO, VA, and Arthur D. Little consultants, and passed with flying colors (well, except for some of the dirty tricks that DoD pulled, trying to make it look bad).  Here is an oral history interview about the system.

Continue Reading »

I was delighted to see the recent  article Vets see promise in Hagel and his short VA tenure.  It talked about his support for the VistA EHR system which was my briar patch back in the 1980′s:

“Hagel met with the programmers. “He found out about it and liked it, so he pushed it at the right time,” [Harry] Walters [Former VA Administrator] said. “Now it’s the most effective electronic health-record system in the country.”

The programmers presented Hagel with a certificate of appreciation at a banquet in 1982. “He stuck his neck out,” Munnecke said. “It was a gutsy decision on his part.”

Stay tuned.

And I continue to be amazed at how powerful the Underground Railroad I designed on a lark 30 years ago remains today.

Here is my original post about him.

I am pleased to read that Chuck Hagel has been nominated to the position of Secretary of Defense.  He was the Deputy Director of the VA when I worked for the Loma Linda VA Hospital, working on what would become the VistA Electronic Health Record System, one of the largest and most successful EHRs.  Starting with very humble beginnings as a “skunkworks”, Chuck played a key role in helping to evolve our early back room prototypes into a VA-wide electronic health record that has won many awards and accolades by physicians.

I was part of the small group of programmers hired by Ted O’Neill to develop a decentralized hospital computer system.  This caused huge tensions with the central data processing folks, who wanted to centralize everything in a massive computer in Austin, Texas.

VistA blazed many trails in health IT.  It was the first to integrate SMTP email (I worked directly with internet pioneer Jon Postel, writing one of the first SMTP servers.)  We used what would today would be called Agile Development, starting with a prototype that was “good enough” and getting it into the hands of real users – the more feedback, the better.  We pioneered Open Systems thinking, making our software public domain and collaborating with Indian Health Service, DoD, National Health Service in Finland, and others.  We pioneered social networking/digital conferencing with VA FORUM, which at one time, supported 50,000 VA employees, all learning/teaching about the system, submitting bug reports, and just plain communicating with each other (a rare circumstance in mega-bureaucracies)

The centralists told upper VA management that our decentralized system would never work, but when then chief medical director Don Custis, MD saw the system actually being used, he quipped, “It looks like we have an underground railroad here.”  Nancy Tomich, editor of US Medicine at the time, described this event.

I took this as a sign that we should name our group the Underground Railroad.  I printed business cards, and started holding banquets to honor people who had made major contributions to the effort.  We had two awards: the Joseph T. O’Neill Outstanding Engineering Achievement award for technical folks (who we called Hardhats) and The Unlimited Free Passage on the Underground for the non-hardhats who helped our cause.

Here is a copy of the award I gave Chuck Hagel when he was deputy director of the VA:

Chuck went on to leverage the immense prestige of this award to become a US Senator, and now, nominee for Secretary of Defense.  He probably cherishes this as much as being designated an Admiral of the Great State of Nebraska.

Congressman Sonny Montgomery, then Chairman of the Committee on Veterans Affairs,  praised Chuck’s efforts in this Nov 5, 1984 letter to the Underground Railroad banquet:

Underground Railroad Members and Guests
Blackie’s House of Beef
1717 22nd Street, N. w.
Washington, D. c. 20817
Dear Friends:

On the occasion of your annual Underground Railroad Banquet, may I take this opportunity to send my greeting to you, and to shares ome of my thoughts concerning the Veterans’ Administration Decentralized Hospital Computer Program system with you.

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!
You, as members of the Underground Railroad, can take great pride in your outstanding accomplishment. The task required great dedication and unselfish personal effort. Regrettably, it resulted in some very calamitous casualties along the way. The job is not over, and I know that all of you will continue in your efforts to make this system the finest medical ADP system in the world. I pledge to you that I will continue my unwavering support of your important work and will maintain close oversight of all activities here in Washington and in the field to ensure that our mutual goal is met.

I recall Chuck as being very intelligent and charismatic, and a natural-born leader.  I thank him for his role in shaping the VistA system, and his vision in supporting the unknown “skunkworks” out of which VistA sprang.

And Chuck: if you are reading this, there is no expiration date on this certificate.  If you need the help of the Underground Railroad to help straighten out the VA/DoD EHR mess, we’re ready to help.  I figure we could save the DoD $10 billion or so.  And if the code we wrote back then might have aged a bit, I think that the principles we espoused are even more current in today’s federal health IT environment.

And an Open Source VistA community is alive and thriving.  The VistA Community Meeting is happening this month in California, and  OSEHRA (Open Source Electronic Health Record Agent) is actively supporting open source VistA.

Ever since my early days in the VA, I’ve hosted VA MUMPS Underground Railroad banquets to recognize people who have participated in helping to make the VA VistA Electronic Health Record system work. Here are some videos from the June 5, 2012 banquet held at George Mason University.

The name comes from a meeting that Donald Custis, then Head of VA’s Department of Medicine and Surgery, first saw the VistA system work. He had been told by the centralists that it was impossible to put an health information system on minicomputers, but when he saw it, he quipped, “Looks like we have an underground railroad here.” I took this as the name of our group, and had membership cards printed up, and started passing out awards for “Unlimited Free Passage on the Underground Railroad” and “Outstanding Engineering Achievement on the Underground Railroad.”

Here are my opening comments:

Here are the comments of Dr. Ross Fletcher, Chief of Staff of the Washington VA Medical Center. Ross has been a poster boy for clinician/programmer interaction in software development, always happy to give a physician’s eye view on how software should work. Having such close interaction between doctors and programmers was a key factor for the success of VistA. In this presentation, he demonstrates iPad apps to access VistA databases:

Here are the comments of Joseph Dal Molin, one of the founders of the World VistA community:

And here is a copy of a 1984 letter from Rep Sonny Montgomery, reinforcing the importance of the Underground Railroad: 1984 nov 5 montgomery letter to Underground Railroad

From the initial 1978 Oklahoma City VA/DoD/IHS meeting, we had envisioned a government-wide health information system, based on shared, open source software.  Things went well for the VA, and we deployed a nation-wide system starting in 1983.  Things looked bright for a while for DoD use, as we installed a shared system at March AFB in Riverside, CA. connected to Loma Linda VA hospital.  Congressman Sonny Montgomery got wind of the effort, and supported it to the hilt.  DoD didn’t like the idea – spending more money on consultants to make it look bad than they did to try out the system to see if it worked.

Here are some papers I’ve scanned in that relate to some of the early history of VA/DoD sharing.

Despite these being nearly 30 years old, the issues they talk about are pretty much current – just part of a never-ending story about VA-DoD integration efforts.

1985 munnecke overview of DHCP to TRIMIS Program Office

1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS

1984 oct 4 montgomery letter to weinberger re DoD use of VA software

1984 nov 5 montgomery letter to Underground Railroad

1986 Anon letter to DOD Inspector General re alleged conflicts of interest in CHCS

1985 first VA DoD email message exchange at March AFB

1984 MITRE report on Utilization of VA software in the TRIMIS program

1984 Octo Barnett responds to MITRE report on DoD methodology

1985 munnecke email re ADL dirty tricks

1997 US Medicine article by tom From DHCP to Vision for Change

1978 Tom Munnecke’s Original DHCP FileMan and Kernel design notes at OK City kickoff meeting

Epic Systems and Judy Faulkner made the NY Times today, describing their rise over 30 years to be one of the leading EMR vendors.  She came out of the same era as the VA VistA world did; the major difference being that the VistA crowd followed an open source, public domain model, while Epic is one of the most closed, expensive systems on the market.

The President’s Council of Advisors on Science and Technology (PCAST)  issued a report complimenting both Epic and VistA as successful examples of large scale electronic health record systems, not recognizing that they were architectural first cousins.  Both are based on the MUMPS language, and both use a meta-data driven approach (VistA uses FileMan, Epic uses Chronicles).  The report also called for a universal exchange language, not realizing that this was essentially MUMPS is – the result of decades of support from NIH, National Bureau of Standards, NLM, and the VA.

Both systems are recognized by their integrated nature – I used to say that VistA was integrated by virtue of not disintegrating.

It is interesting to contrast the decades-long success of VistA and Epic – based on a unified metatdata system – vs other attempts at integrating “best of breed” systems, such as the fiascos of the National Health Service ($17b)  Ontario ($1b), and AHLTA ($4b).

VistA is free and open source software – Here a complete stack of software that can be loaded and run on any Intel box.  It doesn’t come with a treehouse and slide, unfortunately.  For that, you’ll have to pay milllions to Epic for pretty much the same functionality.