Archive for the 'VistA' Category

Apr 17 2013

Chuck Hagel’s Assessment of IEHR: “I didn’t think we knew what the hell we were doing.”

Published by under AHLTA,VistA

Secretary of Defense Chuck Hagel testified before a Congressional hearing yesterday about the Integrated Electronic Health Record project: “I didn’t think we knew what the hell we were doing.” I’m glad that he put the stop to the effort after only $1 billion, the UK National Health Service blew an incredible $17 billion before pulling the plug.

This ratchet ups the rhetoric of Assistant Secretary of Defense for Health Affairs’ Ward Cascells’ 2009 assesment that DoD’s AHLTA system is “Intolerable”

Not to kick a dead horse, but this has been going on for nearly 40 years now.  The DoD had spent $250 million prototyping the TRIMIS system, a collection of incompatible demonstration systems, while we at the VA were delivering a working, integrated hospital information system (DHCP) for 172 hospitals for $82 million.  House Veterans Affairs Committee Chairman Sonny Montgomery hit the roof over this, and partially triggered by the successful VA/DoD integration prototype declared that one of the competitors for the CHCS system would propose an adaptation of DHCP (now called VistA).  I went to SAIC to help with the effort, and we ended up delivering a very successful CHCS system, which is still the core of the DoD EHR system today.

The fundamental problem with the DoD is that they do not understand how to deal with systems of the complexity of a modern health care system.  They understand how to build an aircraft carrier, make sure that the troops have the supplies they need, and other activities from a linear perspective: the whole is equal to the sum of the parts.  Break the carrier into pieces, design all the pieces, and put them back together again to make a whole carrier.  This is (kindof) well and good for things that have this whole-equal-sum-of-parts quality.  Toasters can be taken apart and put back together again, and will still be the same toaster.

But health care is far more complex and dynamic than an aircraft carrier.  The hospital, Peter Drucker said, is the most complex organization in modern society.  Like a cat, we cannot dissect a hospital and put it back together again.

In a lesson straight from Humpty Dumpty, DoD wanted to break the electronic health record system into 17 “best of breed” applications, then hire a “systems integrator” to put Humpty together again.  This is like trying to build the world’s best car by trying to integrate the engine from a Corvette with the seats from a Rolls Royce and the chassis from a Porsche.  But despite how lucrative it is to be one of “all the king’s men,” it is simply not going to happen.

Yesterday, a friend of mine with many decades in the health IT industry called to tell me that he had just signed up with the VA in San Diego.  He said he was amazed at how well coordinated his care was – and this is from someone trying to do this in the private sector for 30 years.  This is the result of a fundamental approach taken from the earliest days of VistA – we were “integrated” by virtue of the fact that we never “disintegrated” into pieces.  We build a tool kit from which we composed the system over time, instead of the DoD’s approach of decomposing the system into pieces and then trying to put them back together again.  In that sense, VistA is remarkably similar to Wikipedia in this sense.  (See my discussion with Wiki inventor Ward Cunningham on this topic.)

So the fundamental issue is that we are dealing with a cat-like problem with toaster-like thinking.

Here’s my proposal: Fund a Skunkworks to get us out of this mess:

I’ve been developing VA/DoD interfaces since 1985.  They were technically correct, but politically incorrect.  I would hope that in the future, we can get past all the political nonsense of the past few decades, and just settle in to getting the technology working.

1.  Give me a contract to form a skunkworks.  I’ll collect 8-10 of the smartest people I know to develop the simplest solution that is “good enough” to get started.  I’ll also define an approach for “making it better.”

2.  I would like a couple of hospitals to work with (preferably in the San Diego area), at least one VA and one DoD.

3.  Connect me with teams of folks (both in the VA and DoD) who are passionate about making this thing work.  Create a bonus pool (for DoD as well as VA) against which I can make recommendations for bonuses for their contributions to the success of the skunkworks.

4.  Run interference for me to keep the bureaucracy out of the way.  We’ll be doing this in San Diego, far outside the beltway.

This can be a parallel activity to whatever the inside-the-beltway thinkers want to mull over.  Just ramp up a small, talented team working on the problem, independent of what is formally happening.  Think of it as portfolio diversification. The skunkworks funding would be just a fraction of what the IEHR-style aproach has been.








Comments Off on Chuck Hagel’s Assessment of IEHR: “I didn’t think we knew what the hell we were doing.”

Apr 11 2013

Jon Stewart offered Prestigious Unlimited Free Passage on Underground Railroad award


After reading Bob Brewin’s piece Did Jon Stewart Foil the Pentagon’s Health Records Plan? I have decided that Jon Stewart is a worthy recipient of the Unlimited Free Passage on the Underground Railroad certificate.  He understands the problems that the hardhats have been facing in the development of VistA over the years, and also seems to understand the success that it has enjoyed despite the hardships over the years.   I hope that this recognition will help him see some opportunities for improving government, health care, and service to our Veterans.  It is also an amazing story of how a bottom-up, decentralized approach to innovation can work, even in the most hardened bureaucracy.

The VA MUMPS Underground Railroad was formed in the early days of the VistA development in response to the attempts of the centralists to shut down a field-based decentralized approach.  The Hardhats were the technical folk who wrote the code to make it happen, but VistA was always more than just source code, so we needed recognize the many others who were involved in making it a success as described in Phillip Longman’s book Best Care Anywhere and this video. US Medicine editor Nancy Tomich describes the situation.  Nancy and I are now working on the New Health Project to carry things to the next generation.

The Underground Railroad has been struggling to build a common vision of VA/DoD health sharing for decades,   and not without its casualties,  so it is good to finally see some media attention to the issue.

Jon Stewart

This is the most prestigious award offered by the Underground Railroad, having previously been given in 1982 to Chuck Hagel:

Chuck Hagel UFP

In keeping with the Underground Railroad’s history, his certificate can only be given in person, with appropriate ceremonial presence.


Comments Off on Jon Stewart offered Prestigious Unlimited Free Passage on Underground Railroad award

Apr 08 2013

VistA and Chuck Hagel on Daily Show again

Published by under AHLTA,VistA

Jon Stewart talked about VistA again on the Daily Show on Apr 4.  He also talked about Chuck Hagel and the original VistA system, which I first noted in this blog entry.

Unfortunately, he missed the whole part about the success of the Underground Railroad, and how it lead to software innovations that triggered improved health care throughout the VA.

Wake Up, Jon….. there’s lot’s of rocks to be thrown at bureaucracies, but here is a great story of something that worked, but it’s being hidden behind what’s failing.  We have a success story to talk about – let’s use that to improve the situation.



Comments Off on VistA and Chuck Hagel on Daily Show again

Mar 28 2013

VistA and AHLTA on the Daily Show

Published by under AHLTA,VistA

I just watched Jon Stewart’s Daily Show for March 27 and was amazed to see VistA and ALHTA mentioned (starting around minute 7), castigating both the VA and the DoD for their “incompatible” medical record systems.  Since this has been my briar patch for 35 years now, it is amazing to see this play out on national TV.

I had a working VA/DoD medical record system working in 1985:
Tom Munnecke, Ingeborg Kuhn, George Boyden, Beth Teeple showing off the first VA/DoD Health IT interface

It was thoroughly studied by Congress, GAO, VA, and Arthur D. Little consultants, and passed with flying colors (well, except for some of the dirty tricks that DoD pulled, trying to make it look bad).  Here is an oral history interview about the system.

Continue Reading »


Comments Off on VistA and AHLTA on the Daily Show

Feb 07 2013

Washington Post article on Chuck Hagel and the VA

I was delighted to see the recent  article Vets see promise in Hagel and his short VA tenure.  It talked about his support for the VistA EHR system which was my briar patch back in the 1980’s:

“Hagel met with the programmers. “He found out about it and liked it, so he pushed it at the right time,” [Harry] Walters [Former VA Administrator] said. “Now it’s the most effective electronic health-record system in the country.”

The programmers presented Hagel with a certificate of appreciation at a banquet in 1982. “He stuck his neck out,” Munnecke said. “It was a gutsy decision on his part.”

Stay tuned.

And I continue to be amazed at how powerful the Underground Railroad I designed on a lark 30 years ago remains today.

Here is my original post about him.


Comments Off on Washington Post article on Chuck Hagel and the VA

Jan 21 2013

VistA Named one of top 10, oldest, most significant Open Source Projects

Published by under Underground Railroad,VistA

I just noticed that the VA VistA Electronic Health Record System was named as one of The 10 oldest, significant open-source programs in the company of Linux, Python, Perl, and other software.   Back in the early days of VistA, the term “Open Source” didn’t exist – we called it “public domain.”

What makes VistA unique in this list is that it is more than just code, it is a longitudinal data base extending back over 30 years across 172 hospitals and millions of patients.  This is an irreplaceable resource – VistA has petabytes of information about clinical conditions, activities, locations, and demographics from a period when most other hospitals were just dreaming about putting their records online.  This information is mapped by a “data dictionary” – a “road map” to the data base that defines the semantics (what it means) of the data, not just the syntax (how it is formatted).  It is also structured around a network model of information (the connectors between the dots), rather than just a hierarchical model (think of how the Dewey Decimal System tries to form a hierarchy of the books in a library).

This data is not complete, perfectly formatted, nor coordinated with precision.  Clinical data is not research-grade clinical trials data, and the patient population is that of (mostly male) veterans.  But it is a treasure trove of information, waiting for future data mining.  And what we consider “junk” information today (say, appointment schedules from 30 years ago), may hold great value in future research (say, how epidemics spread).

I hope the VA has the wisdom to retain this information in its original form.

At the 2012 OSEHRA summit meeting, I happened to be on a panel with medical researcher Leonard D’Avolio who commented on how easy it was to extract research data from VistA.  It was quite a touching moment for me, to have helped set in motion something that captured data that would be available 33 years later.

To quote my friend Doc Searls: “When young you think life is a sprint. When older you see it’s a marathon. And when mature you see it’s a relay race.”

The question, just who are we handing off the baton to?  Is the medical information we’re dealing with today going to be available 33 years from now?

I see a number of necessary conditions for this to happen:

1.  We need a curator.  Some organization needs to take on the role of “librarian” to maintain the archives.  The VA is a likely candidate for this, but even so, I think they might need a bit of prodding to keep them focused on their responsibility.  And maybe if Chuck Hagel (one of the Fathers of VistA) is confirmed as Secretary of Defense, he’ll recall his roots as one of the fathers of VistA and bring DoD medical information along, as well.  Roger Baker, Assistant Secretary for Information and Technology for the Department of Veterans Affairs, is moving things along so well that he earned a VIP membership card in the Underground Railroad.

2. We need to manage the data at a “meta” level.  The VistA Data Dictionary is being updated to more modern semantic web concepts, for example through Conor Dowling’s Semantic VistA.  The PCAST report called for greater use of Metadata, apparently unaware that they were recommending an approach already used by VistA for 1/3 century, which they cited as a case study in a successful large-scale system.  This is a bit like Monsieur Jourdain in Moliere’s play Bourgeois Gentleman, who suddenly realized that he had been speaking prose all his life, but didn’t know it.

3.  We need to continue the tradition of Open Source.  There is some movement in this direction with the OSEHRA effort, but it’s funding is but a tiny fraction of what it needs to be.

4.  We need to fund the effort.  Some folks think that Open Source means “free labor” – that the VA can just declare VistA to be an open source system and programmers will flock to it, offering volunteer time to do the VA’s bidding.  This isn’t going to work.  VA needs to fund these efforts, and release the work as open source.

5.  We need to think of what we are building as an “information space” rather than just an “integrated electronic health record.”  If someone said that they were going to improve our legal system by systematically storing and arranging lawyer’s documents, most of us would be a little skeptical.  But there is a huge chasm between our goal of  improving our health care and the current task of automating the medical record.  As we’ve seen with Twitter, Facebook, Google, Linked In, and Wikipedia, there are huge benefits to focusing on “connecting the dots” in large scale systems, rather than just focusing on specific dots.  Here’s a webinar I gave at Kitware, a leading open-source company.

All in all, though, it’s pretty cool to see VistA placed in this company.






Comments Off on VistA Named one of top 10, oldest, most significant Open Source Projects

Jan 06 2013

Chuck Hagel – One of the Fathers of VistA

220px-Chuck_Hagel_official_photoI am pleased to read that Chuck Hagel has been nominated to the position of Secretary of Defense.  He was the Deputy Director of the VA when I worked for the Loma Linda VA Hospital, working on what would become the VistA Electronic Health Record System, one of the largest and most successful EHRs.  Starting with very humble beginnings as a “skunkworks”, Chuck played a key role in helping to evolve our early back room prototypes into a VA-wide electronic health record that has won many awards and accolades by physicians.

I was part of the small group of programmers hired by Ted O’Neill to develop a decentralized hospital computer system.  This caused huge tensions with the central data processing folks, who wanted to centralize everything in a massive computer in Austin, Texas.

VistA blazed many trails in health IT.  It was the first to integrate SMTP email (I worked directly with internet pioneer Jon Postel, writing one of the first SMTP servers.)  We used what would today would be called Agile Development, starting with a prototype that was “good enough” and getting it into the hands of real users – the more feedback, the better.  We pioneered Open Systems thinking, making our software public domain and collaborating with Indian Health Service, DoD, National Health Service in Finland, and others.  We pioneered social networking/digital conferencing with VA FORUM, which at one time, supported 50,000 VA employees, all learning/teaching about the system, submitting bug reports, and just plain communicating with each other (a rare circumstance in mega-bureaucracies)

The centralists told upper VA management that our decentralized system would never work, but when then chief medical director Don Custis, MD saw the system actually being used, he quipped, “It looks like we have an underground railroad here.”  Nancy Tomich, editor of US Medicine at the time, described this event.

I took this as a sign that we should name our group the Underground Railroad.  I printed business cards, and started holding banquets to honor people who had made major contributions to the effort.  We had two awards: the Joseph T. O’Neill Outstanding Engineering Achievement award for technical folks (who we called Hardhats) and The Unlimited Free Passage on the Underground for the non-hardhats who helped our cause.

Here is a copy of the award I gave Chuck Hagel when he was deputy director of the VA:

Chuck Hagel UFP

Chuck went on to leverage the immense prestige of this award to become a US Senator, and now, nominee for Secretary of Defense.  He probably cherishes this as much as being designated an Admiral of the Great State of Nebraska.

Congressman Sonny Montgomery, then Chairman of the Committee on Veterans Affairs,  praised Chuck’s efforts in this Nov 5, 1984 letter to the Underground Railroad banquet:

Underground Railroad Members and Guests
Blackie’s House of Beef
1717 22nd Street, N. w.
Washington, D. c. 20817
Dear Friends:

On the occasion of your annual Underground Railroad Banquet, may I take this opportunity to send my greeting to you, and to shares ome of my thoughts concerning the Veterans’ Administration Decentralized Hospital Computer Program system with you.

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!
You, as members of the Underground Railroad, can take great pride in your outstanding accomplishment. The task required great dedication and unselfish personal effort. Regrettably, it resulted in some very calamitous casualties along the way. The job is not over, and I know that all of you will continue in your efforts to make this system the finest medical ADP system in the world. I pledge to you that I will continue my unwavering support of your important work and will maintain close oversight of all activities here in Washington and in the field to ensure that our mutual goal is met.

I recall Chuck as being very intelligent and charismatic, and a natural-born leader.  I thank him for his role in shaping the VistA system, and his vision in supporting the unknown “skunkworks” out of which VistA sprang.

And Chuck: if you are reading this, there is no expiration date on this certificate.  If you need the help of the Underground Railroad to help straighten out the VA/DoD EHR mess, we’re ready to help.  I figure we could save the DoD $10 billion or so.  And if the code we wrote back then might have aged a bit, I think that the principles we espoused are even more current in today’s federal health IT environment.

And an Open Source VistA community is alive and thriving.  The VistA Community Meeting is happening this month in California, and  OSEHRA (Open Source Electronic Health Record Agent) is actively supporting open source VistA.




Comments Off on Chuck Hagel – One of the Fathers of VistA

Jun 23 2012

Some old documents relating to VA/DoD Sharing

Published by under VistA

Here are my notes for a 1985  presentation to the Department of Defense TRIMIS Project Office, discussing the state of VistA (then called DHCP) and the potential for sharing.  It didn’t seem to make much difference to the TRIMIS folks, however.

The National Association of VA Physicians (NAVAP) was a big supporter of VistA.  Here is a paper I delivered in 1985 Munnecke Occams Razor alive and well into VA

Here is an email complaining that DoD and their consultants were not looking at the big picture when evaluating DHCP…  1986 munnecke midnight ramblings re March test methodologies

It was actually much worse than this.  DoD had hired Arthur D. Little consultants to study the prototype.  Dr. John Glaser was the project lead, who went on to become CEO of Siemens Health Informatics.  I ran into John at HIMSS this year, and talked to him about the project.  He said he was in a difficult situation, because he was hired with instructions to “Make DHCP look bad.”  But when he got out to see the system, he said, “It didn’t look so bad to me.”

1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS

1984 oct 4 montgomery letter to weinberger re DoD use of VA software

1984 nov 5 montgomery letter to Underground Railroad



Comments Off on Some old documents relating to VA/DoD Sharing

Jun 18 2012

“One Piece at a Time” – next generation federal health IT architecture

Published by under VistA

The details for the Integrated Electronic Health Record (IEHR) are just now beginning to roll out.   It’s pretty much a replay of the “Best of Breed” marketing approach that I’ve seen been pitched for decades.  Basically, collect all the parts and the whole will fall together with “just a bit of integration.”  They are issuing dozens of RFPs and RFIs describing all the pieces they seek; how it all fits together is a lot of hand-waving at the moment, with an inordinate faith in the role of an Enterprise Service Bus to magically make everything work together.

This is a little like someone selling you the world’s best car, based on “best of breed” components from all the best manufactures.   An engine from a Corvette, seats from a Rolls Royce, transmission from a Ferrari, etc. To make it all fit together, they would say it is all “standardized” – every part would use metric nuts and bolts, 12 volts electricity, and a common electrical harness (equivalent of the Enterprise Service Bus), to make sure that all the parts could work together.

We took an opposite approach in VistA, starting with a conceptual model of a system driven by active metadata – a common roadmap to patient database.  We were “integrated” by virtue of not “disintegrating” into pieces in the first place.  We had an overarching Conceptual Integrity to the design that gave us a common foundation from which to grow, climbing up the ladder of abstraction rather than falling down it.

This turns out to have been a good decision.  The President’s Council of Advisors for Science and Technology (PCAST) issued a report the report—Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward— that specifically pointed out VistA as one of the success stories for large scale EHR systems.  It also advocated greater use of metadata: tagging data dynamically through meta rather than “pigeonholing” it into pre-defined, previously synchronized slots.

This argument can be very abstract to some people, particularly to bureaucrats whose entire life has been engulfed by hierarchies and “pigeonhole” systems thinking.  The notion that systems can associate dynamically, spanning hierarchies (or not even being associated with a hierarchy) casts them into unfamiliar uncharted territory.  To them, complex systems require complex organizations to manage and regulate them. They seek efficiency, rather than resilience, adaptability, or robustness.

In order to make this a bit more understandable, I am proposing a theme song for the IEHR architecture: Johnny Cash’s One Piece at a Time.  Think of the IEHR vision as Johnny Cash’s Cadillac.  I’ll let the viewer draw their own analogies.  Of course, Johnny didn’t have billions of taxpayers dollars to spend, nor did he have a gaggle of consultants ready to reap huge contracts to do the “little bit of integration.”


Comments Off on “One Piece at a Time” – next generation federal health IT architecture

Jun 13 2012

Videos from 2012 Underground Railroad Banquet

Ever since my early days in the VA, I’ve hosted VA MUMPS Underground Railroad banquets to recognize people who have participated in helping to make the VA VistA Electronic Health Record system work. Here are some videos from the June 5, 2012 banquet held at George Mason University.

The name comes from a meeting that Donald Custis, then Head of VA’s Department of Medicine and Surgery, first saw the VistA system work. He had been told by the centralists that it was impossible to put an health information system on minicomputers, but when he saw it, he quipped, “Looks like we have an underground railroad here.” I took this as the name of our group, and had membership cards printed up, and started passing out awards for “Unlimited Free Passage on the Underground Railroad” and “Outstanding Engineering Achievement on the Underground Railroad.”

Here are my opening comments:

Here are the comments of Dr. Ross Fletcher, Chief of Staff of the Washington VA Medical Center. Ross has been a poster boy for clinician/programmer interaction in software development, always happy to give a physician’s eye view on how software should work. Having such close interaction between doctors and programmers was a key factor for the success of VistA. In this presentation, he demonstrates iPad apps to access VistA databases:

Here are the comments of Joseph Dal Molin, one of the founders of the World VistA community:

And here is a copy of a 1984 letter from Rep Sonny Montgomery, reinforcing the importance of the Underground Railroad: 1984 nov 5 montgomery letter to Underground Railroad


Comments Off on Videos from 2012 Underground Railroad Banquet

« Prev - Next »

Creative Commons License
Images by Tom Munnecke is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Based on a work at
Permissions beyond the scope of this license may be available at