Archive for the 'AHLTA' Category

Feb 07 2013

Washington Post article on Chuck Hagel and the VA

I was delighted to see the recent  article Vets see promise in Hagel and his short VA tenure.  It talked about his support for the VistA EHR system which was my briar patch back in the 1980’s:

“Hagel met with the programmers. “He found out about it and liked it, so he pushed it at the right time,” [Harry] Walters [Former VA Administrator] said. “Now it’s the most effective electronic health-record system in the country.”

The programmers presented Hagel with a certificate of appreciation at a banquet in 1982. “He stuck his neck out,” Munnecke said. “It was a gutsy decision on his part.”

Stay tuned.

And I continue to be amazed at how powerful the Underground Railroad I designed on a lark 30 years ago remains today.

Here is my original post about him.


Comments Off on Washington Post article on Chuck Hagel and the VA

Jan 06 2013

Chuck Hagel – One of the Fathers of VistA

220px-Chuck_Hagel_official_photoI am pleased to read that Chuck Hagel has been nominated to the position of Secretary of Defense.  He was the Deputy Director of the VA when I worked for the Loma Linda VA Hospital, working on what would become the VistA Electronic Health Record System, one of the largest and most successful EHRs.  Starting with very humble beginnings as a “skunkworks”, Chuck played a key role in helping to evolve our early back room prototypes into a VA-wide electronic health record that has won many awards and accolades by physicians.

I was part of the small group of programmers hired by Ted O’Neill to develop a decentralized hospital computer system.  This caused huge tensions with the central data processing folks, who wanted to centralize everything in a massive computer in Austin, Texas.

VistA blazed many trails in health IT.  It was the first to integrate SMTP email (I worked directly with internet pioneer Jon Postel, writing one of the first SMTP servers.)  We used what would today would be called Agile Development, starting with a prototype that was “good enough” and getting it into the hands of real users – the more feedback, the better.  We pioneered Open Systems thinking, making our software public domain and collaborating with Indian Health Service, DoD, National Health Service in Finland, and others.  We pioneered social networking/digital conferencing with VA FORUM, which at one time, supported 50,000 VA employees, all learning/teaching about the system, submitting bug reports, and just plain communicating with each other (a rare circumstance in mega-bureaucracies)

The centralists told upper VA management that our decentralized system would never work, but when then chief medical director Don Custis, MD saw the system actually being used, he quipped, “It looks like we have an underground railroad here.”  Nancy Tomich, editor of US Medicine at the time, described this event.

I took this as a sign that we should name our group the Underground Railroad.  I printed business cards, and started holding banquets to honor people who had made major contributions to the effort.  We had two awards: the Joseph T. O’Neill Outstanding Engineering Achievement award for technical folks (who we called Hardhats) and The Unlimited Free Passage on the Underground for the non-hardhats who helped our cause.

Here is a copy of the award I gave Chuck Hagel when he was deputy director of the VA:

Chuck Hagel UFP

Chuck went on to leverage the immense prestige of this award to become a US Senator, and now, nominee for Secretary of Defense.  He probably cherishes this as much as being designated an Admiral of the Great State of Nebraska.

Congressman Sonny Montgomery, then Chairman of the Committee on Veterans Affairs,  praised Chuck’s efforts in this Nov 5, 1984 letter to the Underground Railroad banquet:

Underground Railroad Members and Guests
Blackie’s House of Beef
1717 22nd Street, N. w.
Washington, D. c. 20817
Dear Friends:

On the occasion of your annual Underground Railroad Banquet, may I take this opportunity to send my greeting to you, and to shares ome of my thoughts concerning the Veterans’ Administration Decentralized Hospital Computer Program system with you.

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!
You, as members of the Underground Railroad, can take great pride in your outstanding accomplishment. The task required great dedication and unselfish personal effort. Regrettably, it resulted in some very calamitous casualties along the way. The job is not over, and I know that all of you will continue in your efforts to make this system the finest medical ADP system in the world. I pledge to you that I will continue my unwavering support of your important work and will maintain close oversight of all activities here in Washington and in the field to ensure that our mutual goal is met.

I recall Chuck as being very intelligent and charismatic, and a natural-born leader.  I thank him for his role in shaping the VistA system, and his vision in supporting the unknown “skunkworks” out of which VistA sprang.

And Chuck: if you are reading this, there is no expiration date on this certificate.  If you need the help of the Underground Railroad to help straighten out the VA/DoD EHR mess, we’re ready to help.  I figure we could save the DoD $10 billion or so.  And if the code we wrote back then might have aged a bit, I think that the principles we espoused are even more current in today’s federal health IT environment.

And an Open Source VistA community is alive and thriving.  The VistA Community Meeting is happening this month in California, and  OSEHRA (Open Source Electronic Health Record Agent) is actively supporting open source VistA.




Comments Off on Chuck Hagel – One of the Fathers of VistA

Jun 13 2012

Videos from 2012 Underground Railroad Banquet

Ever since my early days in the VA, I’ve hosted VA MUMPS Underground Railroad banquets to recognize people who have participated in helping to make the VA VistA Electronic Health Record system work. Here are some videos from the June 5, 2012 banquet held at George Mason University.

The name comes from a meeting that Donald Custis, then Head of VA’s Department of Medicine and Surgery, first saw the VistA system work. He had been told by the centralists that it was impossible to put an health information system on minicomputers, but when he saw it, he quipped, “Looks like we have an underground railroad here.” I took this as the name of our group, and had membership cards printed up, and started passing out awards for “Unlimited Free Passage on the Underground Railroad” and “Outstanding Engineering Achievement on the Underground Railroad.”

Here are my opening comments:

Here are the comments of Dr. Ross Fletcher, Chief of Staff of the Washington VA Medical Center. Ross has been a poster boy for clinician/programmer interaction in software development, always happy to give a physician’s eye view on how software should work. Having such close interaction between doctors and programmers was a key factor for the success of VistA. In this presentation, he demonstrates iPad apps to access VistA databases:

Here are the comments of Joseph Dal Molin, one of the founders of the World VistA community:

And here is a copy of a 1984 letter from Rep Sonny Montgomery, reinforcing the importance of the Underground Railroad: 1984 nov 5 montgomery letter to Underground Railroad


Comments Off on Videos from 2012 Underground Railroad Banquet

Jun 02 2012

Some historical VA/DoD papers

From the initial 1978 Oklahoma City VA/DoD/IHS meeting, we had envisioned a government-wide health information system, based on shared, open source software.  Things went well for the VA, and we deployed a nation-wide system starting in 1983.  Things looked bright for a while for DoD use, as we installed a shared system at March AFB in Riverside, CA. connected to Loma Linda VA hospital.  Congressman Sonny Montgomery got wind of the effort, and supported it to the hilt.  DoD didn’t like the idea – spending more money on consultants to make it look bad than they did to try out the system to see if it worked.

Here are some papers I’ve scanned in that relate to some of the early history of VA/DoD sharing.

Despite these being nearly 30 years old, the issues they talk about are pretty much current – just part of a never-ending story about VA-DoD integration efforts.


1985 munnecke overview of DHCP to TRIMIS Program Office

1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS

1984 oct 4 montgomery letter to weinberger re DoD use of VA software

1984 nov 5 montgomery letter to Underground Railroad

1986 Anon letter to DOD Inspector General re alleged conflicts of interest in CHCS

1985 first VA DoD email message exchange at March AFB

1984 MITRE report on Utilization of VA software in the TRIMIS program

1984 Octo Barnett responds to MITRE report on DoD methodology

1985 munnecke email re ADL dirty tricks

1997 US Medicine article by tom From DHCP to Vision for Change

1978 Tom Munnecke’s Original DHCP FileMan and Kernel design notes at OK City kickoff meeting

1985 Munnecke Occams Razor alive and well into VA

Comments Off on Some historical VA/DoD papers

Jan 15 2012

Epic Systems in the News

Published by under AHLTA,Heath IT,VistA

Epic Systems and Judy Faulkner made the NY Times today, describing their rise over 30 years to be one of the leading EMR vendors.  She came out of the same era as the VA VistA world did; the major difference being that the VistA crowd followed an open source, public domain model, while Epic is one of the most closed, expensive systems on the market.

The President’s Council of Advisors on Science and Technology (PCAST)  issued a report complimenting both Epic and VistA as successful examples of large scale electronic health record systems, not recognizing that they were architectural first cousins.  Both are based on the MUMPS language, and both use a meta-data driven approach (VistA uses FileMan, Epic uses Chronicles).  The report also called for a universal exchange language, not realizing that this was essentially MUMPS is – the result of decades of support from NIH, National Bureau of Standards, NLM, and the VA.

Both systems are recognized by their integrated nature – I used to say that VistA was integrated by virtue of not disintegrating.

It is interesting to contrast the decades-long success of VistA and Epic – based on a unified metatdata system – vs other attempts at integrating “best of breed” systems, such as the fiascos of the National Health Service ($17b)  Ontario ($1b), and AHLTA ($4b).

VistA is free and open source software – Here a complete stack of software that can be loaded and run on any Intel box.  It doesn’t come with a treehouse and slide, unfortunately.  For that, you’ll have to pay milllions to Epic for pretty much the same functionality.




Comments Off on Epic Systems in the News

Oct 12 2011

Lessons learned from RIM failure in Europe

RIM Blackberry service crashed in Europe and around the world this week:  Global BlackBerry Outages Due To European Backup Failure, supposedly due to the failure of a central switch.

This is a great lesson to be learned in network design.  The Internet was designed NOT to have a single point of failure.  Each packet is routed independently.  If there is a failure, the network automatically “routes around the damage.”  If you are sending information from San Diego to New York, and Dallas goes now, the packet might just go through Chicago (or around the world the other way).

RIM’s network, on the other hand, was completely dependent on a single switch, apparently with a backup switch.  One failed, and then the other, then the whole network crashed.  This might have seemed to be cheaper, but it certainly wasn’t resilient.

Lesson learned: design networks to be resilient and adaptive, not brittle and based on single point of failures.

My first reaction to seeing the disastrous AHLTA system architecture was that DoD was creating a giant single point of failure.



Comments Off on Lessons learned from RIM failure in Europe

May 31 2011

AHLTA Is Not Alone, Part II: Ontario’s $1 Billion health IT fiasco

Published by under AHLTA,Heath IT,VistA

It really pains me to post this kind of stuff, but I fear that we are entering an era of unprecedented Health IT fiascos.  After the $4 billion Department of Defense AHLTA fiasco, (AHLTA is Intolerable), and the $17 billion UK National Health Service fiasco, here is a A scathing report on the eHealth Ontario spending scandal charges that successive governments wasted $1 billion in taxpayer money. Carpet bagger consultants have arrived in full force, with flimsy-if-no experience in the complexities of medical informatics, but a deep, abiding concern for their own income.

The head of the Ontario disaster “billed thousands of dollars for limousine rides… before she resigned from her $380,000-a-year job in June. She was given a $317,000 severance package and received a $114,000 bonus after just 10 months on the job.”  This payment was made for leading the fiasco, not delivering a product.  The Canadian auditor found the agency had fewer than 30 full-time employees but was engaging more than 300 consultants. they also said the effort was “lacking in strategic direction and relying too heavily on external consultations.”

The idea behind eHealth is to create electronic health records for Ontario, something the auditor says could save $6 billion if implemented in every province and territory.  Instead, Ontario “is near the back of the pack” when it comes to electronic health records, having wasted millions on underused computer systems and untendered contracts.

EHealth was set up in 2008 to create electronic health records after Smart Systems for Health spent $650 million but failed to produce anything of lasting value. Smart Systems for Health was quietly shut down last September.

And then there is politics: “the Conservatives and New Democrats complained the agency gave out $5 million in untendered contracts to consultants.   Documents released by the government since then showed the value of those untendered contracts was closer to $16 million, with the biggest ones going to companies the opposition parties say have ties to the Liberal government.”

These kinds of shenanigans are not unique to Canada, of course.  Here is a letter from a group of Wisconsin congressfolk blatantly meddling in VA and DoD internal IT decisions:
Wisconsin reps try to derail VA/Defense open source health records system which happens to be the home of Epic Systems, who hope to provide one of the most proprietary, closed systems in the nation.  Epic denied hiring lobbyists, but a quick search on showed over $1m in political donations from Epic CEO Judy Faulkner and Epic, much directed to the very Congressfolks who took the time to write the letter.  Epic don’t lobby; I guess it’s more efficient just to send cash to their Congressfolks.

This isn’t just about money… Bad health IT software kills people.  When I worked at the VA Loma Linda, I routinely rode in the staff elevator with the gurney of a patient headed towards the morgue.  I would be in meetings when a code blue sounded, and docs would rush out of the room, only to come back breathless, saying, “now, where were we?” This was a sobering and constant reminder of the significance of what hospitals do.  The wrong information, or delays in getting the right information, can kill.  Installing patient software (e.g. the VA’s bar coding of patients for medications) can save lives.

At the core of this – around the world, not just the US – are self-righteous bureaucrats who persist in top-down, centralized, waterfall development, “one-correct-way” thinkers who see their bureaucracy, not the patient, as the center of the IT universe.  We are not creating an “accounting system for disease” as if it is a banking transaction system.  We need to building communities of health, with the individual at the center of the world.  We need to overcome the “failure to communicate” in health care; the Electronic Health Record is but one aspect of a much broader need for transformation in our health care thinking.

I don’t see this shift happening.  Instead, I see self-righteous indignation on the part of the bureaucrats who perpetuated the fiasco in the first place.  It’s as if a carpenter builds a crooked house, and blames his hammer.  All he needs is a another, more expensive hammer, and he’ll build a house twice as expensive.  The US Department of Defense is now on its fourth generation of crooked houses/more expensive hammers, and is in the process of repeating the process all over again, full of self-righteous indignation that VA’s VistA is continuing to evolve while their break-and-replace models continue to fail.  DoD continues to blame “the software” for their failures, oblivious to the fact that they were the ones who perpetrated the fiasco in the first place.  Governments continue to pour money onto the fire, paying for new hammers even when bureaucratic incompetence has been repeatedly demonstrated.

My current Health IT fiasco report card:  I hope this is the extent of the list, but I fear that there are many others to be listed.  Please email me with any links.

UK National Health Service:  $17 Billion

US Department of Defense AHLTA: $4 Billion

Canada: Ontario eHealth System: $1 Billion

(tip of the hat to Joseph Dal Molin for some of these links)



Comments Off on AHLTA Is Not Alone, Part II: Ontario’s $1 Billion health IT fiasco

May 28 2011

AHLTA is Not Alone – UK NHS IT fiasco tops it as $17 billion fiasco

Published by under AHLTA,Heath IT,VistA

I had thought that DoD’s AHLTA held the record for being the world’s greatest health IT fiasco, spending $4-5 billion of taxpayer’s money on a system that was deemed Intolerable by the GAO and an Assistant Secretary of Defense.  It was so bad that it is cited as one of the leading reasons physicians are leaving military service.  One user told me, “The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.”

I knew that AHLTA would be a failure the instant I saw a diagram of it – a giant, centralized single point of failure that ignored everything that I had found successful in doing the VistA architecture.

This is not just dollars we are talking about, or missed opportunity costs.  Bad software kills people.  I don’t know if we’ll ever know how many patients AHLTA has killed, but it has to be significant.

But enough about AHLTA: I just read this article NHS IT system condemned about the UK National Health Service is in the midst of a far greater fiasco:

In a jaw-dropping condemnation of the NHS National Programme for IT, the National Audit Office has exposed a white elephant in the final stages of collapse.

In what read as a final pronouncement, the NAO reported that after nine years and £2.7bn, (US $4.5 billion)  the NHS has failed to deliver its primary aim of an electronic care record for everyone in the country.

The situation looks so dire the system might continue leeching money from the NHS for another decade if the whole scheme and all its software is not seized by state liquidators.

They go on to say that they don’t think its worth going forward with the remaining US$7.7 billion:

The NAO (National Audit Agency) said CSC would likely fail to deliver the rest before its contract runs out in 2016. DoH had been in dispute over its contract with CSC for 18 months, trying to claw back some of the £5bn (US $8.2b) it had promised the supplier.

What is particularly galling about this situation is that both the US DoD and the UK NHS had access to an award-winning, open source hospital information system that has been running at this scale for 25 years… the VA’s VistA system.  As one of the original software architects of this system, I have seen an endless stream of novice health IT folks appear on the scene, thinking that because they know some other aspect of IT that they can apply this to health IT.  In the past, these folks would slink away after losing $100m or so.  But now, it seems the stakes have been raised to the tens of billions.

We took an innovative approach in designing VistA, and it worked…  So, why are people continuing to pour money down the same failed approaches?

The answer is to follow the money:  A $5 billion fiasco is extremely lucrative to the beltway bandits.  Sure, they have to weather a few editorials and a couple of roastings before the various legislators.  But this can all be fixed with a little lobbying effort.  They’ll apologize, promise never to do it again, change the name of the project, and double the price tag of the next round of funding.  And laugh all the way to the bank with the profits they made on the fiasco.  It’s likely that a successful implementation would have probably earned them less profit.

Tthis isn’t just about an atrocious waste of taxpayer’s money.  People die from bad software.



Comments Off on AHLTA is Not Alone – UK NHS IT fiasco tops it as $17 billion fiasco

Apr 09 2011

Tribute to Ted O’Neill

Ths is a tribute to Ted O’Neill, who played a key role in the development of health informatics technology today.  At the National Bureau of Standards, he helped bring the American National Standard MUMPS to reality.  Moving on the the Veterans Administration, he started the office that became the Decentralized Hospital Computer Program, that is now called VistA.

I shot this video at an Underground Railroad Banquet in 2009, and pulled together some of the interviews to talk about Ted’s contributions.

Contact me if you are interested in helping out to build a larger collection of oral histories…


Comments Off on Tribute to Ted O’Neill

Oct 18 2010

My commentary on Fierce Government IT

Published by under AHLTA,Heath IT,VistA

My talk at the Senate last week generated quit a bit of interest.  I just posted this commentary on Fierce Government IT:

In contrast to DoD’s “break and replace” development strategy, VistA has been under continuous evolution over the past 32 years. There have been thousands of errors made in the evolution of VistA, but generally they have been caught in the early stages. Like biological evolution, there VistA progressed by a series of small steps–even with people working on similar ideas in different parts of the country. VistA never started with a vision of the one correct way to the perfect EHR. Rather, it was a process of discovery, searching and amplifying what worked.


Comments Off on My commentary on Fierce Government IT

« Prev - Next »

Creative Commons License
Images by Tom Munnecke is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Based on a work at
Permissions beyond the scope of this license may be available at