Archive for the 'Health IT Fiascos' Category

Aug 12 2014

Interview in Fedscoop about Chuck Hagel and the Underground Railroad

Here is an interview from FedScoop that captures a lot of the dynamics of the VA and DoD health care systems.

I get a kick out of seeing Chuck Hagel’s wry smile at 2:06 in the attached video, where he can’t quite admit to Congress that he is a card-carrying member of the VA MUMPS Underground Railroad.




Comments Off on Interview in Fedscoop about Chuck Hagel and the Underground Railroad

Jul 04 2013

Open Letter to Chuck Hagel: DoD still doesn’t know what the hell they are doing

Dear Chuck (I’m using this informal salutation in honor of your status as one of the fathers of VistA),

I was impressed with your concise and accurate assessment “I didn’t think we knew what the hell we were doing.” before a Congressional hearing Apr. 16. 2013.  I fear, however, that this is still the case.

I can only imagine the endless swirl of acronyms, PowerPoint presentations, and facile phrases being tossed at you.  I’m sure you’ve been told that DoD will have a “seamlessly integrated electronic health record” with VA, and that it will be built of “best of breed” components that will all snap together seamlessly because you have an “enterprise service bus.”  Doing this will improve health care for active duty and veteran population, eliminate the VA eligibility backlog, and be accomplished by the next election cycle for just a few billions of dollars.

These are all very good intentions.  But I fear that you are paving a road to a hellish destination.  Rather than lifting up the VA eligibility problem to a shiny new common information system, you are on the verge of dragging health IT into the same bureaucratic vortex that has already done so much damage in the past.  AHLTA was declared “intolerable” in a Congressional hearing 4 years ago.  Yet, not only is it still around (and absorbing $600m/yr operations and maintenance costs), but it is also serving as a template for the next generation of the IEHR – a top down, mega-centralized administrative system far removed from the clinical needs of health care professionals and patients.  DoD continues to focus on the organization chart, not the patient, closely coupling their software designs to their bureaucratic stovepipes.  Indeed, it is rare for me to even find the word “patient” in any DoD health IT documents.

DoD is taking a “We chew, you swallow” approach to dealing with doctors and other health care providers.  Vice Adm (ret) Harold Koenig, MD, Deputy Assistant Secretary of Defense, Health Care Operations, 1990-1994, recently told me of his disgust with the current trends at MHS:

“DoD Health IT is now designed for the administrators with the patients as the data source and the clinicians as data entry clerks.”

Here is another email message from a military physician:

AHLTA is far worse that you even alluded. It has virtually sucked the life out of our Providers and our MTFs. Yes, there may be some benefits but the pain is worse than the gain. I can’t believe that there will ever be a system that could successfully create a bi-directional interface with AHLTA. Any discussions that CHCS Ancillary functions will be replaced by the AHTLA as an architecture are just smoke screens for the embarrassment that AHLTA really is.   The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.

And here is a 1984 letter from Sonny Montgomery to Secretary of Defense Casper Weinberger re DoD use of VA software:

“Mr. Secretary, I cannot understand the DOD reluctance to try the VA system, which will provide on a timely basis the mandatory system compatibility between the two agencies.”

And here is a letter that Rep. Montgomery sent to the to the Underground Railroad skunkworks in 1985:

“As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!”

We have seen VistA thrive within the VA and in the Indian Health Service (as RPMS).  Ironically, UK National Health Service has just announced that it will spend some of its £260m Technology Fund on further exploring the creation of an NHS version of the US Veterans Health Association’s open source electronic medical record, VistA.

This is ironic because the NHS has recently cancelled a massive Health IT project that was almost a clone of what IEHR is attempting to do.  Here’s my Hello to NHS.

In short, DoD is trying to get out of a hole by digging it deeper.  The current path will exacerbate the VA Claims eligibility problem.  It will further damage the ability of DoD physicians to deliver quality health care.  But will generate enormous profits to systems integrators who will benefit by the system not working, as they see an continuous stream of expensive change orders. This will come at the expense of further suffering of active duty and veteran patients.

I think that the way out of this problem is to rethink the architecture and the ethos of the VA/DoD health care efforts:

  1. Shift to a Patient-Centric ethos.  The current trend is towards a single, mega-centralized, standardized, enterprise-centric “federated” data base environment, supposedly the only way to achieve a “seamlessly integrated” system.  The VistA that you green-lighted 31 years ago was based on a design ethos of a parallel, decentralized, patient-centric system.  Given the computing power (much less than an iPhone’s computing power to run a whole hospital), and communications speeds (1/40,000th of an iPhone’s) we focused on the hospital as the “anchor point.” With the coming effects of the revolution in translational/personalized/genomic/telemedicine/social network medicine, it is imperative to put the patient at the center of the health care universe, not the organization charts of the bureaucracies who run the hospitals.
  2. Accept that a hospital is different from an aircraft carrier.   Adopting health IT, dealing with the complex interplay between providers, patients, and information is a fundamentally different thing than acquiring an aircraft carrier.  Just because they cost the same order of magnitude does not mean that their acquisition can be managed the same way.  Even within a hospital, the administrative information (logistics, billing, accounting, etc) is a fundamentally different problem than dealing with clinical information such as lab, pharmacy, and radiology.  This ignorance has been a fatal flaw in any number of failed systems over the decades.
  3. Decouple the IT architecture from the Organization Chart.  The designs that I’ve seen coming from the DoD are enterprise-focused, “baking in” all of the stovepipes, organizational turf wars, and protecting rice-bowls of the many political, economic, and professional constituencies hoping to influence the architecture.  Instead of patching together an “integrated system” of point-to-point connections, we need to move to a broader vision of creating a common information space.  Note the words of Tim Berners-Lee in his design of the World Wide Web:
    What was often difficult for people to understand about the design of the web was that there was nothing else beyond URLs, HTTP, and HTML.  There was no central computer “controlling” the web, no single network on which these protocols worked, not even an organization anywhere that “ran” the Web. The web was not a physical “thing” that existed in a certain “place.” It was a “space” in which information could exist.”
  4. Uplift the current systems into a higher level of metadata management.  This is equivalent to building a ladder, rather than trying to get out of a hole by digging deeper.  The current approach throws away the conceptual integrity that made VistA such a success, replacing it with an “aircraft carrier” mentality that obliterates the ethos that drove VistA’s success.  The President’s Council of Advisors on Science and Technology published a health IT study that a great job of describing some of the foundations of this metadata approach, and treating Health IT as a “language” problem, not an “interface.”  This is a very nuanced difference, but think of how easy it is to link an book reference to a Twitter post:  you simply drag the URL of the book to Twitter, and press send.  You do not need to interface Twitter to Amazon, or use the “Book reference nomenclature standard,” etc.  It is simply an intrinsic property of the information space.  Similarly, we could build a health information space that that allowed this kind of sharing ( with enhanced patient privacy and security), as an intrinsic of being part of the common information space.  This move to a higher level of abstraction is a bit like thinking of things in terms of algebra, instead of arithmetic.  Algebra gives us computational abilities far beyond what we can do with arithmetic.  Yet, those who are entrenched in grinding through arithmetic problems have a disdain for the abstract facilities of algebra.  The DoD is rejecting the Uplift model, instead succumbing to the “Humpty Dumpty Syndrome” – breaking things into pieces, and then trying to integrate them again.  This is great work for “all the Kings men” as long as the King has the resources to pay them to try to put Humpty together again.  But sooner or later (and I had hoped you would have chosen the “sooner” option) the King needs to cut off this funding.
  5. We need a Skunkworks to develop and prototype a new vision.  The VistA that you greenlighted was designed by a very small group of dedicated, talented people working directly with VA clinical staff.  We were building a community of users, co-evolving the software and the community.  Ward Cunningham, inventor of the Wiki technology, and I talked a bit about the origins of VistA and of Wikipedia.  I’ve already begun collecting the people and ideas to make this a reality.   Just a tiny fraction of the IEHR budget would deliver spectacular results.

We are at a turning point in health IT in the United States and the world, but I fear that you are on the wrong path.  I hope you reconsider the direction you are going.

P.S. The next Underground Railroad Banquet is scheduled to happen in October at the VistA Expo in Seattle, if you or any of your staff who are appreciative of the VistA ethos would like to join us.


Comments Off on Open Letter to Chuck Hagel: DoD still doesn’t know what the hell they are doing

Jun 18 2012

“One Piece at a Time” – next generation federal health IT architecture

The details for the Integrated Electronic Health Record (IEHR) are just now beginning to roll out.   It’s pretty much a replay of the “Best of Breed” marketing approach that I’ve seen been pitched for decades.  Basically, collect all the parts and the whole will fall together with “just a bit of integration.”  They are issuing dozens of RFPs and RFIs describing all the pieces they seek; how it all fits together is a lot of hand-waving at the moment, with an inordinate faith in the role of an Enterprise Service Bus to magically make everything work together.

This is a little like someone selling you the world’s best car, based on “best of breed” components from all the best manufactures.   An engine from a Corvette, seats from a Rolls Royce, transmission from a Ferrari, etc. To make it all fit together, they would say it is all “standardized” – every part would use metric nuts and bolts, 12 volts electricity, and a common electrical harness (equivalent of the Enterprise Service Bus), to make sure that all the parts could work together.

We took an opposite approach in VistA, starting with a conceptual model of a system driven by active metadata – a common roadmap to patient database.  We were “integrated” by virtue of not “disintegrating” into pieces in the first place.  We had an overarching Conceptual Integrity to the design that gave us a common foundation from which to grow, climbing up the ladder of abstraction rather than falling down it.

This turns out to have been a good decision.  The President’s Council of Advisors for Science and Technology (PCAST) issued a report the report—Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward— that specifically pointed out VistA as one of the success stories for large scale EHR systems.  It also advocated greater use of metadata: tagging data dynamically through meta rather than “pigeonholing” it into pre-defined, previously synchronized slots.

This argument can be very abstract to some people, particularly to bureaucrats whose entire life has been engulfed by hierarchies and “pigeonhole” systems thinking.  The notion that systems can associate dynamically, spanning hierarchies (or not even being associated with a hierarchy) casts them into unfamiliar uncharted territory.  To them, complex systems require complex organizations to manage and regulate them. They seek efficiency, rather than resilience, adaptability, or robustness.

In order to make this a bit more understandable, I am proposing a theme song for the IEHR architecture: Johnny Cash’s One Piece at a Time.  Think of the IEHR vision as Johnny Cash’s Cadillac.  I’ll let the viewer draw their own analogies.  Of course, Johnny didn’t have billions of taxpayers dollars to spend, nor did he have a gaggle of consultants ready to reap huge contracts to do the “little bit of integration.”


Comments Off on “One Piece at a Time” – next generation federal health IT architecture

May 31 2011

AHLTA Is Not Alone, Part II: Ontario’s $1 Billion health IT fiasco

It really pains me to post this kind of stuff, but I fear that we are entering an era of unprecedented Health IT fiascos.  After the $4 billion Department of Defense AHLTA fiasco, (AHLTA is Intolerable), and the $17 billion UK National Health Service fiasco, here is a A scathing report on the eHealth Ontario spending scandal charges that successive governments wasted $1 billion in taxpayer money. Carpet bagger consultants have arrived in full force, with flimsy-if-no experience in the complexities of medical informatics, but a deep, abiding concern for their own income.

The head of the Ontario disaster “billed thousands of dollars for limousine rides… before she resigned from her $380,000-a-year job in June. She was given a $317,000 severance package and received a $114,000 bonus after just 10 months on the job.”  This payment was made for leading the fiasco, not delivering a product.  The Canadian auditor found the agency had fewer than 30 full-time employees but was engaging more than 300 consultants. they also said the effort was “lacking in strategic direction and relying too heavily on external consultations.”

The idea behind eHealth is to create electronic health records for Ontario, something the auditor says could save $6 billion if implemented in every province and territory.  Instead, Ontario “is near the back of the pack” when it comes to electronic health records, having wasted millions on underused computer systems and untendered contracts.

EHealth was set up in 2008 to create electronic health records after Smart Systems for Health spent $650 million but failed to produce anything of lasting value. Smart Systems for Health was quietly shut down last September.

And then there is politics: “the Conservatives and New Democrats complained the agency gave out $5 million in untendered contracts to consultants.   Documents released by the government since then showed the value of those untendered contracts was closer to $16 million, with the biggest ones going to companies the opposition parties say have ties to the Liberal government.”

These kinds of shenanigans are not unique to Canada, of course.  Here is a letter from a group of Wisconsin congressfolk blatantly meddling in VA and DoD internal IT decisions:
Wisconsin reps try to derail VA/Defense open source health records system which happens to be the home of Epic Systems, who hope to provide one of the most proprietary, closed systems in the nation.  Epic denied hiring lobbyists, but a quick search on showed over $1m in political donations from Epic CEO Judy Faulkner and Epic, much directed to the very Congressfolks who took the time to write the letter.  Epic don’t lobby; I guess it’s more efficient just to send cash to their Congressfolks.

This isn’t just about money… Bad health IT software kills people.  When I worked at the VA Loma Linda, I routinely rode in the staff elevator with the gurney of a patient headed towards the morgue.  I would be in meetings when a code blue sounded, and docs would rush out of the room, only to come back breathless, saying, “now, where were we?” This was a sobering and constant reminder of the significance of what hospitals do.  The wrong information, or delays in getting the right information, can kill.  Installing patient software (e.g. the VA’s bar coding of patients for medications) can save lives.

At the core of this – around the world, not just the US – are self-righteous bureaucrats who persist in top-down, centralized, waterfall development, “one-correct-way” thinkers who see their bureaucracy, not the patient, as the center of the IT universe.  We are not creating an “accounting system for disease” as if it is a banking transaction system.  We need to building communities of health, with the individual at the center of the world.  We need to overcome the “failure to communicate” in health care; the Electronic Health Record is but one aspect of a much broader need for transformation in our health care thinking.

I don’t see this shift happening.  Instead, I see self-righteous indignation on the part of the bureaucrats who perpetuated the fiasco in the first place.  It’s as if a carpenter builds a crooked house, and blames his hammer.  All he needs is a another, more expensive hammer, and he’ll build a house twice as expensive.  The US Department of Defense is now on its fourth generation of crooked houses/more expensive hammers, and is in the process of repeating the process all over again, full of self-righteous indignation that VA’s VistA is continuing to evolve while their break-and-replace models continue to fail.  DoD continues to blame “the software” for their failures, oblivious to the fact that they were the ones who perpetrated the fiasco in the first place.  Governments continue to pour money onto the fire, paying for new hammers even when bureaucratic incompetence has been repeatedly demonstrated.

My current Health IT fiasco report card:  I hope this is the extent of the list, but I fear that there are many others to be listed.  Please email me with any links.

UK National Health Service:  $17 Billion

US Department of Defense AHLTA: $4 Billion

Canada: Ontario eHealth System: $1 Billion

(tip of the hat to Joseph Dal Molin for some of these links)



Comments Off on AHLTA Is Not Alone, Part II: Ontario’s $1 Billion health IT fiasco

May 28 2011

AHLTA is Not Alone – UK NHS IT fiasco tops it as $17 billion fiasco

I had thought that DoD’s AHLTA held the record for being the world’s greatest health IT fiasco, spending $4-5 billion of taxpayer’s money on a system that was deemed Intolerable by the GAO and an Assistant Secretary of Defense.  It was so bad that it is cited as one of the leading reasons physicians are leaving military service.  One user told me, “The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.”

I knew that AHLTA would be a failure the instant I saw a diagram of it – a giant, centralized single point of failure that ignored everything that I had found successful in doing the VistA architecture.

This is not just dollars we are talking about, or missed opportunity costs.  Bad software kills people.  I don’t know if we’ll ever know how many patients AHLTA has killed, but it has to be significant.

But enough about AHLTA: I just read this article NHS IT system condemned about the UK National Health Service is in the midst of a far greater fiasco:

In a jaw-dropping condemnation of the NHS National Programme for IT, the National Audit Office has exposed a white elephant in the final stages of collapse.

In what read as a final pronouncement, the NAO reported that after nine years and £2.7bn, (US $4.5 billion)  the NHS has failed to deliver its primary aim of an electronic care record for everyone in the country.

The situation looks so dire the system might continue leeching money from the NHS for another decade if the whole scheme and all its software is not seized by state liquidators.

They go on to say that they don’t think its worth going forward with the remaining US$7.7 billion:

The NAO (National Audit Agency) said CSC would likely fail to deliver the rest before its contract runs out in 2016. DoH had been in dispute over its contract with CSC for 18 months, trying to claw back some of the £5bn (US $8.2b) it had promised the supplier.

What is particularly galling about this situation is that both the US DoD and the UK NHS had access to an award-winning, open source hospital information system that has been running at this scale for 25 years… the VA’s VistA system.  As one of the original software architects of this system, I have seen an endless stream of novice health IT folks appear on the scene, thinking that because they know some other aspect of IT that they can apply this to health IT.  In the past, these folks would slink away after losing $100m or so.  But now, it seems the stakes have been raised to the tens of billions.

We took an innovative approach in designing VistA, and it worked…  So, why are people continuing to pour money down the same failed approaches?

The answer is to follow the money:  A $5 billion fiasco is extremely lucrative to the beltway bandits.  Sure, they have to weather a few editorials and a couple of roastings before the various legislators.  But this can all be fixed with a little lobbying effort.  They’ll apologize, promise never to do it again, change the name of the project, and double the price tag of the next round of funding.  And laugh all the way to the bank with the profits they made on the fiasco.  It’s likely that a successful implementation would have probably earned them less profit.

Tthis isn’t just about an atrocious waste of taxpayer’s money.  People die from bad software.



Comments Off on AHLTA is Not Alone – UK NHS IT fiasco tops it as $17 billion fiasco

Creative Commons License
Images by Tom Munnecke is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
Based on a work at
Permissions beyond the scope of this license may be available at