One official said the move reflects frustration among senior Pentagon leaders with MHS efforts to procure new health IT systems, both independently and in partnership with the Veterans Affairs Department to develop the integrated electronic heath record. The departments have spent at least $1 billion over the past five years pursuing an integrated system.

This follows Chuck Hagel’s testimony to Congress that We Don’t Know What the Hell We Are Doing and former DoD Undersecretary for Health Affairs Ward Cascell’s that 2009 revelation that AHLTA is Intolerable.   I got private emails from DoD docs that were even more explicit:

AHLTA is far worse that you even alluded. It has virtually sucked the life out of our Providers and our MTFs. Yes, there may be some benefits but the pain is worse than the gain. I can’t believe that there will ever be a system that could successfully create a bi-directional interface with AHLTA. Any discussions that CHCS Ancillary functions will be replaced by the AHTLA as an architecture are just smoke screens for the embarrassment that AHLTA really is.

The worst part of AHLTA is when you actually have to read some of the documentation it generates…. there is rarely a coherent statement in a 3 page clinical note.

AHLTA is more than Intolerable…It’s the 3rd highest reason listed by the Army at the June 08 AUSA Conference Providers are leaving the military…

The first time I saw the AHLTA design, I thought that this was a reversal of all the successes the federal government had seen in health IT.  I remember thinking, “This is just one giant single-point-of-failure.”

30 years ago, we had two operational VA/DoD sharing sites. Here is March AFB’s Beth  Teeple’s oral history of the March AFB/Loma Linda test.

Thanks in part to Chuck Hagel’s early support of VistA, Rep. Sonny Montgomery, chair of the House Veterans Affairs Committee, noted that while VA had deployed a Core VistA system in all 172 hospitals for $82m, DoD had only produced prototypes of 4 stand-alone modules – for $250m (prices in 1985 dollars).  DoD called this IOCs – Interim Operating Capabilities, but we called them “Incompatible Operating Capabilities.” Each was  completely independent of the others, using incompatible coding systems, hardware, user interfaces, and communications protocols.  “Integration” was intended to come later.

This was classic DoD “Humpty Dumpty” development.  Break the system into pieces, then hire systems integrators to put it all back together again.  This is a wonderful business opportunity for the beltway systems integrators, but after 30 years of broken systems, its time to reevaluate the whole approach.

VistA never broke into pieces, but was based on common metadata and a shared set of tools.  It was “integrated” by virtue of never having been “disintegrated.”  Over the years, I learned that when someone speaks of “integrating” a system, we have to ask, “what disintegrated it in the first place?”  Until those forces are addressed, there is little chance of success.

Here is an excerpt of a letter  Sonny Montgomery sent me in 1984 Sonny Montgomery sent me in 1984:

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!

The VA and DoD forked into two paths: DCHP became VistA, and has won many awards and distinctions.  DoD reluctantly accepted CHCS, but under its management, has spiraled down into the mess we see today.

Here is 1984 Oct 10 Congressional Record authorizing DHCP as competitor in CHCS, my 1985 overview of DHCP to TRIMIS Program Office, 

And here is a 1984 oct 4 montgomery letter to Sec Def Casper Weinberger re DoD use of VA software:

Mr. Secretary, I cannot understand the DOD reluctance to try the VA system, which will provide on a timely basis the mandatory system compatibility between the two agencies.

It’s amazing that we are having the same conversation 29 years later.  Not a whole lot has changed, except that we’ve spent billions of dollars and decades delivering “intolerable” health care to those who most deserve it.

I’m getting tired of rehashing 30 year old events, but it seems necessary.  DoD has been relentlessly trying to do the same thing – and failing.  It’s time we break out of the “More Expensive Failure” mode and move to an approach that works.

In my next post, I’ll present a proposal for some solutions.

This ratchet ups the rhetoric of Assistant Secretary of Defense for Health Affairs’ Ward Cascells’ 2009 assesment that DoD’s AHLTA system is “Intolerable”

Not to kick a dead horse, but this has been going on for nearly 40 years now.  The DoD had spent $250 million prototyping the TRIMIS system, a collection of incompatible demonstration systems, while we at the VA were delivering a working, integrated hospital information system (DHCP) for 172 hospitals for $82 million.  House Veterans Affairs Committee Chairman Sonny Montgomery hit the roof over this, and partially triggered by the successful VA/DoD integration prototype declared that one of the competitors for the CHCS system would propose an adaptation of DHCP (now called VistA).  I went to SAIC to help with the effort, and we ended up delivering a very successful CHCS system, which is still the core of the DoD EHR system today.

The fundamental problem with the DoD is that they do not understand how to deal with systems of the complexity of a modern health care system.  They understand how to build an aircraft carrier, make sure that the troops have the supplies they need, and other activities from a linear perspective: the whole is equal to the sum of the parts.  Break the carrier into pieces, design all the pieces, and put them back together again to make a whole carrier.  This is (kindof) well and good for things that have this whole-equal-sum-of-parts quality.  Toasters can be taken apart and put back together again, and will still be the same toaster.

But health care is far more complex and dynamic than an aircraft carrier.  The hospital, Peter Drucker said, is the most complex organization in modern society.  Like a cat, we cannot dissect a hospital and put it back together again.

In a lesson straight from Humpty Dumpty, DoD wanted to break the electronic health record system into 17 “best of breed” applications, then hire a “systems integrator” to put Humpty together again.  This is like trying to build the world’s best car by trying to integrate the engine from a Corvette with the seats from a Rolls Royce and the chassis from a Porsche.  But despite how lucrative it is to be one of “all the king’s men,” it is simply not going to happen.

Yesterday, a friend of mine with many decades in the health IT industry called to tell me that he had just signed up with the VA in San Diego.  He said he was amazed at how well coordinated his care was – and this is from someone trying to do this in the private sector for 30 years.  This is the result of a fundamental approach taken from the earliest days of VistA – we were “integrated” by virtue of the fact that we never “disintegrated” into pieces.  We build a tool kit from which we composed the system over time, instead of the DoD’s approach of decomposing the system into pieces and then trying to put them back together again.  In that sense, VistA is remarkably similar to Wikipedia in this sense.  (See my discussion with Wiki inventor Ward Cunningham on this topic.)

So the fundamental issue is that we are dealing with a cat-like problem with toaster-like thinking.

Here’s my proposal: Fund a Skunkworks to get us out of this mess:

I’ve been developing VA/DoD interfaces since 1985.  They were technically correct, but politically incorrect.  I would hope that in the future, we can get past all the political nonsense of the past few decades, and just settle in to getting the technology working.

1.  Give me a contract to form a skunkworks.  I’ll collect 8-10 of the smartest people I know to develop the simplest solution that is “good enough” to get started.  I’ll also define an approach for “making it better.”

2.  I would like a couple of hospitals to work with (preferably in the San Diego area), at least one VA and one DoD.

3.  Connect me with teams of folks (both in the VA and DoD) who are passionate about making this thing work.  Create a bonus pool (for DoD as well as VA) against which I can make recommendations for bonuses for their contributions to the success of the skunkworks.

4.  Run interference for me to keep the bureaucracy out of the way.  We’ll be doing this in San Diego, far outside the beltway.

This can be a parallel activity to whatever the inside-the-beltway thinkers want to mull over.  Just ramp up a small, talented team working on the problem, independent of what is formally happening.  Think of it as portfolio diversification. The skunkworks funding would be just a fraction of what the IEHR-style aproach has been.

After reading Bob Brewin’s piece Did Jon Stewart Foil the Pentagon’s Health Records Plan? I have decided that Jon Stewart is a worthy recipient of the Unlimited Free Passage on the Underground Railroad certificate.  He understands the problems that the hardhats have been facing in the development of VistA over the years, and also seems to understand the success that it has enjoyed despite the hardships over the years.   I hope that this recognition will help him see some opportunities for improving government, health care, and service to our Veterans.  It is also an amazing story of how a bottom-up, decentralized approach to innovation can work, even in the most hardened bureaucracy.

The VA MUMPS Underground Railroad was formed in the early days of the VistA development in response to the attempts of the centralists to shut down a field-based decentralized approach.  The Hardhats were the technical folk who wrote the code to make it happen, but VistA was always more than just source code, so we needed recognize the many others who were involved in making it a success as described in Phillip Longman’s book Best Care Anywhere and this video. US Medicine editor Nancy Tomich describes the situation.  Nancy and I are now working on the New Health Project to carry things to the next generation.

The Underground Railroad has been struggling to build a common vision of VA/DoD health sharing for decades,   and not without its casualties,  so it is good to finally see some media attention to the issue.

This is the most prestigious award offered by the Underground Railroad, having previously been given in 1982 to Chuck Hagel:

In keeping with the Underground Railroad’s history, his certificate can only be given in person, with appropriate ceremonial presence.

Unfortunately, he missed the whole part about the success of the Underground Railroad, and how it lead to software innovations that triggered improved health care throughout the VA.

Wake Up, Jon….. there’s lot’s of rocks to be thrown at bureaucracies, but here is a great story of something that worked, but it’s being hidden behind what’s failing.  We have a success story to talk about – let’s use that to improve the situation.

I had a working VA/DoD medical record system working in 1985:

It was thoroughly studied by Congress, GAO, VA, and Arthur D. Little consultants, and passed with flying colors (well, except for some of the dirty tricks that DoD pulled, trying to make it look bad).  Here is an oral history interview about the system.

Creating a shared VA/DoD/Indian Health Service government-wide system was always the vision of our gang. We sought to create a common platform of shared information, controlled by “metadata” (or information-about-information). It was technically a correct solution, but politically incorrect. DoD did not want anything to do with VA, due to a “not invented here” syndrome. Plus, the realized that if they shared information systems, it could lead to shared VA/DoD facilities, which infringed on their turf.

One thing I’ve learned in bureaucracies: Never stand between bureaucrats and their pensions. Creating a more efficient system, which increases their productivity, allows them to do the same amount of work with fewer people.

However, bureaucrats are compensated by the number of people working under them. Fewer people means lower salaries, less job security, and less bureaucratic turf.

The problem of VA/DoD incompatibility is not a technical problem. It is a political turf war, and has been going on for decades. I don’t know if Chuck Hagel has the chops to solve this problem, but it deals with a fundamentally intransigent bureaucracy – in the mid levels – that sees VA/DoD sharing as competition for their jobs and security.

And Veterans and taxpayers lose.

“Hagel met with the programmers. “He found out about it and liked it, so he pushed it at the right time,” [Harry] Walters [Former VA Administrator] said. “Now it’s the most effective electronic health-record system in the country.”

The programmers presented Hagel with a certificate of appreciation at a banquet in 1982. “He stuck his neck out,” Munnecke said. “It was a gutsy decision on his part.”

Stay tuned.

And I continue to be amazed at how powerful the Underground Railroad I designed on a lark 30 years ago remains today.

Here is my original post about him.

For the past decade or so, people have been asking me if I knew about Chemtrails, which are supposedly part of a conspiracy to holds that some trails left by aircraft are chemical or biological agents deliberately sprayed at high altitudes for purposes undisclosed to the general public in clandestine programs directed by various government officials.

The evidence they point to is that some contrails from jets (the “Chemtrails”) persist and spread out, while others were narrow and dissipated rapidly (which they call “real” contrails).  They then move on to state these “chemtrails” are part of a government conspiracy, and then point to the fact that “the authorities” are denying their existence as proof of the conspiracy.

This theory is popularized on various internet-based conspiracy sites, and is similar to the Seattle Windshield Pitting Epidemic.  This was characterized by widespread observation of previously unnoticed windshield holes, pits and dings, leading residents to believe that a common causative agent was at work. It was originally thought to be the work of vandals but the rate of pitting was so great that residents began to attribute it to everything from sand flea eggs to nuclear bomb testing…. Within a week [of the initial observation], the news and the so-called “pitting epidemic” had reached metropolitan Seattle. As the newspapers began to feature the story, more and more reports of pitting were called in. Motorists began stopping police cars to report damage and car lots and parking garages reported particularly severe attacks.

Several theories for the widespread damage were postulated:

• Some thought that a new million watt radio transmitter installed by the Navy was producing waves that caused physical oscillations in glass;
• Some believed it to be the work of cosmic rays;
• Some reporting seeing glass bubble form right before their eyes, believing it to be the work of sand fleas;
• Some attributed it to a shift in the Earth’s magnetic field.

The whole “epidemic” came to a halt when folks realized that this was just normal pitting of windshields.

I find that when someone believes in a preposterous theory, coupled with a conspiracy cover that “the authorities” are covering up the facts, it is nearly impossible to talk about rationality.  I am simply aligned with the deceiving “authorities” and thus part of the conspiracy.  As Robert Park says in Voodoo Science:

I came to realize that many people choose scientific beliefs the same way they choose to be Methodists, or Democrats, or Chicago Cubs fans. They judge science by how well it agrees with the way they want the world to be.

My Proposal: Use Chemtrails as a Science Learning Opportunity.

The chemtrail phenomenon proposes a very specific hypothesis: that the emissions from Jet aircraft are leaving behind abnormal trails of vapor and chemicals that can’t be explained as normal contrails.  Contrails are everywhere, and observable just by looking up.

This is great science.  Now, let’s prove or refute this theory.

It turns out that NASA is already way ahead of this.  It is asking citizen scientists, teachers, and students to study contrails, and is even providing resources to do so.  There is even a global reporting system for students to record their local observations.  If there is so much energy to hype the conspiracy theory, let’s turn it into energy to scientifically research it.

And in the tradition of great science, studying this phenomenon from one perspective can lead to another discovery: the contrails play a big role in our understanding of climate change.  They may change the albedo of the planet significantly – and folks all over the world can make simple measurements to contribute to our understanding of this phenomenon.

So, next time I hear someone talk about Chemtrails as a conspiracy, I’m pointing them to this post as a “why don’t you find out for yourself?”

I’m sure that Apple is going through a lot of turmoil with the release of your recent results and 10% drop in share price.  I used to be a very enthusiastic Apple customer, feeling that I was getting value for my money, even with the higher prices I paid for your products.

However, this has changed in the past few years.  I’m not getting quality, I’m getting fleeced.  Where I used to feel that Apple was opening me to a new realm of creativity and innovation, now I feel that you are trying to lock me up inside the walls of your walled garden.

I used to trust you as a partner in my activities, thinking that the future would get better as your engineers improved you software and products.  Now, I cringe when I see new software announced, wondering what capabilities you are going to be taking away from me, and what intrusive software you will be shoving down my throat.

Companies are supposed to give value to their customers in exchange for their purchases.  Apple has somehow forgotten that basic business premise.  Instead you have been taking things away from us.  Here are just a sample of the things you have taken away: (I won’t even mention Google Maps):

1. I used to be able to take time lapse photos, import them into Quicktime 7, and produce a movie.  I could do “quick and dirty” editing of the movie and pass it on quickly and easily.  You took away this capability in the current Quicktime,  so now I have to keep two versions around.
2. I moved to Apple in 2006 after extreme frustration using Windows to do video editing.  I made a huge investment in Apple’s Final Cut Pro, buying the hardware and software to run it, as well as learning how to edit movies.  My local Apple store was a huge help with this, giving me hands-on training in Final Cut Pro with a really talented editor/Apple trainer.  However, when you moved to Final Cut Pro X, you made all of my prior work obsolete. I have spent hundreds of hours sorting my family movies into FCP bins and sequences that are simply not compatible with FCP X.
3. I have been using a Mac Pro desktop machine for my editing, based on Snow Leopard.  I had been holding off installing Lion, waiting to see if I wanted to move to Mountain Lion.  The day you announced Mountain Lion, you dropped the availability of Lion.  I am now caught in a catch 22 situation of incompatible hardware, software, and hours of video editing.

I’ve decided to move away from Apple products because of this continuing stream of customer-hostile actions you have forcing on me.  I don’t want to buy a separate printer to print from IOS devices, nor do I want to be locked into iCloud or iTunes for information exchange.

I don’t want to be locked in to your walled garden, and your disappointing sales report proves that others feel like me.  The same social networking effect that drove IOS upwards can drive it downwards, as customers flee from your oppressive and arrogant tactics.

I would hope that Apple can recover by returning to basic competitive principles of delivering value for money.   Customers need to look forward to new products as introducing new value, not just part of a shrinking  platform of planned obsolescence.

Here’s hoping.  But in the meantime, I’m moving away from Apple products…

What makes VistA unique in this list is that it is more than just code, it is a longitudinal data base extending back over 30 years across 172 hospitals and millions of patients.  This is an irreplaceable resource – VistA has petabytes of information about clinical conditions, activities, locations, and demographics from a period when most other hospitals were just dreaming about putting their records online.  This information is mapped by a “data dictionary” – a “road map” to the data base that defines the semantics (what it means) of the data, not just the syntax (how it is formatted).  It is also structured around a network model of information (the connectors between the dots), rather than just a hierarchical model (think of how the Dewey Decimal System tries to form a hierarchy of the books in a library).

This data is not complete, perfectly formatted, nor coordinated with precision.  Clinical data is not research-grade clinical trials data, and the patient population is that of (mostly male) veterans.  But it is a treasure trove of information, waiting for future data mining.  And what we consider “junk” information today (say, appointment schedules from 30 years ago), may hold great value in future research (say, how epidemics spread).

I hope the VA has the wisdom to retain this information in its original form.

At the 2012 OSEHRA summit meeting, I happened to be on a panel with medical researcher Leonard D’Avolio who commented on how easy it was to extract research data from VistA.  It was quite a touching moment for me, to have helped set in motion something that captured data that would be available 33 years later.

To quote my friend Doc Searls: “When young you think life is a sprint. When older you see it’s a marathon. And when mature you see it’s a relay race.”

The question, just who are we handing off the baton to?  Is the medical information we’re dealing with today going to be available 33 years from now?

I see a number of necessary conditions for this to happen:

1.  We need a curator.  Some organization needs to take on the role of “librarian” to maintain the archives.  The VA is a likely candidate for this, but even so, I think they might need a bit of prodding to keep them focused on their responsibility.  And maybe if Chuck Hagel (one of the Fathers of VistA) is confirmed as Secretary of Defense, he’ll recall his roots as one of the fathers of VistA and bring DoD medical information along, as well.  Roger Baker, Assistant Secretary for Information and Technology for the Department of Veterans Affairs, is moving things along so well that he earned a VIP membership card in the Underground Railroad.

2. We need to manage the data at a “meta” level.  The VistA Data Dictionary is being updated to more modern semantic web concepts, for example through Conor Dowling’s Semantic VistA.  The PCAST report called for greater use of Metadata, apparently unaware that they were recommending an approach already used by VistA for 1/3 century, which they cited as a case study in a successful large-scale system.  This is a bit like Monsieur Jourdain in Moliere’s play Bourgeois Gentleman, who suddenly realized that he had been speaking prose all his life, but didn’t know it.

3.  We need to continue the tradition of Open Source.  There is some movement in this direction with the OSEHRA effort, but it’s funding is but a tiny fraction of what it needs to be.

4.  We need to fund the effort.  Some folks think that Open Source means “free labor” – that the VA can just declare VistA to be an open source system and programmers will flock to it, offering volunteer time to do the VA’s bidding.  This isn’t going to work.  VA needs to fund these efforts, and release the work as open source.

5.  We need to think of what we are building as an “information space” rather than just an “integrated electronic health record.”  If someone said that they were going to improve our legal system by systematically storing and arranging lawyer’s documents, most of us would be a little skeptical.  But there is a huge chasm between our goal of  improving our health care and the current task of automating the medical record.  As we’ve seen with Twitter, Facebook, Google, Linked In, and Wikipedia, there are huge benefits to focusing on “connecting the dots” in large scale systems, rather than just focusing on specific dots.  Here’s a webinar I gave at Kitware, a leading open-source company.

All in all, though, it’s pretty cool to see VistA placed in this company.

I was part of the small group of programmers hired by Ted O’Neill to develop a decentralized hospital computer system.  This caused huge tensions with the central data processing folks, who wanted to centralize everything in a massive computer in Austin, Texas.

VistA blazed many trails in health IT.  It was the first to integrate SMTP email (I worked directly with internet pioneer Jon Postel, writing one of the first SMTP servers.)  We used what would today would be called Agile Development, starting with a prototype that was “good enough” and getting it into the hands of real users – the more feedback, the better.  We pioneered Open Systems thinking, making our software public domain and collaborating with Indian Health Service, DoD, National Health Service in Finland, and others.  We pioneered social networking/digital conferencing with VA FORUM, which at one time, supported 50,000 VA employees, all learning/teaching about the system, submitting bug reports, and just plain communicating with each other (a rare circumstance in mega-bureaucracies)

The centralists told upper VA management that our decentralized system would never work, but when then chief medical director Don Custis, MD saw the system actually being used, he quipped, “It looks like we have an underground railroad here.”  Nancy Tomich, editor of US Medicine at the time, described this event.

I took this as a sign that we should name our group the Underground Railroad.  I printed business cards, and started holding banquets to honor people who had made major contributions to the effort.  We had two awards: the Joseph T. O’Neill Outstanding Engineering Achievement award for technical folks (who we called Hardhats) and The Unlimited Free Passage on the Underground for the non-hardhats who helped our cause.

Here is a copy of the award I gave Chuck Hagel when he was deputy director of the VA:

Chuck went on to leverage the immense prestige of this award to become a US Senator, and now, nominee for Secretary of Defense.  He probably cherishes this as much as being designated an Admiral of the Great State of Nebraska.

Congressman Sonny Montgomery, then Chairman of the Committee on Veterans Affairs,  praised Chuck’s efforts in this Nov 5, 1984 letter to the Underground Railroad banquet:

Underground Railroad Members and Guests
Blackie’s House of Beef
1717 22nd Street, N. w.
Washington, D. c. 20817
Dear Friends:

On the occasion of your annual Underground Railroad Banquet, may I take this opportunity to send my greeting to you, and to shares ome of my thoughts concerning the Veterans’ Administration Decentralized Hospital Computer Program system with you.

As you know, the Committee and I fully supported Chuck Hagel’s decentralized ADP plan when he announced it in March of 1982 during his tenure as the VA Deputy Administrator. After Chuck left the VA, the plan, which relied heavily on the resources of the Underground Railroad, was derailed and appeared to be approaching its demise.

In order to get it back on track, I wrote a strong letter to the Administrator, and solicited the help of Chairman Boland of the HUD-Independent Agencies Subcommittee of the Committee on Appropriations. Subsequently, the Congress provided the funds and the VA, with the outstanding assistance of the Underground Railroad, performed a near miracle in bringing the largest health care system in the western world into the present day ADP world!
You, as members of the Underground Railroad, can take great pride in your outstanding accomplishment. The task required great dedication and unselfish personal effort. Regrettably, it resulted in some very calamitous casualties along the way. The job is not over, and I know that all of you will continue in your efforts to make this system the finest medical ADP system in the world. I pledge to you that I will continue my unwavering support of your important work and will maintain close oversight of all activities here in Washington and in the field to ensure that our mutual goal is met.

I recall Chuck as being very intelligent and charismatic, and a natural-born leader.  I thank him for his role in shaping the VistA system, and his vision in supporting the unknown “skunkworks” out of which VistA sprang.

And Chuck: if you are reading this, there is no expiration date on this certificate.  If you need the help of the Underground Railroad to help straighten out the VA/DoD EHR mess, we’re ready to help.  I figure we could save the DoD $10 billion or so.  And if the code we wrote back then might have aged a bit, I think that the principles we espoused are even more current in today’s federal health IT environment.

And an Open Source VistA community is alive and thriving.  The VistA Community Meeting is happening this month in California, and  OSEHRA (Open Source Electronic Health Record Agent) is actively supporting open source VistA.