Introducing Anna Karenina Development ™
© Doug Meil, 2021
Update March 2021
A more serious and professional version of this was published on the Association for Computing Machinery blog site. https://cacm.acm.org/blogs/blog-cacm/251396-anna-karenina-on-development-methodologies/fulltext
The Actual History Of Explorys, Part 2 (2015 to 2018, The IBM Watson Health Years)
© Doug Meil, 2021
See the prior article “The Actual History Of Explorys, Part 1 (2009 to 2015)” for the first part of the story.
https://themeildeal.blogspot.com/2021/01/the-actual-history-of-explorys-part-1.html
Explorys - IBM Watson Health Years
Much has been written about IBM Watson Health during the years 2015 through 2018…
https://www.statnews.com/2018/06/11/ibm-watson-health-problems-layoffs/
https://www.theregister.com/2018/05/25/ibms_watson_layoffs/
… for the record, I was not the source of any material for these articles (or any others).
Some of the public criticisms of Watson Health were fair-hits and deserved, and some of what was written were cheap-shots and/or flat-out wrong. What was undeniable was that Watson Health’s plans were ambitious, and that things didn’t turn out like the 2015 strategic PowerPoint decks had predicted and a lot of people got hurt in the process. Why these plans didn’t work out are where the details matter. What follows are my thoughts on some of the more frequent comments and conceptions about Watson Health, based on my experiences during this period.
It should be noted that my experiences with IBM went back many years before Explorys was acquired by IBM in April 2015. I worked closely with IBM in the 1990’s, spoke once at a CICS and MQ Series Technical Conference, and also was on a project that was a case study in IBM marketing material. IBM was also an early supporter of Java, which I respected. I did have a few concerns about IBM, though. For example, at a prior job we tried out Rational Studio but got tired of its interminable startup time, and went back to Eclipse which started up in few seconds and was a lot snappier. But I still came into IBM an enthused employee and deeply knowledgeable of IBM’s history.
1) IBM & Healthcare
From IEEE Spectrum:
Long before Watson starred on the Jeopardy! stage, IBM had considered its possibilities for health care
From STAT:
“When IBM introduced Watson Health , it already had a group of employees working with healthcare clients using its own secure cloud to store data, two former employees told STAT. The Watson division, though, began marketing another cloud, which not only appeared duplicative of IBM's existing offerings, but created competition from within.”
IBM had been in healthcare for a long time before Watson Health. IBM Research had been undertaking various healthcare projects, and the consulting arm/s of IBM had been performing a variety of custom healthcare projects (typically on-prem deployments) utilizing IBM products such as DB2 (database), UDMH (healthcare data model), MDM (entity matching), MQ Series (messaging) and IIB (integration bus), among others. The development of Watson for Oncology pre-dated the creation of the Watson Health division by several years. IBM had also once been in medical devices long ago with a division in Minnesota, I was told.
But in general IBM didn’t have much in the way of healthcare applications. IBM would sell you a set of tires, an engine, and a steering wheel, and the consulting hours to put them together, but it couldn’t sell you a vehicle. That’s what Watson Health was going to address. This was a reasonable and admirable ambition.
For starters, IBM’s cloud situation was – pun intended – somewhat cloudy in 2015 as there were multiple hosting groups within IBM. The primary consumer-facing cloud service at the time was SoftLayer, itself an IBM acquisition in June 2013 - less than two years before the formation of Watson Health. SoftLayer was “sort of IBM and sort of not IBM” in the sense that IBM Watson Health was pretty much paying the same price the public was paying, so it was hard for an acquisition make a compelling financial case for the switch from their existing hosting environments. And from a technical perspective SoftLayer was a “cloud” (so that part was true) but it wasn’t the same class of “cloud-native” offering as AWS, Azure, and GCP for supporting programmable infrastructure-as-a-service, at least at the time.
2) IBM Watson Health Origin & Dynamic
From IEEE Spectrum:
“We pivoted so many times.”
“While the engineers tried to follow the shifting directions from above, Phytel didn’t deliver anything new to the market.”
“they really thought IBM would take it to the next level—not destroy it within three years.”
From STAT:
"There was a lot of internal friction," one of the former employees told STAT. "There were two products that did exactly the same thing, that were being sold by two different parts of the company, that had two different development roadmaps."
“Watson Health's various business units were too focused on their own growth, which discouraged collaboration. Coupled with the company's failure to combine their newly acquired companies and align their cultures and operations, IBM was laden with duplicative products and services in a way that was all but cost efficient.”
Anyone familiar with acquisitions knows that successfully integrating a single company is hard. What IBM tried with Watson Health was well beyond that. On top of creating the entirely new Watson Health division in April 2015, IBM acquired a bunch of healthcare software companies…
· Curam (social services software, acquired 2011. Moved into Watson Health after the formation of the division in April 2015)
· Explorys (healthcare & life sciences population analytics, acquired April 2015)
· Phytel (healthcare analytics & patient contact, acquired April 2015)
· Merge Healthcare (imaging, Fall 2015)
· Truven (claims analytics, April 2016)
… to seed this new division, and then also mixed in several IBM Research healthcare assets. At the formation most of the actual products and most of the actual customers at the formation of the division were from acquisitions. The overall Watson Health strategy, though, was to rebuild everything that had just been acquired in the “Watson Health Cloud” on SoftLayer.
It shouldn’t take a software or management guru to see the potential downsides of this approach. For starters, there were some overlaps in acquisition functionality, which were obviously not the fault of any individual acquisition, but even that wasn’t the biggest problem. The strategic rebuild plan, while probably beautiful when viewed from 250,000 feet, placed scant priority on evolving the acquisitions and their already deployed solutions to keep their existing customer bases happy (and growing). This effectively would require hitting a multi-year pause for the existing solutions, and most customers don’t appreciate that. In too many ways Watson Health operated like a billion-dollar research project that was disconnected from the market, yet still burdened with the financial expectations of a public company that wanted quarter by quarter stratospheric results. I think even the IBM top-brass would have to grudgingly admit in retrospect that this was crazy-making.
There is also a process for acquisitions called “Blue Washing” (actual IBM term). All of the acquisitions were being requested/strongly-encouraged/mandated to convert as many technical frameworks as possible to IBM products, and this list was long. On paper, this process does make some sense. The trick is that when push comes to shove, who is more important? The answer depended upon whom was asked, and within IBM there are a lot of people with opinions. Just stack-ranking the priorities of internal IBM stakeholders – irrespective of the priorities of paying customers - is an enormous task. The acquisitions had to balance this dynamic along with everything else that was happening in Watson Health, which was a lot.
The Explorys engineering team still produced some great work in this period, kudos to the team. But this was done within a cyclone of distractions. In 2017 alone we had 3 Stop The World And Re-Prioritize Everything events driven by Offering Management (what IBM calls “product management”), where products went from being the #1 priority to not happening at all, and then in some cases back again. That was just one year. It’s really hard to get anything done with that as a backdrop - let alone sustain any velocity - because what everybody knows but nobody wants to acknowledge is that if you want plausible software estimates, you need software engineers to make them. And if you are frequently asking said software engineers to stop what they are doing and re-plan then they aren’t building software.
Watson Health was a good idea, but as undertaken where were too many conflicting priorities, and it was too much to manage at once.
The results, at least from the Explorys perspective, could have been predicted by any relationship book or country music song:
· Phase One: The Yelling
o Customers wanting to know roadmaps for their existing products, and looking to see continual product progress to make sure they aren’t being abandoned.
· Phase Two: The Silence
o It seems like things are getting better when customers stop complaining, but what is actually happening is RFPs are being sent out.
· Phase Three: The Leaving
o It was only when the contract terminations went full boil that Watson Health leadership started to care about the acquisition customer bases, but at that point it was too late.
3) Can We Talk About Watson?
From IEEE Spectrum:
In part, he says, IBM is suffering from its ambition: It was the first company to make a major push to bring AI to the clinic. But it also earned ill will and skepticism by boasting of Watson’s abilities. “They came in with marketing first, product second, and got everybody excited,”
When attending big data conferences during this period one could hear major tech companies talk about AI and machine learning projects with a fairly standard vocabulary: unsupervised and supervised learning. Clustering algorithms. Classifiers and regression algorithms. But IBM had Cognitive Computing. Huh? What?? That approach had contrasting effects: technical people didn’t take Watson seriously, and business people took Watson too seriously, causing inflated and unrealistic expectations.
On top of that, IBM tried at the time to tie Watson to a variety of other IBM products, like only running on IBM hardware, or only being available on the IBM Cloud. Watson was not just confusingly described, but its intelligence was artificially limited. Acquisitions still running on their existing hosting infrastructure couldn’t easily take advantage of Watson - whatever it was – even if they wanted to.
I give IBM a lot of credit for its Deep Blue Chess projects and its Jeopardy effort. That was some innovative and out of the box thinking. I will freely admit to having my picture taken by the Jeopardy game podium at the Yorktown Heights Research Center. I even had a member of the Jeopardy team perform a fantastic guest talk at the nerd meetup I run (Cleveland Big Data Meetup). Like I said, I’m a fan of these efforts. But IBM lost its way on Watson by focusing on magic instead of technical excellence. IBM later relabeled Cognitive Computing as something else, but the damage was done. It was another lost opportunity.
4) Explorys & Epic
From STAT:
“one of Watson's companies, Explorys, found a way to get behind Epic's firewall and use patient data to fulfill its contractual obligations with its client hospitals”
Firstly, Explorys was a part of Watson Health, not Watson. Those were two different divisions in IBM. While I thought IBM was pretty clear on this point for some reason it became a surprisingly frequent point of confusion in the public, and I’ve lost count how many times I had to explain this.
Secondly, Explorys did not “hack” or “get behind” Epic’s firewall. The above quote is at least half-right in that Epic integration was performed under contract with client hospitals. Bypassing a firewall, though, is a serious allegation that minimally implies disregard for security policy.
During the timeframe Explorys was in business, Epic was typically deployed on-premise (i.e., in a data center leased or operated by the healthcare customer), and especially so for mid/large-sized deployments.
Epic has a few core data components: Chronicles (the operational data store), and Clarity (the data warehousing platform). Chronicles is supported on a database technology called Intersystems Cache. Clarity is supported on Oracle, SQL Server, and Teradata.
These technical factoids about Epic are all over the internet, and are not secrets.
Explorys was just really good at data integration. Per the first article, I had been doing distributed data collection for over 5 years even before Explorys, and then Explorys took it to another level.
5) The Parable Of The Whiteboards
How do bad decisions happen? It’s easier than you might think, especially at a big company. One factor is whether the person making the decision has to live the consequences of the decision.
Take for example the Explorys office during the 2016-2018 period, which was the first “Designed By IBM” office. Per the first article the History of Explorys, moving offices was a core competency. Rather than assembling our own desks as in previous Explorys offices, in this new space we now had real grown-up office furniture, which was exciting.
But when we moved in, we saw this…
(photo by Doug Meil)
… in every breakout room. One room would have been a hilarious accident. This was on purpose.
Software engineers like monitors and whiteboards. But this wasn’t a Reece’s Peanut Butter Cup situation where adding them together makes it better.
I don’t fault the installers for this. I’m sure when they received the work-orders they probably shook their heads in puzzlement, but then thought “that’s what the paperwork says” and started making holes. A discrete inquiry was later made as to the design intention, and the alleged response was that the configuration was a “best practice” that “merged digital with the physical.” Which is fair, because merging things is what screws and a power drill tend to do. It appeared that the facilities people had neither owned nor leased many dry-erase markers. But the real question is how many people knew about this idea before it happened and said nothing about it? Or were unable to raise concerns for fear of being labelled an “acquisition troublemaker” and a “whiteboard purist”?
Fortunately, this situation had a remedy: our engineering executive finally persuaded facilities to get another set of whiteboards ordered and mounted on the opposite walls, this time without monitors installed on them. Problem solved! On top of managing a development organization and all of the other topics mentioned above in this article, obtaining functioning whiteboards also became an executive-level responsibility.
Sometimes it’s possible to recover from a bad decision. Sometimes not so much.
Coda
I was appointed an IBM Distinguished Engineer (DE) in 2016, which I appreciated. It’s a big promotion in the general sense, and it comes with a framed certificate. It’s a process in some ways like the application for academic tenure in that one must prepare an extensive package for review by various boards and all the way up the IBM chain, but different in that DE appointments are for a specific area of executive business and technical responsibility. While it does recognize past accomplishments, it is fundamentally a forward-looking position. And executive has particular meaning at IBM, as DE’s are also supposed to be looking out for IBM as a whole. I was told that DE’s should be like knights on a chessboard, which was an analogy that I liked. It could take a couple of years to really get comfortable in the role, I was told. As IBM DE’s are D-bands in the IBM job-grade hierarchy, the appointment also comes with executive pay structure. The bonus algorithm is supposed to be 1/3rd about the DE, 1/3rd for the business unit results, and 1/3rd for IBM results as a whole. I worked a full year in 2018 which was a bad year for Watson Health but IBM still made some money. I would have been happy with a couple of Dicks Sporting Goods gift certificates and a ham sandwich as a token thank you for trying because you have been warning us for years about what eventually happened as I had left in January 2019. When the 2018 bonuses were paid out in March 2019 mine was zero dollars and zero cents. The message was clear: monitors belong on whiteboards, and don’t you forget it.
I wish IBM the best and hope it learned something from this experience. It was costly in terms of money, time and people.
See this link for "Explorys, a PostScript" (2023)
https://themeildeal.blogspot.com/2023/10/explorys-postscript.html
Appendix
This writeup was directed at IBM and the macro-decisions that determined Watson Health’s fate. There is always more detail to any story, but these were the major themes as I saw them. Explorys did bring its own dash of executive dysfunction into Watson Health, however. Reviewing what happened after Explorys is insightful into how Explorys operated both pre-acquisition and the early critical years in Watson Health. The article below explains Cleveland’s Amazon HQ bid, and how the Unify Project somehow wrote itself into the city’s bid and described itself as being one of the most important assets in the region without having actually having done anything. And it had blockchain too. That really happened. So there’s that.
And the story continued…
Also for the record, I had no part in any of those articles.