Miscalculated Risk


“We’re in the final stages of testing and anticipate your system will go live three weeks from today.”


After too many months and too much money, the tech company was finally reporting that it was just a few minor tweaks away from launching the shiny new, custom software system that would make life so much better for the partners of the high-powered venture capital firm.

One of the most influential companies in the industry, the firm enjoys a reputation for taking smart risks and returning big rewards for its investors. This time, it had invested heavily in its own future with a desperately needed upgrade to its portfolio management system.

Now it was time to take the system for a test ride. The entire company was excited to see what this thing could do.

And it did…nothing.

It failed its very first task. Then, like a house of cards, it fell apart completely.

Calculations – valuations, projections, balances – the core of everything the system was supposed to do, didn’t calculate. And that was just the beginning.

Now what?

The partners, true to form, went into high stakes mode and unabashedly put out an SOS. John, one of our senior executives, listened intently at a social gathering as his friend, Dave, a partner in the firm, described the disastrous scenario.

“John, nobody can figure out why it won’t work. It’s crashing and burning right in front of our eyes.”

John suggested we conduct a thorough review of the system. A fresh set of experienced eyes can find programming flaws no longer seen by the original developers. And it’s very common to have a third party conduct a peer code review, especially for a project this important.

Dave was visibly relieved.

So we popped the hood, expecting to find trouble spots here and there. With hundreds of data fields and thousands of calculations to inspect, finding problem needles in the custom software haystack would be demanding, but achievable.

What we found shocked us. We’d conducted dozens of these reviews, and likewise, our work had been reviewed dozens of times – but we had never had a situation like this.  What was supposed to be an engine in need of fine-tuning was instead a box of rocks. There was nothing useful in any of the code that had been developed. Nothing.

The system upgrade, which had already cost over $1 million, was a bust.

The firm knew the software wasn’t as ready as the vendor had reported, and was prepared to hear they should expect further delays. But they were stunned to learn the whole thing was completely unusable.

Still in need of better software, however, the firm quickly regrouped and adjusted its strategy, asking us to salvage what we could, cobble together something functional, and chart a long-term solution.

We quickly re-coded the most critical parts of the system and salvaged the data, bringing a core slice of the system to life. Then we began professional development of a scalable and extensible system that would support the firm’s ever changing needs and increasing opportunities.

And it works! Today, after the rescue, this version supports the firm’s entire ecosystem of investors, companies and prospects, and puts risk-reducing analytics into the hands of its partners as they manage their investments and survey the ever changing world of innovation and opportunity.

How Did This Happen?

How could a custom software build of this magnitude and complexity get so far down the wrong development path?

It’s easy in retrospect to point the failure finger solely at the original vendor, but the firm also recognized its own responsibility in this disaster. In our post mortem, we learned that there had been little to no project management discipline and the communication between the original vendor and the firm had been poor.

The firm had not required interim milestones, nor insisted on rigorous communication and status updates. This left the vendor free to develop without the firm’s guidance or input, and the project became more the vendor’s than the firm’s.

Lessons learned:

  1. Business first, technology second. Ensure your vendor has appropriate consulting experience, and can articulate an understanding of your business challenge and how to apply technology to meet your needs. Developers are wired to build, and left unchecked, will build and build and build.
  2. Validate progress and completion with data, not reports. So when your vendor reports that the job is 95% completed, you have already defined exactly what “completed” means. Include coding, testing, and customer approval in this agreement.
  3. Own the Process. You make the final determination on project completion – and success – not your vendor. Don’t wait until the end, or even the middle of the project to figure out what the status of your own project is. Engage in your project from the outset and arm yourself with knowledge.
  4. Take little bites of risk. Focus on the riskiest pieces of your systems first and get them up to speed. Then invest piece by piece and you’ll lower your overall risk – and avoid a catastrophic failure.

Image Source

Comments are closed.