“Things we don’t know we don’t know” in IT

“There are known knowns. These are things we know that we know.

There are known unknowns. That is to say, there are things that we know we don’t know.

But there are also unknown unknowns. There are things we don’t know we don’t know.”

Donald Rumsfeld’s quote has been somewhat ridiculed, but I think there is quite a lot of truth in it, at least as regards release of new IT systems.

We have just completed  the successful major upgrade of an important system.  We had of course tested and tested and then tested the new system again,  We had worked very closely with our business colleagues on designing the process and the training.  And then we had tested it all again.

That is to say, we had worked hard on the “things that we know” and the “things that we know we don’t know”.

However, it is indeed the “unknown unknowns” which you cannot predict that you have to be prepared for.   In IT, it is the concatenation of issues and the interaction of different systems that give you some of the hardest challenges.

With an IT release or upgrade or migration, you have to be ready for the unexpected: this is because you can’t test everything, you can rarely simulate in full the launch of new systems and functionality into live conditions.  Indeed, during our cut-over it was the behaviour of the new user-portal under full load, and the interaction of external databases and the main system that provided our challenges.  One issue turned on the interaction of logic buried deeply in the code of the new system release and external database.  It only appeared when they interacted and were run together under a particular set of circumstances.

I have long held that the quality that determines the excellence of an IT department is how it deals with unforseen and unforseeable issues, since problems and bugs and interactions will always occur in complex environments.

So you need constant vigilance during a cut-over, and access to the best experts who not only understand the systems themselves but also the interactions.   Cool-heads are also essential to diagnose the problem and identify the root cause, and above all not rush towards obvious solutions.   Always consider “what you don’t know…the unknown unknowns.”

About paulcoby
I am CIO at the John Lewis Partnership in the UK. I was Chair of SITA - the airline solutions company owned by the Air Transport Community - for 11 years. I am also on the Boards of Clydesdale and Yorkshire Bank and Pets at Home. Previously I was Head of BA Services and for 10 years CIO at British Airways. I am interested in Roman and Military History. The views expressed are entirely my own not my employers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: