The Customer is Always Wrong – Along with the Rest of Us

Before you fire me as an arrogant consultant hear me out. I love customers! They’re vital to my survival. I need to listen closely to them, learn from their expertise, and partner with them to succeed. And I love living in a culture where maxim “the customer is always right” is widely followed, and usually leads to great vendor service and responsiveness.

But Data Science results are not a commodity like a lunch or widget order; they can only be obtained through a creative and iterative discovery process. Virtually all our projects end as a strong success, despite (or due to?) most experiments and hypotheses failing along the way.

So, I could title this “We Are Almost Always Wrong”. Why am I highlighting the other party? Because over-fealty to the customer’s guidance can prevent us from discoveries even more valuable to them. The customer may know their pain well, but not the diagnosis or treatment. We must “trust but verify” as we discover how best to help them.

My First Story

I had a great early customer in the Southwest who’d built his mail-order company from scratch. He’s a colorful former fighter pilot who’d earned a technology PhD late in life to help him with his business – an ideal early-adopter client for the Age Before data science was well-known. He hired us to improve their lift from a wide variety of mailing lists and we made good progress. But we noticed an inconsistency in the prompts (catalog mailings) different prospects received; cross-checks we developed didn’t add up as we inferred they should.

I proposed we spend time (and money) to figure this out, only to be met by, “Stick to what I’m paying you for, d*** it!”

So, what should we do?  Obey the client or our judgment?
(The best answer is both, though we risk not being able to charge for the latter.)

Our Discovery

Following the “loose thread” led to our discovering that their merge/purge service (where disparate mailing lists are turned into a single file) was wrongly removing all international addresses. Fixing that flaw, and using our improved prospect scoring, doubled their sales per catalogue. Our willingness to push back (and its success) strengthened our connection with the client.

My Second Story

We were once tasked by a vendor of a very complex software product to study user sessions to find such things as:

Their software allowed each user to opt-in to record all their keystrokes for research purposes. A minority volunteered, and that complex, variable-length data had been stored for years without anyone looking at it. After creating a way to ingest and visualize it in many ways, we found something that was far more important than anything we’d been tasked with – so important in fact, that it immediately halted the project.

Our Discovery

We discovered that a sizable minority of the users had not paid for the software! (And that was just of those who’d volunteered to be tracked.) The client immediately worked to shore up this deadly breach to their business, then re-authorized the project with new appreciation for our work.

Isaac Asimov was one of my favorite authors when I read Sci-Fi voraciously as a youngster. Perhaps the seed for my fascination with data science was planted by his award-winning Foundation series where a genius with a world-sized computer could predict the future with uncanny accuracy. But that novel really gets going only after a powerful anomaly sends galactic events spinning in a scary new direction.

Relatedly, in his professional scientific writing Asimov opined (which still gives me chills):

“The most exciting phrase in Science isn’t ‘Eureka!’;
it’s ‘That’s odd…’” 

We must be on the lookout for important data artifacts completely unexpected by everyone — client included. Note that, to feel surprised, one needs to have expectations – to understand what “normal” looks like.

There are so many examples of how unanticipated discoveries of great value can only be found by immersing oneself in revealing data, with guidance from a client steeped in its meaning. But don’t limit yourself just to client instructions. They hired you precisely because they don’t know exactly what to do. Find out what matters to the business and be meticulous, creative, and relentless in searching for value.

Finally, unexpected discoveries best highlight how Data Science is best performed by motivated, caring, experienced humans, and can’t be replicated by any algorithm, no matter how fancy. You can’t program what you don’t know. But you can turn loose expert explorers and expect fantastic results.