The absolutely coolest thing in science and engineering is machine learning, when computers learn from the experience encoded in data. I shall now support that hypothesis.
Note: This article is based on a transcript of The Dr. Data Show episode. View the video here.
We humans build and explore. We build tall buildings and airplanes — and we explore down to the depths of the ocean, to the microscopic sizes of quantum particles, and then back up high into space so far there’s no air to breathe. When I was a kid I wanted to be an astronaut, but now I realize outer space is a vacuum and vacuums totally suck.
Transcending all that, the ultimate “final frontier” is for us to make use of our automatons (computers) for the most all-encompassing of tasks: getting better at tasks — that is, learning.
Computers can learn to perform well — at games, at identifying objects in photographs, at medical diagnosis, and at predicting behavior — of customers, employees, voters, convicts…
Now, many things are simply impossible to predict with high accuracy. We can only at best put rough probabilities on whether a certain person will… click, buy, lie, die, or any other outcome or behavior — even if we had a reading of every neuron in their brain. Computers of course usually know far less about a person than that — normally a few dozen or perhaps a few hundred details about the person and their prior goings on.
But even lousy predictions are super valuable. Just predicting better than guessing helps organizations improve decision-making at a massive scale. For example, if 5% of your customers cancel each month, knowing which of them are three times as likely to do so, that is, who have a probability of 15% to cancel, makes a world of difference for the marketing, making it more efficient and effective and in some cases literally multiplying the profit many times over. And the same exact thing applies for predicting which patients are at a greater risk of dying — that empowers more efficient and effective healthcare.
Now, when machines learn to make these judgement calls, it’s basically like a kind of magic. Here’s what I mean. The machine’s got to take into consideration all the dozens or hundreds of factors known about a case, situation, or individual. Like, for predicting the future success of say a new startup company. Given a long list of factors such as about the founders and their history, the nature of the business, etc., how should the computer weigh or combine all these factors to calculate the most precise probability as to whether the startup will succeed? More to the point, how could the computer automatically learn to do that?
Well, you give the computer training data, a long list of prior examples of startup companies and whether each one went on to succeed or fail. And then machine learning discovers what it can from that data. It’s an automatic trial and error process that derives, and tries out, and keeps on modifying and improving patterns or formulas that will help predict for new startups where — of course — we don’t yet know what the future outcome will be. In this way, the computer is basically programming itself.
What makes this process “magic” is that the generalizations it draws from past examples turn out to work — to hold true — when applied to new, never-before-seen situations. The technical word for this is induction: the act of generalizing from examples, of leaping from a set of particulars to universals. When your computer does this, that means it’s capturing truth about how the world works, discerning the means behind the madness of the universe. It’s a kind of scientific discovery, done automatically.
Now, there’s an art to all this, to taking on this ultimate challenge. To engineer and design these learning methods, you need not only some clever math, but also a kind of art, by which I mean informal, human intuition. The field of machine learning couldn’t succeed without that human creativity.
And succeed it does. So I hereby designate machine learning the most interesting, fascinating, awesome, and promising branch of technology, period.
And, let’s take it a step further. It ain’t just a neat idea on paper — machine learning is extremely valuable and, I would dare say, important. It’s the latest evolutionary step of the Information Age, where we’ve
moved from the application of engineering to collect and manage bigger and bigger data, to the application of science to learn from that data, from all the experience encoded in that data.
The ability to predict behavior and outcomes better than guessing — which relies on data and machine learning — is the Holy Grail for all the main functions undertaken by companies, governments, law enforcement, and hospitals. With prediction, you combat risk, fortify healthcare, boost sales, conquer spam, streamline manufacturing, toughen crime-fighting, and win elections.
Every day, machine learning affects all of us, ‘cause it actively drives millions of decisions as far as whom to call, mail, test, diagnose, investigate, incarcerate, set up on a date, and medicate. Harvard Business Review calls machine learning “the most important general-purpose technology of our era.” It’s the coolest.