In this episode of Mining Your Own Business, Evan Wimpey chats with Eric Siegel, bestselling author and founder of Machine Learning Week. Tune in as he shares why businesses need to focus on machine learning projects that work in the real world.
Eric also dives into the importance of measuring the impact of machine learning projects, the need for business professionals to understand the technology, and the potential challenges associated with overhyped AI expectations.
Learn more about why we created the Mining Your Own Business podcast.
Eric Siegel | GuestEric Siegel, Ph.D. is a consultant, former Columbia University professor, and founder of Machine Learning Week. He’s also an instructor for the “Machine Learning Leadership and Practice” course, executive editor of The Machine Learning Times, and a sought-after keynote speaker. Eric authored the bestselling book “Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die” widely used in university courses. His latest book, “The AI Playbook,” is now available at bizml.com.
Eric’s interdisciplinary work bridges the stubborn technology/business gap. At Columbia, he won the Distinguished Faculty award teaching graduate computer science courses in ML and AI. Later, he served as a business school professor at UVA Darden. Eric has appeared on numerous media channels, including Bloomberg, National Geographic, and NPR, and has published in Newsweek, HBR, SciAm blog, WaPo, WSJ, and more.
Follow Eric on LinkedIn
Evan Wimpey | HostEvan Wimpey is the Director of Analytics Strategy at Elder Research where he works with organizations to transform deficient data into tangible business value that advances their mission.
He is uniquely suited for this challenge by pairing his professional experience in management and economics at high-functioning organizations like the Marine Corps and Goldman Sachs with his technical prowess in data science. His analytics skillset was strengthened while earning his MS in Analytics from the Institute for Advanced Analytics at NC State University.
Evan almost always has a smile on his face, which is at its widest when he is helping organizations use data in innovative ways to solve complex problems. He is also, in a strictly technical sense, a “professional” comedian.
Follow Evan on LinkedIn00:00 Introduction
01:02 Diving into analytics use case examples
05:56 Measuring the effectiveness of analytics efforts
06:49 Discussing Eric’s new book and using metrics to drive decisions
11:37 Talking about best practices around end-to-end processes
14:13 Discussing the need for useful ML models and change management
16:26 Exploring the need for stakeholders and data professionals to understand each other
18:28 Talking about the Machine Learning Week conference
22:58 Delving into Eric’s predictions for future advancements and generative AI
27:24 Chatting about Eric’s dream analytics project and current projects
32:00 Wrapping up the show
Evan Wimpey: Hello and welcome to the Mining Your Own Business podcast. I’m your host, Evan Wimpey. And today I’m super excited to introduce Dr. Eric Siegel. Eric has been a big name and a very impactful name in the analytics and AI space for quite a while now. He’s been a professor at Columbia. He’s taught at UVA’s Darden School of Business.
He has founded one of the most successful and long running machine learning conferences, the Machine Learning Week series of conferences. It’s got all kinds of writings and interviews out there, most notably a couple of books. He literally wrote the book on predictive analytics: “Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die,” which is just a great title.
And a new book that will AI Playbook, Mastering the Rare Art of machine learning deployment. We’re super excited to have Eric on the show to chat with him today. Eric, thanks so much for joining us.
Eric Siegel: Thanks for having me, Evan. Great to be here.
Evan Wimpey: All right. I want to jump in with a fun one—with an interesting one.
So you’ve been in the space for a while. You’ve seen a lot of use cases. You’ve seen a lot of change in the past couple of decades in analytics. Maybe we’ve got sort of the stereotypical use case of analytics, you know. Who’s going to click, who’s going to click on this ad or who’s not from the subtitle of your book, but do you have any, do you have any sort of favorite or interesting or maybe not as intuitive use cases where analytics has been applied that, that you really like?
Eric Siegel: Yeah. I mean, we always see new use cases crossing our desk presented our conference. You know, what do you predict? What do you do about it? That defines the use cases. The ones that excite me the most are when sort of older lines of business, brick and mortar financial service processes that have been established for many decades when they adopt and successfully deploy a model deploy machine learning to improve their operations.
So a couple that come to mind that I’ve actually covered in my new book, the AI Playbook, are UPS. And these are some stories that are pretty big, but haven’t gotten the attention they deserve. A lot of people haven’t heard about it. UPS actually predicts which packages are going to need to be predicted delivered tomorrow in order to optimize how they pack and plan the routes for all the delivery trucks.
So it’s sort of that last mile of the delivery or last umpteen miles of the delivery process from the shipping center on the trucks. So they need to start loading the trucks before they know all the packages necessarily that are going to get delivered. They start loading the trucks overnight or very early in the morning.
We need to plan and start the loading process and, and in order to do that more optimally, they need to make these per package per delivery predictions. And by doing that they actually—and getting it deployed across all the shipping centers across the entire United States. They save an estimated 18 and a half million miles a year.
That’s an estimate. It’s actually 10 percent of a larger savings of 185 million miles a year when you combine that with a complementary effort to actually plan the particular driving routes more effectively. And that also included 8 million gallons of fuel saved a year and 185,000 metric tons of emissions.
So I love those kinds of concrete use cases. We’re an established process practice. They had to take on change management, not just do the analysis, but actually deploy it. That’s actually the focus of my new book and not just my book, my career. I’m taking this on as a mission that there’s a lot of excitement and hype about the core technology, but it’s like we’re more excited about the rocket science than the actual launch of the rocket.
And the rockets very often routinely fail to launch so that you don’t actually end up capturing value. from the machine learning part. I also mentioned the other example, FICO. FICO is very famous for their credit score. But what’s little known, and I think is an even bigger part of their business, is their fraud detection model, which is called Falcon, that’s used by two thirds of the credit cards.
So the banks that control two thirds of the world’s credit cards, 90 percent in the U.S. and in the U.K., in real time, use that one model by FICO, called Falcon, to instantly decide whether to authorize each payment card transaction, , on the fly based on this fraud screening, this fraud detection model, and that’s just an amazing overhaul of the fraud detection process, and it only works because all the banks that are customers that use Falcon actually contribute data as a consortium of banks to FICO.
So these kinds of stories where you get. People, you get established enterprises and established industries to change their process. It’s a, it’s great with the big five tech have done ordering search results as you mentioned targeting ads, translating languages. That’s incredible. But when we know we’ve really made it.
As an industry is when outside of that sort of elite small niche of society, which is sort of the online world and you go to bricks and mortar, you go to establish financial services and now we’re seeing those things get deployed for real and getting measurable results, incredible value. In fact, I’ll just mention for FICO that Falcon model has, I have my notes here, reduced fraud losses in the U.S. by 70%, which amounts to around 20 billion a year.
Evan Wimpey: Yeah, that’s incredible. Those, both the UPS and the FICO metrics, that’s good. One of the things I like about those two use cases as well is it lends itself to explicit metrics. Sometimes it’s really hard to measure, you know, did this machine learning, did this analytics effort actually help anything?
You know, certainly when, when you’re trying to get it off the ground or we want to be able to anticipate that. So being able to have those hard metrics is great as well.
Eric Siegel: Metrics, metrics, man, dude, it’s metrics. If you’re not measuring the value, you’re not pursuing value. Measuring the value is just as fundamental as developing the model.
And it goes hand in hand. The two are coupled. There is no difference. But that second part, which somehow is less sexy than the incredible rocket science that is machine learning, which is why I got involved in the first place. I think it’s so cool to learn from data. I love the technology. But we love it too much.
Evan Wimpey: Very fair. And I think, you know, you’ve, you’ve mentioned your, your new book and, you know, that’s really why am I super excited to talk to you today because your new book speaks directly to the folks who, who, who are our audience here in the podcast, we’ve got these rocket scientists that are building these great ML tools, we’ve got organizations that.
Need to make better decisions to reduce their fraud or their miles driven. How do we plug them together? Where, where’s the connection? And you sort of alluded to this and in, in, in the idea behind the book, but can you sort of talk about maybe just a super high-level overview of what the book is the new book, “The AI Playbook” and your motivation for writing it.
Eric Siegel: Yeah, well, my motivation is that models don’t get deployed. Projects flop. Now Elder Research and my partnership with Elder Research for decades now with the conference, some projects, and my personal relationship with John Elder, who’s been a great friend and mentor to me. That’s really helped me congeal what’s happening here.
I think Elder’s way ahead. Elder Research is way, way ahead of the curve. I love the name of your podcast. It’s also the name of your CEO and Jeff Deal’s book, “Mining Your Own Business. So the point of the book is to formalize and establish the enterprise practice, the process for running a machine learning project from end to end—from the inception to its deployment.
So it maps as a six-step process that culminates with the three technical steps that as data scientists were also well familiar with prep the data, train the model and deploy it. But before that, you need these pre-production steps which is establishing the prediction goal, the deployment plan, and the metrics exactly which metrics how you’re going to measure them.
But the bigger picture here is that we need an end-to-end practice that’s well known and established, not just to data scientists. But to business leaders so that everyone participating can collaborate deeply, be familiar with the practice, and even those non data scientists need to ramp up on semi-technical content, the idea of what does it mean to predict something per individual to use those predictions, usually in the form of probabilities.
And then how are they going to integrate into operations? And then lastly, metrics. How do you measure the performance of a model? How do you understand as a non-data scientist—and put numbers on and quantify—the performance of a model, even though the model, you know, perhaps predicts less well than a magic crystal ball, and yet it’s still potentially quite valuable by essentially predicting better than guessing?
How do you quantify that? So we need to ramp up business stakeholders on that semi technical on the end and practice, and we need to establish a practice. So my book introduces a new buzzword that I’m here to promote today, and the buzzword is BizML. BizML. So unlike some of the other buzzwords like AutoML and MLOps, which focus on technical practices and technical solutions, this is an organizational process.
That goes through those six steps and makes it clear to business stakeholders what semi technical knowledge they need to understand very concretely how these predictions are going to help and how much they’re going to help how much value it’ll deliver so that they can in an informed manner Participate and collaborate closely with data scientists with [data professionals Throughout the project from its inception to its actual launch and that that launch can be planned from the get go So the book’s actually structured around that practice.
And so I’ve made the website based on that buzzword. The website for the book is bizml.com, and although the book hasn’t been published yet, it’s coming soon—you can get the audiobook immediately. We have a special deal, pre-order the book as a hardcover or an e-book, and then you can immediately get an advanced copy of the audiobook version.
So go to bizml.com, and you can check out how to do that.
Evan Wimpey: Awesome. That’s great. I love the term there. We’ll make sure to put a link in the show notes, wherever you’re listening, you’ll be able to click. I think you’ll probably be able to remember bizml.com. So I mean that’s a buzzword, right?
Eric Siegel: That’s a buzzword waiting to get famous and adopted. I mean, do you know how long it took me to come up with those five letters? Just the right buzzword. BizML, not that long, but it’s catchy, right?
Evan Wimpey: Yeah, now we’re happy to introduce the buzzword here, but I’m going to ask the pressing questions.
Was this your invention or was it ChatGPT came up with BizML?
Eric Siegel: With BizML? I mean I’ll never tell. No, no, I did. I was playing with letters and syllables and endlessly in my Emacs text editor. I’m just like—
Evan Wimpey: Finding a domain name that’s available.
Eric Siegel: You know, I’m not one to coin buzzwords. So this was an exception. I’m like, we need a word for this. We need this to, so that, so that it can be branded and it can catch on and people can say, Hey look, we’re going to adopt this best practice. It’s one thing to know that there should, most senior data scientists know that there is or should be an established end to end enterprise practice.
Most junior data scientists probably haven’t quite wrapped their head around this. But the fact is, most business stakeholders who are touched by or should be involved with the deployment projects of machine learning haven’t really come to realize that there needs to be a specialized business practice for running, running a project.
So even the existence of the need for that thing kind of needs to be branded—needs to be broadcasted. So that’s why I worked hard for the five letters, BizML.
Evan Wimpey: Love it. Yeah, I think it’s great. And I really like the, the sort of juxtaposition with MLOps, which has been a really popular term, almost like with the idea of focused on, well, this is how you get it deployed.
But you’re right. It’s very much the technical aspects. How do you build the infrastructure? How do you have the data?
Eric Siegel: It’s not, it’s not the tail. That’s the tail wagging the dog. You need the tail. It’s absolutely critical. MLOps, you know, provides incredible value and is necessary for the infrastructure to support and deploy and manage models.
But more than that, you need stakeholders understanding what the heck is going on so they can, in an informed and reasonable way green light the actual deployment when it comes to that. It doesn’t catch them by surprise. Oh, you want to change my big—all my major operations—my larger scale operations, especially those that are real time.
And you want to make an actual change based on probabilities. This all seems so arcane. But it’s not the rocket science part, right? It’s the semi technical part that you’re using what the rocket science gives you, which is predictions. A lot of people are not comfortable with probabilities. So there’s a bit of ramp up, but if we’re not acting on rock probabilities, we’re not, you know, deploying the best of science to improve business.
Evan Wimpey: Yeah, absolutely. I suspect that it’s really all parts of BizML really, really need to be followed to, to get deployment off the ground. But is there, is there a place you’ve consulted with a lot of companies? A most common sticking point or a, you know, a toughest sticking point to get a, to get across, to get a business, to be able to deploy usefully.
Eric Siegel: Well, yeah, I mean, it’s, it’s, and it’s something, as I’ve mentioned, I’ve learned so much from Elder Research. Like, so Gerhard Pilcher, your CEO and Jeff Deal at Elder, who coauthored the namesake of your podcast, Mining Your Own Business. I quote that book because it says that, right? There you go.
And by the way, Mining Your Own Business is a joke that I had in my slides way before they wrote that book. I’m sure it’s a coincidence, and I didn’t make it up. An old buddy of mine was like, Mining Your Own Business, he also came up with, What’s mined is yours. So I had those in my instructional slides many decades ago, especially when we were calling it data mining.
So I actually quote the book, “Mining Your Own Business,” where they say something along the lines of, Hey, the problem is that people aren’t planning from the get go con and socializing concretely exactly what deployment will entail, you’re going to change these large scale operations.
It’s a change management issue. I mean, that’s a big thing. Change management is not a new concept. Everyone knows it’s challenging. Everyone knows there can be ups and downs with it. Real source of the problem here is that people don’t conceive of machine learning projects as change management projects.
It’s almost as if by calling it a data science or a machine learning project, we’re actually hurting the potential value because really it’s a business project, an operations improvement project that uses machine learning as a key technical component. So maybe we should rename it. It’s a little bit less sexy sounding, right?
So there’s a problem here. We need another cool buzzword for it. Yeah,
Evan Wimpey: I would be a BizML scientist for sure. Although, I mean, you bring up a very good point. Like I work with data scientists and the front of their mind is what is their job? It’s to build models to do analytics, but very much. That’s not it. That’s sort of the means we’re trying to solve problems.
Eric Siegel: Yeah, it’s always someone else’s job. So just sort of to simplify the world, we have business professionals, data professionals, including a data scientist. And they both kind of think it’s the other people—in the other people’s court.
So a data scientist can be like, I make a model. It’s, that’s my job. I’m the technical expert. I’m the quantitative expert. And my job is to make a great model. And it’s value. It’s obvious. It’s a no brainer. Of course, the organization could or should deploy it and getting it to deployment, you know, maybe technically I help with that, but getting the stakeholders to understand the line of business managers, the people running the operations and getting them to approve it.
Even if it’s an executive who needs to make the call, you know, the practice, the business management practice that you need for that. That’s not my job. In the meanwhile, on the flip side of the fence, we’ve got business professionals saying, Oh, all that stuff, the idea of semi-technical information, what’s predicted, how well, and what’s done about it.
That’s, you know—I delegate all that stuff. That’s what I have data scientists for. So everyone’s pointing elsewhere. You know, and the business professionals say, I don’t need to know how internal combustion works in order to drive a car, right? I can just operate the car, which is true.
But that’s not where the, that’s not how the analogy holds. To drive a car, you do need to be an expert. You need to know friction, momentum, how the car operates, the rules of the road, what to expect from other drivers and what they expect from you. That’s a great deal of expertise. And to drive a machine learning project, you also need semi-technical, not rocket science expertise on what it means to integrate and implement probabilities.
Evan Wimpey: I absolutely agree. And Eric, you’ve, you’ve been in a space, certainly consulting and speaking with a lot of businesses, but you’ve also been on the academic side and maybe not really academic, but in the conference world and machine learning week, there’s a lot of people that come there to learn.
There’s a lot of people that come to share knowledge. Have you seen through this? Maybe not the term BizML, but have you seen any focus on You should learn the business or business people. You should learn some semi technical knowledge.
Eric Siegel: So in recent years, we’ve usually—at the U.S. conference—we’ve usually had about seven tracks.
And the first of those is, has been, and this is by my design has been the operationalization, you know, deployment, business management prep track. It’s basically just called operationalization, but that track really has been the most popular in general. I mean, Depends on the session and the date, time of day during the conference, but in terms of the number of perspective speakers, it’s a pleasant surprise.
There’s so many people who have things to say, we end up packing that track mostly with 20-minute sessions so that we can fit them all in there. And then there’s lots of times where there’s standing room only in that room, depending on the room allocations. So yeah, that’s gotten good traction. I would say in the academic world though, they’re still stuck at a local minimum here, and that the business practice is only mentioned as an afterthought for the most part.
And that that does need to change. In fact, that’s sort of the, the minds of data scientists starts from the get go. The burgeoning data scientist is like, oh, the only thing I want to do is load the data and start modeling, as if the data prepared itself, as if all the pre-production business decisions, and buy-in, and deployment planning just got skipped over.
That all gets manifested in the data prep, which is again, which is, as most data scientists know, a bigger technical challenge, at least in terms of calendar days than the core modeling itself. No, look at the end-to-end practice. We’re trying to improve business operations. We’re trying to provide actual business value by implementing change.
That means you can’t just focus on this really awesome, exciting rocket science as much as we’d like to. In fact, when it gets to that chapter in the book. So my book, “AI Playbook,” is organized around the six steps. So to establish the deployment goal, the prediction goal specifically, and the metrics.
And then the three production steps. Prep the data, train the model, and then deploy it. When it gets to that. So this is an ML book where ML comes almost last. It’s step five. It’s literally chapter five Right. So if you include chapter zero, which sort of gives the overview in the intro, right? It’s sort of the—it’s like the seventh chapter in the book that you finally get into the core modeling methods and that chapter starts [out by saying if you’re like me, you probably skipped right to this chapter.
No, no, no, no, no. You have to pay your dues first, right? This is the fun, exciting part. If you have read up to here, all the previous chapters, great, now you have permission to have fun. And then that kind of unpacks, in an accessible way for business readers, sort of all the ins and outs of the predictive modeling step and gives an overview of that.
I also—I generally pride myself in presenting concrete knowledge about modeling to non-technical readers in my presentations in both of my books, so it does get concrete there. So the whole world is so enamored with modeling methods, but we need to start first with the business value proposition and all the ins and outs it’s going to take to actually get there.
Evan Wimpey: Absolutely agree. Yeah. Yep. Love it. And I also love that your, your book is zero indexed that you start with, with chapter zero. So hopefully
Eric Siegel: that’s good. Well, yeah, yeah. I mean, we’ll have to do that.
Evan Wimpey: So, so you’ve written this book, hopefully, hopefully this, this term and not just the term, but the concept really gets adopted.
And I think folks will be able to see a lot more value in ML as it does. Can you look? Is the man who wrote the book on predictive analytics. Can you make some predictions for us in the next five years, 10 years, either in the business organizational place or in, in the technical advancements, things that you think will happen technically in AI and ML?
Eric Siegel: Yeah. So, I mean, I think that there will be an AI winter in the year.
Evan Wimpey: That’s all the time we have, folks. Thanks.
Eric Siegel: You know, I don’t know. I think anyone claims and knows when the A.I. winter’s coming, but it’s less than 10 years. Right. There’s so many variables. It’s like predicting when there’s going to be an economic recession. I mean, there’s so many reasons to think it could happen in four months. And there’s so many reasons to think it’s going to be postponed more and more, but I believe it’s coming.
Generative AI is incredible as somebody who spent six years when I was a Ph.D. student at Columbia in the Natural Language Researching Group. What it’s capable of is beyond anything I expected to see in my lifetime. And because of that, it’s going to continue to postpone the AI winner. But the winner is not about how cool the technology and impressive the technology is.
It’s about its captured value in organizational operations. And I’m not saying there’s no value in generative AI. But let me put it this way. As astonished and how much I want to geek out on what it’s capable of doing when you play with a large language model. The world is basically ten times as excited as it should be.
Right? It should be really excited, but instead it’s really, really, really, really excited. I mean, and it’s, it’s sort of, it’s, it’s implicitly a lot of mismanaged expectation management and sort of overpromising because despite the thing seeming human, that’s a very big difference from the idea that it’s taking a step towards what you know they call AGI.
And I think that the AGI thing, Artificial General Intelligence, underlies, even if not entirely consciously in many cases, really underlies the hype because of the sense of this thing is getting so human like it’s going to super be as least as good as a human any moment now or relatively soon. Right.
And I think that that’s a myth. So I don’t think it will be able to capture value the way people are imagining it as valuable as it is for many things like, you know, could help a customer service agent doing a chat room. You know, here’s a paragraph that you could type to the customer now. Oh, that looks great.
I’ll just copy paste that, right? So they’re doing studies and they’re seeing that that actually improves enterprise efficiencies in those types of processes, right? Whereas does that necessarily mean it’s going to compete with what the enterprise efficiency use cases we’re already looking at with machine learning where all the main large scale operations we do for targeting, marketing, fraud, ad targeting, search results—all those ways that machine learning adds value today in those operations, you don’t need large language models, you need prediction.
You can’t predict like a crystal ball, you can predict better than guessing. It doesn’t matter how fancy the model is, whether it’s human level capable, we don’t have clairvoyance, we can’t compute, can’t expect computers to do, there’s a sort of an upper limit at some point there of how well it can predict, because we don’t have magic crystal balls. But what there is, is a lot of potential value with making predictions. And that’s where driving large scale enterprise operations, basically at the level of decision automation or decision support, has great value. So, I think that this spells recipe for, lack of a better word, disaster?
I mean that’s what an AI winter is. But I can’t really say when. We’ll look at, we’ll talk offline. You know, when you say, when you, when you make a more concrete prediction, you have to think of it in terms of the false positive and false negative costs. Right. And obviously the false positive cost is really, really low because that’s what prognosticators, you know, and soothsayers and, and, and, and, and, and futurists.
Right. My prediction is that futurism will be entirely out of style within five years. That’s a, that’s a meta joke, but that’s what futurists have discovered that the false positive cost is very, very low.
Evan Wimpey: That’s fair. And the true positive benefit is really high. Look at this great prediction I made that was hard to make.
Eric Siegel: That’s right. Exactly.
Evan Wimpey: Awesome. So away from predictions and maybe, maybe back to the practical, the real, you’ve seen a lot of cool things. You’re working on a lot of cool things. Let’s say you’ve got, you know, BizML takes off, and Eric Siegel gets to just retire and do whatever he wants with this time, which may not be too far from the truth right now.
But if you could work on any analytics type project that you wanted to, you had all the resources that you needed, the help, the support, you could talk to the right people and you just got to work on whatever project you wanted. What would you enjoy doing? What would you want to want to work on?
Eric Siegel: Well, I am working on it and it’s the theme behind the book and everything else I’m doing, which is let’s get these things deployed. Let’s get actual value. Let’s get the actual value captured. Let’s get organizations running more effectively, not just talking about it, not just doing the analysis towards that.
And I love that mission. I think that Elder Research has always been implicitly doing that and tracking that performance in their own client facing. Okay. Projects much more so than any other organization I know, but very few in general are doing that. And, you know, there’s going to be a point where the executives get fed up and say we’re not getting returns.
I mean, right now, so IBM recently came out with industry research results showing that there’s no returns to be specific, that the average returns from AI projects are lower than the cost of capital. That doesn’t mean we aren’t having plenty of amazing successes. But on the whole, there’s this sort of endemic of failure, and I believe that’s mostly because they fail to deploy.
When you pull, when you survey the data scientists, which I’ve now done in two rounds, most recently, most recently, excuse me, most recently, along with Rexer Analytics, as part of my one year analytics professorship at Darden School of Business. Same thing. The data scientist makes models, and they just don’t get deployed.
I mean, let me give you some concrete stats. Only 22 percent of data scientists, when they’re talking about new capability initiatives, say that their models usually deploy. 43 percent say that 80 percent or more fail to deploy. This is only specifically, in the way we phrase the question, models created with the intention of being deployed.
So it’s this disappointment. Oh, the stakeholder gets wet feet, basically. There’s a sort of lack of planning and socializing and concrete input from the business side, essentially. So, if I was going to do anything I wanted, that would, because I was now able to retire, actually the result is the opposite of retirement.
So I will now officially say for the first time publicly, so we consider this a soft launch. We’re still in stealth mode, there’s no info, but I am co-founding, have co-founded a startup, and the name of the startup is Gooder AI, like make your AI more gooder and we’re focused on the metrics part, right?
So if you talk, if you’re looking at this enterprise practice, which I call BizML and the need for that, that’s a human endeavor, it’s an organizational endeavor, but there is a missing technical component to that, which is the evaluation piece. And right now models are evaluated in terms of their technical performance, like lift accuracy, which is usually the wrong metric area under the curve, which is a notorious measure, which also similar problems to accuracy. And technical metrics have value for data scientists, but they only tell you the relative performance like comparison to the baseline, like guessing and in comparison to a upper limit, like a magic crystal ball, you know, how good is the model?
How well does it perform? Relatively speaking, in contrast to its absolute business value, which would be business metrics like profits, number of customers saved, right? Number of dollars saved, number of driving miles saved by UPS. Those business metrics are very rarely put in place. So we’re working on a solution for evaluating models that involve business metrics and make it accessible and understandable to business stakeholders.
So it’s a key component to what needs to happen across the organization to plan. And execute machine learning projects so that they actually deploy successful.
Evan Wimpey: Very exciting. Yeah. It’s not just a hypothetical. You’ve got an answer, and you’ve got a plan for it. That’s great. And thanks for sharing it on the show along with brand new buzzword BizML, which people will find in your new book if they go to bizml.com. Eric, is there anywhere else people should go to find you follow your work or is it all BizML?
Eric Siegel: Well, business mail is there. There’s a page about me on that website. You can also click the banner at the top if you want to get the audio book immediately, which is that free offer if you preorder the hardcover or the e-book.
Evan Wimpey: Awesome. Well, we will link to that in our show notes. Eric, thanks so much for coming on the shows. Delight to talk with you today.
Eric Siegel: Great speaking with you, Evan. Thanks so much.