This blog is part of Aryng’s resource for analysts, those looking to transition their career to analytics or business leaders looking to improve their success with predictive models.
In 15 years of experience, I have seen countless predictive models—but very few useful ones. Most take too long to build, and then sit unused on the shelf. The model becomes a memory, and a bad one at that. Behind the scenes, countless hours were spent figuring out what to model, getting agreement on definitions and parameters, building the model, optimizing it and preparing for the presentation to senior management. Then “boom”: something goes amiss, all the effort goes down the drain, and yet another model meets its arch nemesis—the shelf. Worse, the analytics group’s reputation suffers for “not producing anything”.
So why do some models succeed and many fail? What are the traits of the successful model and how can you build one?
Three levers to build successful predictive models
- Successful predictive models need pre-work. I am a big proponent of using simpler methods, such as correlation analysis, to solve a business problem before moving to more complex methods, such as regression and decision tree. Pre-work with simpler methods not only delivers low hanging fruit quickly while the more complex method is being considered, but it also qualifies the opportunity. Pre-work determines the model’s likelihood of delivering greater results over the simpler method and the potential for an ROI.For example, consider the case of a company’s customer churn analysis. A simpler method—a correlation analysis—reveals the top drivers of churn and addressing those brings the churn percentage close to the industry average. Since very few strong hypotheses remain on possible causes of churn, likely most of the churn now is from uncontrollable factors. In this case, predictive modeling would likely be a wasted effort and should not be undertaken. Unfortunately, most analysts don’t do this pre-work and immediately begin working on a model doomed for the shelf.
- Successful models are built with organizational buy-in. You can build the best model with significant uplift and revenue opportunity, but unless the organization is ready to adopt and operationalize the model or the insights, those gains can’t be realized. Also, predictive models are resource and time-intensive and, to top that, they deteriorate fairly quickly, requiring constant improvement. Given such a high investment, model building should not be undertaken unless the right stakeholders are ready to operationalize it and a proper action plan is already laid out to launch the model and its insights.
- Successful models are built by strong modeler following structured approach. Building predictive models with high accuracy and low misclassification is not trivial. A successful modeler uses Aryng’s 5-step BADIR framework (below) or a close variant.
Five steps to successful model building using the BADIR framework. BADIR is an acronym for the below 5 steps:
- Model building starts with clearly understanding the Business question. What is the model trying to solve for and how is it measured? This needs to be arrived at by input from all the stakeholders and all of them need to agree with the goal for the project.
- Then, brainstorming hypotheses with stakeholders generates the most likely predictors for the model. Hypotheses inform the Analysis plan, which also includes the analysis goal, the methodology, sampling strategy, data specification, timelines, resources and milestones. All the stakeholders need to agree to the analysis plan before the project moves forward. This key step ensures that the model will be ready to be acted upon when the results roll out.
- Next, Data is pulled, cleaned and validated. Surprisingly, this step may take more than 50% of the project time, depending on the maturity of the organization.
- The Insights step is where the model is built. Irrelevant columns are dropped, missing values and outliers are resolved, variables are transformed, and reduced sets of variables are plugged into the model to create the first set of models. These models are then validated and examined to finally produce the winner model. The impact of the model and insights are discussed with the stakeholders and their input is taken to fine tune the model.
- Finally at the Recommendation step, the model is deployed and/or the insights are formally presented. This is a critical stage. The model is often presented to a nontechnical audience, so the insights and the inner workings of the model must be explained in the context of the business and the attendees. Unfortunately, good presentation skills are all too rare. Thus, many models get killed at this stage because the results aren’t communicated appropriately to those who need to take action.
If you are a current or future analyst, you can learn how to build effective predictive models by first assessing your analytics aptitude and if you score high, taking our hands-on business and predictive analytics course online here. Make sure to use the launch discount codes listed on the page before 8/31/2015 to avail yourself of steep discounts. And once you have the modeling skills - practice, practice and practice some more to fully realize the power of predictive analytics.
If you found this useful, my book “Behind Every Good Decision”, has an entire chapter discussing the five steps of predictive model building and their powerful application in the world of business.
My Book on Amazon| Follow @AnalyticsQueen | Sign up for our Newsletter | http://www.aryng.com | Book 15 minutes FREE on my calendar here
