When Variable Reduction Doesn’t Work

Data Science Central Featured Article Summary: Exceptions sometimes make the best rules. Here’s an example of well accepted variable reduction techniques resulting in an inferior model and a case for dramatically expanding the number of variables we start with. One of the things that keeps us data scientists on our toes is that the well-established rules-of-thumb don’t always work. Certainly one of the most well-worn of these rules is the parsimonious model; always seek to create the best model with the fewest variables. And woe to you who violate this rule. Your model will over fit, include false random correlations, or at very least will just be judged to be slow and clunky. Certainly this is a rule I embrace when building models so I was surprised and then delighted to find a well conducted study by Lexis/Nexis that lays out a case where this clearly isn’t true. A Little Background In highly regulated industries like insurance and lending the variables that are allowed for use are highly regulated as are the modeling techniques. Techniques are generally limited to those that are highly explainable, mostly GLM and simple decision trees. Data can’t include anything that is overtly discriminatory under the law so, for example, race, sex, and age can’t be used, or at least not directly. All of this works against model accuracy. [Click here]( to read the full article. --------------------------------------------------------------- Announcements - [Become a 3-Dimensional Data Scientist with Notre]( (Whitepaper) - [Live webinar series: 2018 Top 10 Business Intelligence Trends]( --------------------------------------------------------------- [Videos]( | [Hire a Data Scientist]( | [Search DSC]( | [Classifieds]( | [Find a Job]( | [Post a Blog]( | [Ask a Question]( Follow us on Twitter: [@DataScienceCtrl]( | [@AnalyticBridge]( Data Science Central, 2428, 35th Ave NE, Issaquah, WA 98029, United States You may [unsubscribe]( or [change your contact details]( at any time.