Section 1
Preview this deck
6. Draw up and Validate the Predictive models
Front
| 5 | 0 | ||
| 4 | 0 | ||
| 3 | 0 | ||
| 2 | 0 | ||
| 1 | 0 |
Active users
0
All-time users
0
Favorites
0
Last updated
6 years ago
Date created
Mar 1, 2020
Cards (109)
Section 1
(50 cards)
6. Draw up and Validate the Predictive models
a. This step is the most valuable, this may take the form of a calculation of a score, a predictive model, for each segment produced in the preceding step, followed by verification of the results in a test sample that is different from the learning sample b. May also be concerned with detecting the profiles of customers, for example according to their consumption of products or their use of the services of a business
Cluster Sampling
is a matter of drawing families of individuals (the 'clusters') at random and choosing all the individuals in each cluster, this being known as a census. We may, for example, choose certain urban districts at random and then ask questions of all the customers from these districts. Or we can choose a family name at random and then carry out a census of all customers whose family name starts with the letter drawn at random.
Ordinal data
data that is discrete and in order Examples: Low, medium, high ordinal data fall into categories, but the numbers placed on the categories have meaning. For example, rating a restaurant on a scale from 0 (lowest) to 5 (highest) stars
Fisher
Homoscedasticity can be verified by the this test its the least robust if normality is not present
9. Monitor the Models
a. Whenever a new data mining application is brought into use, the results must be analyzed
Aberrant value
is an erroneous value corresponding to an incorrect measurement, a calculation error, an input error or a false declaration. While extreme values are not always aberrant, aberrant values are not always extreme, and this makes them harder to detect, possibly requiring a thorough knowledge of the data
4. Explore and Prepare the Data
a. Check the origin of the data b. Create relevant indicators c. Reduce the number of dimensions of the problem
Identify the relevance of statistics in data mining.
Decision assistance is becoming an objective of data mining and statistics; •the combination of AI artificial intelligence and statistical analysis to discover information that is "hidden" in the data, things that can be "hidden" in data: Forecasting, associations, sequences, classifications, anomalies, grouping/clusters/segments
1. Defining the aims
a. Define the target population b. Define the statistical entity to be studied c. Define some essential criteria and especially the phenomenon to be predicted d. Plan the project, deciding on the expected operational use of the information extracted and the models produced e. Specify the expected results
Descriptive/Exploratory data mining
designed to bring out information that is present but buried in a mass of data (you might see clusters, or groups, you didn't know about, or find associations between products or medicines and relief of symptoms that you didn't initially see)
the phases of a data mining project are as follows:
. defining the aims; . listing the existing data; . collecting the data; . exploring and preparing the data; . population segmentation; . drawing up and validating the predictive models; . deploying the models; . training the model users; . monitoring the models; . enriching the models.
Bivariate Analysis
means "two variable(s)". With bivariate data you have two sets of related data that you want to compare. Example: a bike shop might want to measure how many bicyclists come in on warm weather days vs. cold weather days.
10. Enrich the Models
a. The first results are measured and the model can be improved, either by adding independent variables that were not considered initially or by using feedback from experience to determine the profiles to be predicted if these results were not available before
Transforming Variables: Data Normalization
o The goal of data normalization is to make the variables proportional to one another o For example, if one variable is 50 times larger than another (on average), then you may want your two variables to be approximately equivalent in the model. That way, the coefficients will reflect meaningful relative activity between each variable (i.e., a positive coefficients will mean that the variable acts positively towards the objective function, and vice versa, plush a large coefficient versus a small coefficient will reflect the degree to which that variable influences the objective function. o Normalization of a continuous variable is done by transforming the variable with a mathematical function, which compresses its distribution, brings it closer towards normalization
Shapiro - Wilk (P-P plot)
- this test is used to verify if the data is normally distributed (aka normality test) - the cumulative distribution of the data is shown on a normal probability scale, called a P-P (probability - probability) plot -SW value of 1 means that the data is perfectly normally distributed. The closer to 0 that the SW value gets, the more non-normally distributed our data is.
Transforming Variables: Data Discretization
o Discretization is the process of converting continuous data into a typically small number of finite values (for example high, medium, low). The variation in the original data is maintained in the discretized dataset. o Discretization is necessary precursor to data mining and is accomplished by assigning each value in a dataset to a bin. The data ranges (bin boundaries). o Tips for creating bins: • Avoid too many differences in classes between variables • Avoid to many classes for a variable • Avoid classes too small • About 4 or 5 classes is good
Tests for HOMOSCEDACITY are:
Levine (best) Bartlett Fisher
Define data mining.
is the set of methods and techniques for exploring and analysing data sets (which are often large), in an automatic or semi-automatic way, in order to find among these data certain unknown or hidden rules, associations or tendencies; special systems output the essentials of the useful information while reducing the quantity of data Data mining has some of the following distinctive features: - The development phase cannot be completed in the absence of data - The development of a model is primarily dependent on data - Development and testing are carried out in the same environment, with only the data sets differing from each other - To obtain an optimal model, moving frequently between testing and development is common - The data analysis for development and testing is carried out using a special-purpose program, i.e., SAS, SPSS, etc. - Some programs also offer the use of the model, which can be a realistic option if the program is implemented on a server - - Consciseness of the data mining models: unlike the instructions of a computer program, which are often relatively numerous, the number of instructions in a data mining model is nearly always small
Customer data
data that includes: - Lifetimes: i.e. age, lifetime as a customer of the business, length of time in present job - Relational, Attitudinal and Psychographic Data: i.e., responses to market campaigns and offers, loyalty, satisfaction, lifestyle, personality - Sociodemographic Data: i.e, sex, education, occupation, income, geographical information - Channel through which contact, orders and delivery came through
Discrete data
represent items that can be counted, they take on possible values that can be listed out, there is no continuum between values. Examples: number of items bought
Transactional Data
data collected during transactions. We ask the following questions: o Where? (geographical locations, business where the transactions took place, internet, etc.) o When? (frequency and recency of the transactions) o How? (method of payment) o How Much? (number and value of transactions) o What? (what has been purchased)
Product Data
information can be collected on different products, including: o Numbers o types o options o process o Date of purchase or subscription o data and reason for cancellation or return of products o mean product life expiry date o payment date and method o discount granted to the customer o profit margin on this product for the business, etc.
System Sampling
the individuals are drawn not at random, but in a regular way, if we carry out a 'one in a hundred' sampling, we take the first individual, then the 101st, then the 201st and so on. We must pay attention to cyclical data with this form of sampling: if we use customer numbers, the hundreds number may be a family number, and if we take one customer in every hundred, we will never choose two individuals in the same family. However, this sampling mode can also provide a degree of comprehensiveness.
Nominal data
categorical data that is not in any order Examples: Blue, white, orange, etc
Levine
-Used to verify homoscedasticity of data by testing that variance is equal for all samples -Best used for data that is NOT normally distributed -DO NOT use Levene's test for norm. dist. data, use Bartlett instead if data is norm. dist. -Assumes samples from the populations under consideration are INDEPENDENT
3. Collect the Data
a. This step leads to the construction of the database that will be used for the construction of models. This analysis base is usually in the form of a table, having one record (one row) for each statistical individual studied and one field (one column) for each variable relating to this individual
Bivariate Tests to use with two discrete variables:
• You might have two discrete variables: such as when measuring the link between gender and smoking o Tests to use: • Cramer's V • Chi-Squared (X^2)
Univariate data
means that we are looking at one variable, you can describe univariate data using numerical measurements, such as: mean, median, mode, standard deviation.
Continuous data
have possible values that cannot be counted and can only be described using intervals on the real number line. Examples: length, height, salary
Collinearity
Defined as simply correlation among the predictors in a multiple regression. Because of this "redundancy", it confuses the effects of the predictors. For instance, if height and weight are both used to predict something (anything, but let's say intelligence), your results will be skewed because height and weight are related (generally, the taller someone is, the more he weighs related to someone shorter).
Bivariate Tests to use with one discrete and one continuous variable:
•You might have one discrete and one continuous variable, as when measuring dosage of a medicine and recovery time. Test to use: • Parametric ANOVA test • Requires normality / homoscedacity assumption •Non-Parametric approaches • Wilcoxon-Mann-Whitney (2 groups) • Kruskal-Wallis (>2 groups)
Simple Random Sampling
involves drawing n individuals at random without replacement from a population of N, each individual having a probability of 1/N of being drawn.
detecting Extreme values
This step must be carried out for all types of models. It is optional for decision trees, which can easily tolerate extreme values. However, it is recommended, since extreme values may be aberrant values. An extreme value is not necessarily an aberrant value, though. It may relate to a specific profile or a specific category of individuals, which may or may not have to be retained in the study; we will have to decide this on a case-by-case basis
8. Train the Model Users
a. Users must know the objective, the principles of the tools, how they work, their limits, the methods for using them (while pointing out that these are decision support tools, NOT tools for automatic decision making), what the tools will contribute, and how the user's work patterns will change
Tests of Normality
Shapiro - Wilk (P-P plot) Kolmogorov - Smirnov Anderson-Darling
7. Deploy the Models
a. Deployment involves the implementation of the data mining models in a computer system, before using the results for action (adapting procedures, targeting, etc.) and making them available to users (information on the workplace, etc.)
Kolmogorov - Smirnov
- this test is used to verify if data is normally distributed (aka normality test) - it involves measuring the maximum deviation D (in absolute terms) between the distribution function (cumulative density function) of the variable tested and the distribution function of a Gaussian variable (or, more generally, of a continuous variable whose distribution is to be compared with that of the observed variable). - We then calculate the probability of observing such a large value of D on the hypothesis H0 that the tested data come from a normal distribution. - If this probability is below a given threshold of 0.05 or 0.10 (a higher threshold can be used if the sample sizes are smaller), we reject H0 and conclude that the data do not come from a normal distribution.
technical data
is not used within a data mining analysis but is used to determine whether individuals are included in the analysis base. This type of data includes: o Type of customer o non-acceptance of direct marketing o Bad payer status o status as employee of the business o Title, surname, forename
Homoscedacity
- refers to the points all having same scatter or variance, i.e. data values or points on a graph are all roughly the same distance from the slope line. If your data points on a graph generally follow a straight line, i.e. your plot looks like / or \ then this is indicative of homoscedastic data. - A formal requirement for some statistical analyses which is used to compare the means of two or more groups
Text data
- it has much in common with lexicometry or lexical statistics - the set of techniques and methods used for the automatic processing of natural language data available in reasonably large quantities in the form of computer files, with the aim of extracting and structuring their contents and themes, for the purposes of rapid (non- literary) analysis
5. Segment the Population
a. May be necessary to segment the population into groups that are homogeneous in relation to the AIMS of the study, in order to construct a specific model for each segment, before making a synthesis of the models. b. This method is called STRATIFICATION OF MODELS c. It can only be used where the volume of data is large enough for each segment to contain enough individuals of each category for prediction
Missing values
most statistical methods are unable to handle them, and the corresponding observations must be eliminated from the study. Missing values can occur, for example, in reported variables where the input was optional and was not done; in responses to surveys; or in chemical analyses where the concentrations of some elements are below the detection thresholds. This poses two problems: on the one hand, there may be only 1% of missing values for each variable, but more than 10% of observations in which one of the variables has a missing value; on the other hand, if the values are not missing by chance and there are systematic differences between the complete and incomplete observations, the removal of incomplete observations introduces a bias into the analysis.
Customer Segmentation
involves looking at the behavior and developing a descriptive profile for your customers
Predictive
Designed to extrapolate new information based on the present information, this new information being qualitative (in the form of classification or scoring) or quantitative (regression)
Customer relationship management (CRM)
understanding the expectations of customers and anticipating their needs The 1st step in analytical CRM phase is the collection of customer information The 2nd step is the analysis of customer information The basic definition states that CRM: -Is a way of using data about customers to manage and understand customer needs & expectations of a company/product -Goal is to increase profitability and customer loyalty while controlling risk and using the right channels to sell the right product at the right time -Uses data analysis about customer's history with a company to improve business relationships with customers -Specifically focuses on customer retention and ultimately driving sales growth
Anderson-Darling
this is most often used where a family of distributions is being tested, in which case the parameters of that family need to be estimated and account must be taken of this in adjusting either the test-statistic or its critical values.
Stratified Sampling
we divide the population, for example by dividing the customers into age ranges, and then draw customers at random from each stratum to obtain a sub-sample for each stratum; we can then bring all these sub-samples together.
Geodemographic Data
• Not relating to individuals, but to surrounding/living environment • Modeling and analysis of all factors between customer's residence and mode of consumption (Franck Bleuzen) • Place of residence in terms of economics, sociodemographics, housing and competition includes: o Competition o population o working population o customer population o unemployment rates o economic potential o product ownership rates, etc., in the area of residence of the customer or prospect, etc.
2. List the existing Data
a. List the data that will be useful, accessible, exploitable, reliable and sufficiently up to date and where they can be found b. If the AIM is to construct a predictive model, it will also be necessary to find a second type of data, namely the historical data on the phenomenon to be predicted
Bartlett
Homoscedasticity can be verified by the this test if the distribution is normal
Section 2
(50 cards)
Pearson correlation
useful for measuring how well related 2 sets of data are, by showing the linear relationship between 2 sets of data (via graph)
Right Skewed data
positive skewness (tail to the right), the order of values are mode<median<mean - to data transform take roots or logarithms or reciprocals (roots are weakest). This is the commonest problem in practice.
classification
it is an operation that places each variable from a population of a study into a specified class or classes based on the variable's characteristics the dependent variable is qualitative
Kohonen map (k-maps)
- it is an unsupervised learning neural network that is used strictly to classify and organize datasets to learn the structure of the data and find clusters that are hidden - classification method only
R
- is the open source counterpart of SAS (SASis costly used mostly in private corporations), - Because of its open source nature, updates get released quickly. - the only language that uses <- as the assignment operator - all of its code is available to the public for users to create their own packages and modify code to suit their needs, very customizable
propensity
- analysis application of data mining in CRM - studies the probability that a customer will be interested in a product or service - enables targeted marketing campaigns to be refined - saves a company money on marketing by advertising specific products to specific people based on the individuals propensity to consume a product
Cramer's V
- test useful for indicating how strongly associated 2 different categorical/nominal variables are with each other (link between variables) - is used on contingency tables
decision tree
- the aim of dividing the individuals of a population into n classes, we must know how to choose the variable which best separates the individuals of each class - used to map possible outcomes of a series of related choices AND also allows that possible actions to be weighted against each other based on their potential outcomes and benefits so that decisions can be made based on the data - BOTH classification and prediction method
Logistic Regression
is a classification statistical technique used to describe data and to explain the relationship between one dependent binary variable (yes or no) and one or more interval or ratio scale independent variables. You might want to know whether body weight and body fat percentage influence the occurrence of diabetes (yes or no) in a person, predictive modeling
Association Analysis method
a descriptive method, that is also called Market Basket Analysis. In a given set of records, each will contain a number of items. Association Analysis allows you to determine the degree to which the items tend to be associated with one another. For example, people who buy hamburger buns will also likely buy ketchup and mustard and hamburger meat. You can associate items together and create rules.
SAS Enterprise Miner
is a data mining software -can filter the extreme values according to standard deviations or percentiles
Non Parametric Tests
These tests operate on the ranks of the values, rather than on the values themselves. -Wilcoxon-Mann-Whitney (2 sample) -Kruskal-Wallis (3+ samples) -median -Joncheere-Terpstra (3+ samples)
PCA
a descriptive method - Reduces the number of variables in a model by identifying which variables to remove - helps ensure that variables are independent of one another - makes variables less interpretable - measures how each variable is associated with each other via Covariance matrix - Breaks covariance matrix down into 2 separate components (direction and magnitude) - show the directions in which data is dispersed via eigenvectors - show importance of those different directions via eigenvalues - combines predictor variables and allows the dropping of eigenvectors that are unimportant to model
RFM
- recency, frequency, monetary values - cross tabulates recency of the last purchase in the period being studied with the frequency of purchases in that period, then examines the distribution of purchases - analyzes where was product purchased? what was purchased? how it was paid for? How much was purchased? what was purchased?
cluster analysis
- is used for identifying structures in datasets and organizing variables with similar features into homogeneous groups - descriptive method useful for finding patterns or associations between objects in a data set
Data Mining Techniques:
methods that are descriptive or predictive
imputing by the mean
it is a process for fixing missing values
Parametric Tests
These tests tests operate on the assumption that the data is normally distributed. -Chi-Square (2 categorical variables) -ANOVA (3+ samples) -Student and Welch's T-Tests (2 sample means from different populations) -Mann-Whitney (2 sample) -Fisher's Test
Support Vector Machine (SVM)
a prediction technique that is is associated with algorithms that can, given a set of training examples, build a model that assigns new examples into one category or another -supervised learning method used for classification and detecting outliers in data
Supervised Classification Techniques
- are grouped when the dependent variable is binary or discrete Linear Discriminant Analysis (LDA) Logistic Regression Naive Bayes Classifier
Descriptive Methods:
• Unsupervised • Cluster Analysis • Principle Component Analysis (PCA) • Association Analysis
data preparation functions are:
- file handling (merging, aggregation, transpose, etc) - data display, color individuals based on criteria - detection, filtering, Winsorization of outliers - analyze and impute missing values - transform variables (recode, standardize, normalize, discretization, etc) - create new variables - select best independent variables, discretization, interactions
Cluster Analysis
a descriptive method, Given a set of data points, each having a set of attributes and a similarity measure among them, find clusters such that the data points in one cluster are more similar to one another and less to similar to those in separate clusters. Consider a bookstore with different areas for the types of books: History, Self-Help, Romance, Mystery & Crime, etc. The books in each of these clusters are more similar to each other than they are to other clusters
Predictive Methods: Supervised
to predict the value of a specific attribute (a dependent variable) based on the value of other attributes. Two important features critical to the classification of a data mining technique as a supervised learning method are: The use of previously observed events, typically referred to as target variable data, and the description of the target variable in terms of a set of explanatory variables. For example, you might want to judge how a high school student will perform in college based on school grades, extracurricular activities, and standardized test scores These are grouped as either: o Prediction Techniques o Classification Techniques
Python
originally a source scripting language, it has libraries and functions for almost any statistical operation or model building and has become very strong in operations on structured data.
Profitability Data
• Special type of data • It is the difference between the profits to the business from a customer, segment, or market, etc., and the costs incurred, namely the acquisition and structural costs, commercial costs, operation processing costs and the cost of finance • Profitability to a business from a customer or business segment o Determine who the most profitably customers o Hard to measure precisely o Important measure is the lifetime value (LTV) Net present value of profitability of customer
Student's t-test (also Welch's t-test)
compares 2 averages (means) and indicates if there is a significant difference in MEANS between 2 independent groups (i.e. men vs women, treated vs control group)
ANOVA
used for testing means between 3 or more independent groups to see if there is a difference in the means of the groups
MOSAIC
- a geodemographic typology based on sociodemographic data, lifestyles, behavior and preferences - used in experian credit reporting - comprised of financial variables like education level, size of household, occupation, income -is based on 300 variables, including the data from the 10-yearly general Census of the population
SPSS
is the data mining software produced by IBM, it is common among some companies (but less so than either SAS or R), it works with large data systems. - requires license and users cannot modify the code
Left Skewed data
negative skewness (tail to the left), the order of values are mean<median<mode - to data transform take squares or cubes or higher powers
Naïve Bayes
is a classification statistical technique (algorithm) that uses a set of training data to construct a model that can classify new data points. - based on finding functions that describe the probability of data belonging to a class given certain features (red, round, classified as apple for fruit groupings) - predictor variables assumed independent - requires very large data sets -
Kurtosis
A positive value tells you that you have heavy-tails (i.e. a lot of data in your tails). A negative value means that you have light-tails (i.e. little data in your tails). The standard normal distribution has a kurtosis of 3, so if your values are close to that then your graph is nearly normal.
Log transformation and Square Root transformation
to produce a normal distribution on right skewed data, both the log transformation and the square root transformation can be used to reduce right skewed data, we must make note that the log transformation is more powerful and robust than the square root transformation when it comes to reducing right skew in data. Thus the log transformation is usually preferred over the square root transformation for right skewed data.
Unsupervised method
a descriptive method, that do not predict a target value (i.e., there is not a dependent variable) but focus more on the structure, relationships, and interconnectedness of the data. Descriptive methods reduce, group, and summarize data. For example, you might want to identify the web pages that are accessed together
Square transformation
this transformation is only useful for LEFT skewed data
what are data transformations
methods that are defined as mathematical functions that are applied to data to reduce skewness in the data set and make the data easier to work with common data transformations are: log, cube root, square root and square
time series analysis
is a predictive method is used to study sequences of measurements of a variable or variables where the measurements are taken over regular time intervals it uses past events to help forecast and predict future events
Supervised Prediction Techniques
are grouped when the dependent variable is continuous Linear Regression Decision Trees SVM (Support Vector Machine) Neural Networks
Neural Networks
a prediction technique that do not typically reflect a linear relationship between a target and some set of inputs. Neural network models (Kohonen maps, RBF, etc) emulate aspects of human cognition and are flexible with regards to their ability to fit a target to non-linear relationships and can approximate functions that depend on a large number of inputs. However, neural networks may become computationally intensive. Note: if used to apply to existing data and not to new data, the Neural Network method would be considered a Classification Technique.
Heteroscedasticity
It has data points that are unequal distances from the slope line on a graph. If your data points on a graph are cone shaped, i.e. they look like < or >, then this is indicative of heteroscedastic data.
Forward Stepwise Selection
is a regression model used to predict future values from current data
Linear Regression
A prediction technique of analysis that is a common way of estimating the relationship between one or more input variables and a continuous target variable. LR assumes that the relationship between the continuous target variable and its corresponding inputs is linear - i.e. best described by a straight line. the fitting of a linear regression model involves the estimation of parameter estimates or "weights" that describe the influence of each input on the target variable value. These weights can also be used to predict the value of a new event
linear model assumptions are
- residuals (error terms) are normally distributed - best fitting regression line is a straight line - residuals (error terms) have constant variance at every value of x - residuals (error terms) are independent - residuals have mean of zero (error terms sum = 0)
association rules
it is used for detecting relationships and associations between values of independent categorical variables - example is how Amazon has "customers who viewed this also viewed" and "frequently bought together" recommendations for every item
tests for normality
Tests of normality must be performed for a Fisher discriminant analysis or linear regression, because of the assumptions of these models. It is preferable to deal with the extreme values beforehand. -Shapiro Wilk - Kolmogorov Smirnov - Cramer V (Cramer-von Mises) - Anderson Darling
S-PLUS
is a statistical software The R software is based on the same language
SAS
has been the market leader for commercial applications. The software offers many statistical functions, has good user interface for people to learn it quickly and providers technical support. However, it ends up being the most expensive option and is not upgraded as rapidly as some of the others. - requires a license and users cannot modify the code
Linear Discriminant Analysis (LDA)
is a classification statistical technique. The purpose is to classify (or discriminate) a qualitative outcome based on a set of inputs (usually quantitative). For instance, you might want to classify people based on a set of features - credit officers might want to determine if someone is a good risk or a bad risk (discriminate categories) based on different features, such as income, credit scores, etc.
survival analysis
is used when asked to analyze the retention rate and churn rate for customers - a predictive method for analyzing data being predicted in the time until an event of interest occurs
Section 3
(9 cards)