Book Notes: Implementing Analytics - Nauman Sheikh
My reading notes from the book
Table of Contents
ch01 Defining Analytics
Challenge of Definition
dictionaries
logical analysis method
logic with analysis
turning data into insights into action
definition 1: business value perspective
three variations of value
present
past
future
data created
in operational system
definition 2: techincal implementation perspective
four data analysis methods
forecasting
descriptive (clustering, association rules ...)
predictive analytics (classification, regression, text mining)
decision optimization
Analytics Techniques
4 techniques
forecasting
descriptive analytics (primarily clustering)
predictive analytics (primarily classification, also regression and text mining)
decision optimization
algorithm vs. analytics model
each technique implemented
by an algorithm
algorithm: like an instrument
tune you produce: analytics model
forecasting
related terms
time series analysis
sequence analysis
descriptive analytics
clustering
grouping data points
predictive analytics
based on
how similar observations were classified in past
also known
directed analytics
training data shows what to predict
prediction
what class a new observation belongs to
ex
probability that a coupon offer will be redeemed leading to a sale?
probability a credit card will default
probability that a product has a defect leading to a warranty claim
how it works
training data:
with known outcomes
algorithm learns
from pattern of known outcomes
what variables influence the likelihood of a certain outcome
discriminatory power of a variable
prediction vs. forecasting
forecasting
time series
new value may not have occurred in the past
no likelihood value
regression
prediction
historical value musth ave occurred
ex: weather
forecasting:
what is high temperature tomorrow?
prediction
likelihood that it will be 72?
prediction methods
regression
classic
data mining
decision trees
neural networks
naive bayes
decision trees
simplest
ex: fig 1.6 training data set
text mining
predict categories of text
decision optimization
ch02 Information Continuum
ch03 Using Analytics
Healthcare
Emergency Room Visit
probability that a patient will be back in ER in next 3 months?
analytics solution
predictive model
predictive variable:
ER_Return
historical ER data
3-5 years
paitent visit
age
gender
profession
marital
diagnosis
procedure 1
procedure 2
date
vitals 1-5
previous diagnosis
previous precedure
current medication
last visit to ER
insurance coverage
Patients with the Same Disease
identify common patterns among patients with same disease
analytics solution
goal: clustering
variables
age, gender, economic status, zip code, children?, pets?, medical conditions, vitals?
CRM
customer segmentation
build customer segments based on the similarity of their profile
analytics solution
one record per customer
variables
age, gender, children, income, luxury car, zip code, number of products purchased, avg products, distance from store, online shopper
propensity to buy
track how many coupons are utilized? overall benefit of the campaign?
problem: probability that a customer buys the product upon receiving coupon
analytics solution
predictive problem
predictive variable:
Uses_Coupon
age, demographics, purchases, departments and products purchased, coupon date, type, coupon delivery method
Human Resources
not as widespread
ex
predict employees potentially leaving
employee attrition
probability a new employee will leave in first 3 months
analytics solution
predictive variable
Left_Company
one record per employee
variables
employee personal profile, education, interview process, referring firm, last job details, reasons for leaving last job, interest level, hiring manager, hiring department, new compensation, old compensation
tricky:
interest leven in new position
resume matching
resume run through text mining
prediction as to the class of resume
problem: probability that this resume is from a good java programmer?
analytics solution
existing known resumes
should be labeled
business analyst, database developer etc.
consumer risk
mature adoption in banking industry
products:
auto loans, credit cards, personal loans, mortgages
borrower default
probability that customer default on loan in next 12 months
analytics solution
predictive variable
Defaulted
one record per loan account
variables
customer personal profile, demographic, type of account, loan disbursed amount, term of loan, missed payments, installment amount, credit history from credit bureau
Insurance
customer fills an application, model calculates rate
Probability of a claim
probability that policiy will incur a claim within 3 years
analytics solution
preditive variable
Incurs_Claim
one record per insurance policy
variables
policiyholder personal profile, policiy type, open date, maturity date, premium amount, payout amount, sales representative, commission percentage
Telecommunication
Call Usage Patterns
build clusters of customers
analytics solution
one record per customer
variables
calls, sms messages, data utilization, bill payment, demographics, legnth of relationship
Higher Education
problem: will the student offered admission, will accept and join the school?
admission and acceptance
probability that a student will accept admission
analytics solution
predictive variable
Accepts
one record per student application
variables
sat, essay score, interview score, economic background, ethnicity, personal profile, high school, school distance, siblings, financial aid requested, faculty applied
Manuafacturing
to manage supply chain
driving forces
3d printing => new designs
global supply chains => large choice
volatiliy in pricing => procurement and storage very complex
customization => adjustments on short notice
warranty claims
Predicting Warranty Claims
probability that next product will lead to warranty claim
analytics solution
predictive variable
Warranty_Claim_Paid
data grain: product level
related to
product specs
production schedule
employees
customer details
Analyzing Warranty Claims
clustering similar claims to break down the defect problem into chunks
analytics solution
one record per claim
variables
product details
product specs
customer details
production schedules
line workers' details
sales channel details
defect details
Energy and Utilities
after deregulation: power generation is separate from distribution
customer care and billing done by energy services
new challenge
analytics solution
weather forecasting
correlate weather to load
predictive model
generation company: probability of firing up power generation plants
services company: predict additional customers to come online at peak load hits
clustering models to group their customers
decision optimization and pricing algorithms to max services companies
Fraud Detection
classification and clustering
benefits fraud
goverment benefits
frauds:
payment transactions
has two children but receives for four
citizien is not eligible
misrepresenting his income
probability that an application is fraudelent
analytics solution
predictive variable
Is_fraudelent
one record per application
variables
similar to loan application
Credit Card Fraud
transaction that violates expected behavior
analytics solution
clustering problem
clusters of customers based on their behavior
grain of data
customer level
all purchases on card aggregated and related to customer record
variables
average spending on gas, shopping, dining out, travel
full payment, min payment
percentage of limit utilized
geography of spending
clusters
their values are thresholds
breaking threshold => fraud alert
Patterns of Problems
prediction problem
break down problem into 1 or 0
event occurred or not
ch04 Performance Variables and Model Development
Performance Variables
Terminology
columns or data fields
variables
performance variables
aggregate or summary variables
input variables
input to analytics algorithm
characteristics
input variables with their relative weights and probabilities in model
What are performance variables?
Reasons for creating performance variables
ex:
ABC sells computers online
data fields
entity | data fields
customer | name, address, credit score ...
product | product id, type, class, ...
sales channel | channel id, category, partner ...
sales transaction | transaction id, code, timestamp, channel ...
issues
most data fields empty
if filled out, with quality issues
lots of fields have same values
assume
# fields: 70
mostly with null: 23
good data: 24
input variables: 24
second test
are these fields filled only for customers who bought?
if so, not useful
how much data is enough?
40% of all transactions should be available
90% for training
10% for testing
what if no pattern is available?
use performance variables
aggregate variables built using historical data
same level of grain as input variables
ex: performance variables
purchased_in_last_12_mo
no_of_visits_online
no_of_customer_service_contacts
1st_ever_purchase_date
last_purchase_date
define new performance variables
benefit of using performance variables
as business evolves
remove some input variables
add newer performance variables
creating performance variables
grain
is the level of detail at which analytics algorithm will work
range
continuous variables are not good
spread
how the population is broken out
ex
predict likelihood of an insurance policy to redeem a claim
performance variable
customer_profit_lifetodate_amount
value range: -1200-7300
frequency might be 1-2 for most values
might be skewed
Designing Performance Variables
discrete vs. continuous
how to convert continuou into discrete?
ex
how many buckets to build?
their ranges?
best:
uniform frequencies in each bucket
nominal vs. ordinal
atomic vs. aggregate
Working Example
performance variables:
popular_products_purchased
multiple per customer record
not at same grain level
thus divide it:
popular_products_purchased_1
popular_products_purchased_2
popular_products_purchased_3
Model Development
What is a Model
difference: model vs. algorithm
algorithm
takes input data
to produce model
model
takes input data
produces an output (prediction, forecast etc.)
in Predictive Modeling
ex: loan application. produce risk score
ex: characteristic: age
input variable: age_range_code
max. weight: 200
ex:
value | weight
< 17 | rejected
18-24 | 100
25-45 | 150
ex: characteristic: previous loan status
input variable: 12_month_previous_loan_history_status
ex:
value | weight
one or more loand and uptodate payments | 200
no loans before | -50
one loan, two payments late | -100
total risk max: 1000
two variables: 450 / 1000. 45%
input variables converted into characteristics to form the model
relative weights/importance of variables emerge
ex: Risk Scorecard
characteristics | % weight | max score
age | 20% | 200
previous loans | 25% | 250
income | 15% | 150
in Descriptive Modeling
model built. you have:
number of clusters
cluster affinity (closeness)
cluster characteristics:
cluster name/id
input variables and their range of values
probabilities and correlations of variables in each cluster
Model Validation and Tuning
essential bc of blackbox nature of ml algorithms
predictive model validation
low false-positives
error rate of model's predictions
if probabilistic output
ex
shipping company
predict which shipments will get delayed
historic data
pred variable: delayed or not
confusion matrix
actual shipments on time | actual shipments delayed
predicted on time | 7670 | 170
predicted delayed | 940 | 1220
Challenger: A Culture of Constant Innovation
ch05 Automated Decisions and Business Innovation
Automated Decisions
school of thought
empower data scientists
give them all models
let them find out
democratization of analytics
not limited to problems where data scientists are available
required skills
data and analytics
very rare
automated decision approach
converting subjectivity
into objective transparent set of rules
Decision Strategy
decision strategy
most important part of the book
nothing matters
if decision strategy is not properly designed
analytics has gray area
not all scenarios
can be accomodated
have enough training data
scenarios can have conflicting outputs
change by conditions
assumes business as usual
Nassim Taleb
Fooled By Randomness
The Black Swan
weakness of models
when surprising event occurs
surprise events
not addressed by historical data
trick
find balance in decision strategies
limiting damage from automated decisions
morgage crisis of 2008
risk models dubbed risky customers as subprime
decision strategy was agressive
morgtgage backed securities risk models
called them AAA
didn't factor performance variables
understanding models
far more difficult for managers
vs. managing the decision strategy
Business Rules in Business Operations
business rules
have decision variables (DVs)
to complete a transaction
ex
offer a discount to a good customer
what is good?
don't lend to a customer with poor credit
what is poor?
reroute a package using a premium service if can delay
what is "can"?
stop suspicious money laundering transfer
what is suspicious?
hire most suitable candidate
what is suitable?
two categories of rules
expert
quantitative
Expert Business Rules
devised by expert people
they are
policiy designers, mentors, to get advice employees, know how to handle things
business as usual rules (80% of normal business) are
either baked into operational systems
or employees are well trained to address them
Quantitative Business Rules
based on hard numbers and absolute values
ex
good customer
has been a customer >1 year
purchased >300 $
suspicious money transfer
> 7500 $
if customer has account < 200 $
Decision Automation and Business Rules
analytics models -> new business rules
document the process
describe model
its output
context of decision
Joint Business and Analytics Sessions
business input
reasons
1. they understand their business
2. their buy-in is needed
Examples of Decision Strategy
Retail Bank
car loan product
probability that loan will not be paid
bank does:
1. historical data of load data. identify loans paid and defaulted
2. predictive load default model
3. identify patterns that differentiate
4. test model
5. tune model
decision variables
probability of deafult
loan amount
down payment
decision strategy
IF any of the Policy Variables are TRUE
THEN Reject
ELSE
IF the Probability of Default > 0.62 (62%)
THEN Reject
ELSE
IF Probability of Default is between 0.28 and 0.62
THEN IF Loan Amount Requested <= 10,000
THEN Approve with 25% down payment
ELSE (i.e. loan amount requested is > 10,000) Request for a Co-Signer
ELSE Approve the loan (i.e. probability of default is < 28%)
performance of the model
measured using results of decision strategies
decision variables and cutoffs
DVs
probability of default
from risk predictive model
loan amount
part of business transaction
Insurance Claims
decision strategy
fig 5.2
Decision Strategy in Descriptive Models
Decision Automation and Intelligent Systems
Learning versus Applying
purpose of analytics systems
analyze data
learn from analysis
use that knowledge
insight to optimize business operations
optimize business operations
means
operational system changes
two dimensions of decision automation
learning
data warehous, analysis, analytics models
application of that learning on business operatinos
scope: decision strategies
school of thought: active data warehousing
do decision making in data warehouse
since all relevant data is available
requires
dw connected to operational system
Strategy Integration Methods
problem
how to add decision strategy into operational system?
ETL to the Rescue
encompasses
all aspects of data in motion
single record, large data set, messaging, files, real time, batch
ETL: glue that holds entire analytics solution
integration
ETL layer
receives real time event from operational system
prepares input record around event
invokes analytics model
get output from analytics model
pass it to decision strategy
reach a decision
return decision back into operational system
Strategy Evaluation
evaluation of decision strategies
validating automated decision making
testing alternate approach
Retrospective Processing
use historical data
they have known outcome
compare them with strategy
Reprocessing
run event through existing decision process
randomly select 10% of transactions
run through new strategy
save results
after some time
when benefit of decision is evident and measurable
compare them
Challenger Strategies
ch06: Governance Monitoring and Tuning of Analytics Solutions
Analytics and Automated Decisions
The Risk of Automated Decisions
incorrect decisions carried out automatically
inaction leading to lost opportunity
wasted resources
Monitoring Layer
Audit and Control Framework
automated decisions
2 parts
analytics
strategy
information
analytics model name / id, type, version
decision strategy name / id, version
input values
output
decision path
input date/time, source system, id of input, output date
real data
still in operational system
Organization and Process
audit and control organization
data moved into audit datamart
Audit Datamart
fig 6.1
tables
decision summary fact table
records decisions
grain: decision level
decision code
decision that strategy eventually takes
3 typical decision codes
accept
reject
review
predictive model
assigns probability of default
strategy
looks at overall situation of applicant, loan offer, product specifics, liquidity, lender's overall strategy
threshold table
wrt. automated decisions
threshold on certain decision code
allow certain values of decisions
breaches get recorded in alerts table
data warehousing best practices
use of ETL
to load data
regularly scheduled data feeds
data quality controls
etc.
Unique Features of Audit Datamart
schedule shared
data discrepancy go to investigation
changes to datamart signed off by auditors
separate logs for access and running queries
tight security controls
application for threshold table
alerts table runs nightly
Control Definition
Best Practice Controls
Expert Controls
Analytical Controls
Reporting and Action
ch07: Analytics Adoption Roadmap
Learning from Success of DW
main
how to convert into an analytics driven enterprise
penetrate all aspects of business
top-down
from top executives
bottom up
our proposal
Simplification
original definition of DW (Bill Inmon 1992)
reporting strains operational systems
there is no single version of truth integrated
solution
take all reportable data
move into dw
Quick Results
adoption:
first top-down approach
large projects
problem: analysis paralysis
years to integrate with no value
then: datamart based approach
functional use facing databases
integrating all relevant data
for that function's reporting needs
dw was built datamart after datamart
Evangelize
no need to convince management this is important
the more they see, the more they ask for
Efficient Data Acquisition
capability in handling
various forms of data
at different schedules
called: ETL
vast field: ETL
involves all aspects
small industry: data integration
integration
with all operational systems
mechanism for receiving and sending data
Holistic View
business requirements
come from users of datamarts
dw teams has business analysts
that has complete functional and data perspective
business users trust dw business specialists
Data Management
in addition to infrastructure
data dictionary
data model
data lineage
how did data reach reporting display
data quality controls
data dependencies
transformation business rules
other metadata
dw is prerequisite for analytics
information continuum
first levels: dw
analytics on top of dw
The Pilot
Business Problem
constraint of analytics projects
business does not understand
what they want
IT doesn't know
what to build
Management Attention and Champion
don't pick a problem
for which relevant data is not in dw
The Project
deliverables:
problem statement
candidate variables as input to model
mapping of input variables to dw data
design of specialized analytics datamart
design of ETL: dw > transform > analytics datamart
integrate analytics datamart with analytics software
identify large data sample for pilot
load analytics datamart
separate 90% for training, 10% for testing
run build training models
validate results, add more variables, repeat to improve
feed new data to model, return output to business for actions
tendency to get expensive software
resist by R and SQL
Results, Roadshow, and Case for Wider Adoption
Analytics Organization and Architecture
Organizational structure
bicc: business intelligence competency center
responsible for
implementation (incl dw)
analytics
decision strategies
across Information Continuum
BICC Organization Chart
ETL
design and architecture
scheduling and maintenance
ETL development
Data Architecture
data modeling
data integration
metadata, tool, repository
Business analysis
data analysis
analytics analysis
requirements analysis
Analytics
analitics modeling
analytics implementation
decision strategies
Information delivery
reporting development
analytical application development
operational integration development
Additional support teams
Database administration
technology infrastructure
project management
quality assurance and data governance
information security
compliance, audit, control
Roles and Responsibilities
ETL
Data Architecture
Business Analysts
Analytics
Information Delivery
Skills Summary
Analytics Analyst
Analytics Architect
Analytics Specialist
Technical Components in Analytics Solutions
Analytics Datamart
Base Analytics Data
Performance Variables
Model and Characteristics
Model Execution, Audit, and Control
Referred Books
Kimball
Innon
Han
