Tuesday, January 13, 2015

Book reading - "The signal and the noise - why so many predictions fail and some don't" by Nate Silver


I bought this book in hard copy around 3 weeks. I felt really happy when I finished the book. The last time I finished a book in hard copy might be 4-5 years ago.
The interest of the book comes from my phone interview with Discover Financial in Nov 2014. I didn't handle the questions well on assumptions of regression. Then I realized I have produced so many "accurate" financial reports in my daily work and almost lose the healthy suspicion on the model. I think this book does answer part of the question.

This books covers different topics of forecast and explains the common issues in some of the forecast. I spend some time to summary them.

In the next blogs, I will examine the forecast I've done in my work or life, and find why they fail and if possible find a way to improve it in the future.


ForecastExisting forecastWhy Fail (Challenging)Why SucceedHow to improve
RecessionMost economists failed1. Fail to forecast house bubble
- Limit data in US history
2. Fail to forecast impact of house bubble to whole economics
- Limit data in US history
- Large leverage
3. Rating agency provided wrong rating from their incentives
- Wrong assumption of independant events
Political forecastTV show forecast bad
Some tools are good
1. Bias because of party position
BaseballSome tools are good1. Difficult to bring quality attributes into model1. Lots of input data
2. Only a few output variable
WeatherSome tools are good1. Nonlinear behavior
2. Simplification of linear requires lots of calculation
1. Lots of input data
2. Lots of validation
3. Simple physics rules
EarthquakeAlmost impossible to predict time and location1. Very little sample data
2. Overfitting
3. Variables (like stress) in physics model can't be measured
GDP growthMost economists failed1. Weak mechanism
2. Both input and output macro economic data not accurate
3. Policy impact
Epidemic1. Extrapolation with big prediction interval
2. When data is brought from one country to another, the underline condition has changed
3. Accurate simulation requires lots of input for human behavior
Basketball game1. Using Bayes rule to update propensity with new facts
Chess1. human vs machine. each has its own advantage
human good at pattern identification and imagination
machine good at in-depth thinking
Poker game1. Using Bayes rule to update propensity with new facts
2. Validation of forecast and differentiate skill with luck
Stock price
Global warming1. Combine mechanisms with statistics
Terrorist attack1. Focus on frequency instead of when and where
2. For forecast of rare events, need imagination on scenarios




No comments:

Post a Comment