# Structured Machine Learning Projects (week1) - ML Strategy 1   2018-07-24

• 理解如何诊断机器学习系统中的错误
• 能够优先减小误差最有效的方向
• 理解复杂ML设定，例如训练/测试集不匹配，比较并/或超过人的表现
• 知道如何应用端到端学习、迁移学习以及多任务学习

## 2. Orthogonalization

Orthogonalization [ɔ:θɒɡənəlaɪ’zeɪʃn] 正交化

And when I train a neural network，I tend not to use early shopping.

When a supervised learning system is design, these are the 4 assumptions that needs to be true and orthogonal.

No. strategy solutions
1. Fit training set well in cost function If it doesn’t fit well, the use of a bigger neural network or switching to a better optimization algorithm might help.
2. Fit development set well on cost function If it doesn’t fit well, regularization or using bigger training set might help.
3. Fit test set well on cost function If it doesn’t fit well, the use of a bigger development set might help
4. Performs well in real world If it doesn’t perform well, the development test set is not set correctly or the cost function is not evaluating the right thing

## 3. Single number evaluation metric

Ref: sklearn中 F1-micro 与 F1-macro区别和计算原理

## 4. Satisficing and optimizing metrics

It’s not always easy into a single real number evaluation metric

So more generally, if you have N metrics that you care about, it’s sometimes reasonable to pick one of them to be optimizing. So you want to do as well as is possible on that one. And then N minus 1 to be satisficing.

## 5. Train/dev/test distributions

Training, development and test distributions

Setting up the training, development and test sets have a huge impact on productivity. It is important to
choose the development and test sets from the same distribution and it must be taken randomly from all
the data.

Guideline

Choose a development set and test set to reflect data you expect to get in the future and consider important to do well.

## 7. When to change dev/test sets and metrics

• 算法A: 喵咪图片识别误差是3%，但是可能会一不小心就给用户发了一些少儿不宜的图片
• 算法B：误差是5%，但是不会给用户推送不健康的图片

## 8. Why human-level performance?

• 蓝色虚线：表示人类识别的准确率
• 紫色曲线：表示机器学习不断训练过程中准确率的变化
• 绿色虚线：表示最高的准确率，即100%

## 9. Avoidable bias

Humans error 与 Training Error 之间的差距我们成为 Avoidable bias
Training Error 与 Dev Error 之间的差距我们成为 Variance

## 10. Understanding human-level performance

Scenario A

Scenario Bayse

Scenario C
Avoidable Bias=0.2%，Variance=0.1%，二者相差无几，但是此时训练的模型准确率还是不及人类，所以没办法咱们还得继续优化，都说枪打出头鸟，所以继续优化bias~

## 11. Surpassing human-level performance

Scenario A

• Avoidable Bias=0.1%，Variance=0.2%，所以此时应该将重心放到减小Variance上去

Scenario B

• Avoidable Bias=-0.2%，Variance=0.1%.乍一看可能会有点不知所措，而且训练集准确度也超过了人的最好成绩，不知道应该选择优化哪一项了，或者说这是不是就说明可以不用再优化了呢？

（还是可以继续优化的。不可否认在图像识别方面人类的确其优于机器的方面，但是在其他方面，如在线广告推送，贷款申请评测等方面机器人要远远比人类优秀，所以如果是在上面课件中提到的一些领域，即使机器准确度超过了人类，也还有很大的优化空间。具体怎么优化。。。以后再探索。。。）

## 13. Reference

如果您觉得这篇文章对您的学习很有帮助, 请您也分享它, 让它能再次帮助到更多的需要学习的人. 您的支持将鼓励我继续创作 !