決定下一步做什么

這里我們以房?jī)r(jià)預(yù)測(cè)為例，假設(shè)我們需要使用線性回歸模型對(duì)房?jī)r(jià)進(jìn)行預(yù)測(cè)，其代價(jià)函數(shù)J(θ)如下圖所示。當(dāng)我們將已經(jīng)訓(xùn)練好的模型用來(lái)預(yù)測(cè)房?jī)r(jià)時(shí)，我們發(fā)現(xiàn)有較大的誤差，那么我們下一步應(yīng)該怎么做？

我們可能會(huì)想到以下幾種方法：

• 獲取更多的樣本；
• 嘗試減少特征變量的數(shù)量；
• 嘗試獲取更多的特征變量；
• 嘗試增加多項(xiàng)式特征；
• 嘗試減小正則化參數(shù)λ的值；
• 嘗試增大正則化參數(shù)λ的值；
  ......

這些方法可能有用也可能沒(méi)用，我們更不應(yīng)該在實(shí)際應(yīng)用中隨機(jī)選擇上述方法。還有一點(diǎn)要說(shuō)明的是：上述方法中的任意一個(gè)方法，在具體實(shí)踐過(guò)程中都可能轉(zhuǎn)變?yōu)橐粋€(gè)為期半年甚至?xí)r間更長(zhǎng)的項(xiàng)目。因此，我們需要引入機(jī)器學(xué)習(xí)診斷法來(lái)幫助我們決定下一步該做什么才是有效的方法。

Question:
Which of the following statements about diagnostics are true? Check all that apply.
A. It's hard to tell what will work to improve a learning algorithm, so the best approach is to with gut feeling and just see what works.
B. Diagnostics can give guidance as to what might be more fruitful things to try to improve a learning algorithm.
C. Diagnostics can be time-consuming to implement and try, but they can still be a very good use of your time.
D. A diagnostics can sometimes rule out certain courses of action (changes to your learning algorithm) as being unlikely to improve its performance significantly.

我們不難選出B，C和D這三個(gè)正確答案。

假設(shè)評(píng)估

對(duì)于我們之前所提及的欠擬合問(wèn)題和過(guò)擬合問(wèn)題，我們是通過(guò)畫圖的方法來(lái)檢驗(yàn)的。如若訓(xùn)練集中有較多的特征變量時(shí)，我們就無(wú)法將函數(shù)圖給呈現(xiàn)出來(lái)。

對(duì)此，我們可以將數(shù)據(jù)集分為訓(xùn)練集和測(cè)試集兩個(gè)部分，其中訓(xùn)練集占數(shù)據(jù)集的70%，測(cè)試集占數(shù)據(jù)集的30%。首先，我們利用訓(xùn)練集將代價(jià)函數(shù)J(θ)最小化，然后利用測(cè)試集測(cè)試模型誤差，從而出模型是否出現(xiàn)欠擬合或過(guò)擬合問(wèn)題。

注：如果數(shù)據(jù)集有一定規(guī)律，則要從數(shù)據(jù)集中隨機(jī)選取70%的樣本作為訓(xùn)練集，30%的樣本作為測(cè)試集。

線性回歸模型

• 利用訓(xùn)練集將代價(jià)函數(shù)J(θ)最小化，得到此時(shí)參數(shù)θ的值
• 利用測(cè)試集計(jì)算誤差：

邏輯回歸模型

• 利用訓(xùn)練集將代價(jià)函數(shù)J(θ)最小化，得到此時(shí)參數(shù)θ的值
• 利用測(cè)試集計(jì)算誤差：

除此之外，對(duì)于邏輯回歸模型我們還能計(jì)算誤分類率來(lái)幫助我們理解邏輯回歸模型的誤差。

誤分類率

因此，我們可將誤差測(cè)試的表達(dá)式改寫為：

Question:
Suppose an implementation of linear regression (without regularization) is badly overfitting the training set. In this case, we would expect:
A. The training error J(θ) to be low and the test error J_test(θ) to be high
B. The training error J(θ) to be low and the test error J_test(θ) to be low
C. The training error J(θ) to be high and the test error J_test(θ) to be low
D. The training error J(θ) to be high and the test error J_test(θ) to be high

我們不難選出A這個(gè)正確答案。

補(bǔ)充筆記

Evaluating a Hypothesis

Once we have done some trouble shooting for errors in our predictions by:

• Getting more training examples
• Trying smaller sets of features
• Trying additional features
• Trying polynomial features
• Increasing or decreasing λ

We can move on to evaluate our new hypothesis.

A hypothesis may have a low error for the training examples but still be inaccurate (because of overfitting). Thus, to evaluate a hypothesis, given a dataset of training examples, we can split up the data into two sets: a training set and a test set. Typically, the training set consists of 70 % of your data and the test set is the remaining 30 %.

The new procedure using these two sets is then:

1. Learn Θ and minimize J_train(Θ) using the training set
2. Compute the test set error J_test(Θ)

The test set error

1. For linear regression:

2. For classification ~ Misclassification error (aka 0/1 misclassification error):

This gives us a binary 0 or 1 error result based on a misclassification. The average test error for the test set is:

This gives us the proportion of the test data that was misclassified.

模型選擇以及訓(xùn)練集、交叉驗(yàn)證集和測(cè)試集的劃分

假設(shè)我們要在以下的多項(xiàng)式模型中選擇一個(gè)合適的模型：

• (d = 1) h_θ(x) = θ₀ + θ₁x
• (d = 2) h_θ(x) = θ₀ + θ₁x + θ₂x²
• (d = 3) h_θ(x) = θ₀ + θ₁x + θ₂x² + θ₃x³
  ......
• (d = 10) h_θ(x) = θ₀ + θ₁x + θ₂x² + θ₃x³ + ... + θ₁₀x¹⁰

其中，參數(shù)d表示多項(xiàng)式的次數(shù)。對(duì)于這種情況，我們使用將數(shù)據(jù)集劃分為訓(xùn)練集和測(cè)試集的方法。

上圖中，我們假設(shè)d=5時(shí)其測(cè)試誤差最小。但這時(shí)我們只是找到了一個(gè)對(duì)于測(cè)試集非常擬合的模型，我們無(wú)法判斷其實(shí)際泛化誤差是否完美。

因此，我們不能再將數(shù)據(jù)集只分為兩部分，訓(xùn)練集和測(cè)試集。對(duì)此，我們引入交叉驗(yàn)證集（Validation Set）。我們將數(shù)據(jù)集分為三個(gè)部分，訓(xùn)練集（60%）、交叉驗(yàn)證集（20%）和測(cè)試集（20%）。

對(duì)于上例，我們需要計(jì)算三部分?jǐn)?shù)據(jù)集的誤差。

最終，我們可得到d=4時(shí)測(cè)試誤差最小。

Question:
Consider the model selection procedure where we choose the degree of polynomial using a cross validation set. For the final model (with parameters θ), we might generally expect J_CV(θ) To be lower than J_test(θ) because:
A. An extra parameter (d, the degree of the polynomial) has been fit to the cross validation set.
B. An extra parameter (d, the degree of the polynomial) has been fit to the test set.
C. The cross validation set is usually smaller than the test set.
D. The cross validation set is usually larger than the test set.

終上所述，我們不難選出A這一正確答案。

補(bǔ)充筆記

Model Selection and Train/Validation/Test Sets

Just because a learning algorithm fits a training set well, that does not mean it is a good hypothesis. It could over fit and as a result your predictions on the test set would be poor. The error of your hypothesis as measured on the data set with which you trained the parameters will be lower than the error on any other data set.

Given many models with different polynomial degrees, we can use a systematic approach to identify the 'best' function. In order to choose the model of your hypothesis, you can test each degree of polynomial and look at the error result.

One way to break down our dataset into the three sets is:

• Training set: 60%
• Cross validation set: 20%
• Test set: 20%

We can now calculate three separate error values for the three different sets using the following method:

1. Optimize the parameters in Θ using the training set for each polynomial degree.
2. Find the polynomial degree d with the least error using the cross validation set.
3. Estimate the generalization error using the test set with J_test(Θ^(d)), (d = theta from polynomial with lower error);

This way, the degree of the polynomial d has not been trained using the test set.