Homework #4

\(~\)

Directions:

• Homework must be completed individually. Any guidance or help received from mentors, classmates, online resources other than course materials (including AI/LLMs) must be acknowledged.
• Clearly organize your responses so that each question (1, 2, etc.) and sub-question (Part A, B, C, etc.) are clearly identifiable.
• Please submit a single Jupyter notebook (.ipynb file) displaying all output and recording textual answers using neatly formatted markdown chunks.
• Your submission should be made via Canvas no later than 11:59pm on the assigned due-date.

Question 1 (gradient boosting and linear algebra)

Consider gradient boosting using the squared error cost function with linear regression as the base learner. You may use the fact that linear regression has a closed form solution for its weights (see our lectures on 10/27 and 10/29), and you do not need to re-derive this solution yourself.

• Part A: Let \(\mathbf{r}^{(1)}\) denote the vector of residuals for the first base learner in the ensemble. Write an expression for the weights of the second model (call them \(\hat{\mathbf{w}}^{(2)}\)) in the ensemble in terms of \(\mathbf{r}^{(1)}\), the residuals from the previous fit.
• Part B: Show that the weights learned by the second base learner are exactly zero. That is, show \(\hat{\mathbf{w}}^{(2)} = \mathbf{0}\).
• Part C: Provide a conceptual argument for why gradient boosting fails to improve the linear regression model beyond the first iteration.

\(~\)

Question 2 (gradient descent)

Poisson Regression uses a model with the form: \(\mathbf{\hat{y}} = e^{\mathbf{X}\mathbf{\hat{w}}}\)

In statistics, the standard approach to Poisson regression is estimate the unknown weights using maximum likelihood estimation. However, for this question, I’ll ask you to estimate a reasonable set of weights using the squared error cost function and optimizing using gradient descent (Note that this is not equivalent to maximum likelihood estimation in this scenario).

• Part A: Use chain rule to show that the gradient of the cost with respect to the weight parameters is \(\tfrac{-2}{n}(\mathbf{y} -e^{\mathbf{X}\mathbf{\hat{w}}})^T \cdot diag(e^{\mathbf{X}\mathbf{\hat{w}}})\cdot \mathbf{X}\), or equivalently the transposition, \(\tfrac{-2}{n}\mathbf{X}^T\cdot diag(e^{\mathbf{X}\mathbf{\hat{w}}}) \cdot (\mathbf{y} -e^{\mathbf{X}\mathbf{\hat{w}}})\), where \(diag(v)\) is used to denote a diagonal matrix whose diagonal elements are given by \(v\) and whose off-diagonal elements are zero.
• Part B: Write your own Python functions to calculate the cost (given \(X\), \(\hat{w}\), and \(y\)) and perform gradient descent for this scenario. You should use the functions provided in this unit’s labs as examples. You should consider using np.diag() to set up a diagonal matrix. For reference, the minimum cost for the example data (see below) should be between 4.69 and 4.70.

## Set up trial data
ic = pd.read_csv("https://remiller1450.github.io/data/IowaCityHomeSales.csv")
ic_y = ic['bedrooms']
ic_X = ic[['assessed','area.living']]

## Scale X
from sklearn.preprocessing import StandardScaler
ic_Xs = StandardScaler().fit_transform(ic_X)

## Fit via grad descent, 250 iter w/ 0.01 learning rate 
gdres = grad_descent(X=ic_Xs,y=ic_y,w=np.zeros(2),alpha=0.01, n_iter=250)

## Min of cost function
print(min(gdres[1]))

• Part C: Graph the cost function for 300 iterations of gradient descent with a learning rate of 0.001. Comment on whether this learning rate is appropriate.
• Part D: Graph the cost function for 300 iterations of gradient descent with a learning rate of 0.1. Comment on whether this learning rate is appropriate.

Additional Comments: Poisson regression can be fit via maximum likelihood in sklearn using the PoissonRegressor function. The arguments alpha=0 and fit_intercept=False can be used to mimic the model fit to the example data in this question. However, you should expect somewhat different weight estimates since maximum likelihood estimation is not equivalent to minimizing squared error loss for the Poisson regression model. Further, in this example, the variables “assessed” and “area.living” are highly correlated, so there are many combinations of weights that will fit the training data similarly well.

\(~\)

Question 3 (neural nets and backpropagation)

Consider a simple neural network consisting of 1 hidden layer containing 4 neurons that use the sigmoid activation function, and a final output layer that applies the sigmoid function to a linear combination of activated outputs from the hidden layer. You are to apply the squared error cost function to the final activated output.

The input data used in your network will be generated from the make_moons() function in sklearn:

## Set up trial data
X, y = make_moons(n_samples=200, noise=0.1)

Note that data has 2 input input features and a binary outcome.

• Part A: How many weight and bias parameters are used to generate the neurons in this network from the input features? Briefly explain.
• Part B: How many weight and bias parameters are used to generate the predicted outcome from this network from the hidden neurons? Briefly explain.
• Part C: Consider data on a single training example with its predictors stored in the vector \(\mathbf{x}_i\). To prepare for stochastic gradient descent, differentiate the cost function for a single observation with respect to each weight parameter in the final layer of the network.
• Part D: Now find the gradient components for each of the remaining parameters, all of the biases and the remaining weights. Clearly show your expression for each component, using the notation from our class and using matrices and vectors wherever possible. Significant deviations from the notation we’ve used during class will be penalized. Hint: it is simplest to derive the gradient expressions for blocks of the first layer weights (such as all weights entering a particular second layer neuron) rather than trying to find an expression for all of them simultaneously. I encourage you to adopt this simpler approach.
• Part E: Using functions from this unit’s labs as a guide, write a function that uses stochastic gradient descent to train this network. Your function does not need to generalize to scenarios other than this toy data set, so it is okay if certain attributes (such as network architecture and object sizes) are hard-coded.
• Part F: Train your network using an appropriately selected learning rate (try something small and increase as needed) and graph the cost function over 100 passes through the entire training data set (epochs) for a learning rate you identify as reasonable.
• Part G: Once you think you’ve found an appropriate learning rate in Part F, create an independent validation set by using make_moons() again with the same parameters (and a different random state if you may have set one). Add a line to your previous graph showing the cost function evaluated using this new data set at each training iteration.