Exercises: Introduction to Neural Networks

Exercise 1: Forward Pass Calculation

Objective: Understand the propagation of inputs through a neural network.

1. Given:

   • Input (single example): $$X\in {\mathbb{R}}^{1\times 2},X=[1,0.5]$$
   • Weights: $$W\in {\mathbb{R}}^{2\times 2},W=\left[\begin{array}{ccc}0.2& 0.80.4& 0.3\end{array}\right]$$
   • Biases: $$b\in {\mathbb{R}}^{1\times 2},b=[0.1,0.1]$$

2. Tasks:

   • Calculate the weighted sum $$Z=XW+b.$$
   • Apply the ReLU activation function component-wise: $$A=\text{ReLU}(Z).$$

Exercise 2: Backpropagation

Objective: Compute gradients for a simple neural network.

1. Setup (two-layer network):

   • Input: $$X\in {\mathbb{R}}^{1\times 2},X=[0.5,-0.2]$$
   • First-layer weights: $$W_1\in {\mathbb{R}}^{2\times 2},W_1=\left[\begin{array}{ccc}0.1& 0.3-0.2& 0.4\end{array}\right]$$
   • First-layer biases (for simplicity): $$b_1=[0,0]$$
   • Hidden representation: $$Z_1=X{W}_1+b_1,H=\text{ReLU}(Z_1)$$
   • Output-layer weights (binary output): $$W_2\in {\mathbb{R}}^{2\times 1},W_2=\left[\begin{array}{c}0.2-0.5\end{array}\right]$$
   • Output bias: $$b_2=0$$
   • Output pre-activation and prediction: $$z_2=H\cdot W_2+b_2,\hat{y}=\sigma (z_2)$$
   • True label: $y=1$

2. Tasks:

   • Perform the forward pass as defined above (ReLU for the hidden layer, Sigmoid for the output).
   • Calculate the binary cross-entropy loss: $$L=-(y\log (\hat{y})+(1-y)\log (1-\hat{y})).$$
   • Derive the gradients $\frac{\partial L}{\partial W_1}$ and $\frac{\partial L}{\partial W_2}$ using backpropagation (apply the chain rule).

Exercise 3: Data Preprocessing

Objective: Explore the impact of scaling on neural networks.

1. Given:

   • Dataset $X\in {\mathbb{R}}^{3\times 3}$, with rows as samples and columns as features: $$X=\left[\begin{array}{ccccccc}5& 20& 1015& 5& 2510& 30& 15\end{array}\right].$$

2. Tasks:

   • Apply Min–Max scaling per feature to rescale each column to the range $[0,1]$: $$x_{\text{new}}=\frac{x-x_{min}}{x_{max}-x_{min}}.$$
   • Standardize each feature to have zero mean and unit variance: $$x_{\text{new}}=\frac{x-\mu }{\sigma }.$$
   • Compare the two approaches and explain when each would be preferred.

Exercise 4: Activation Functions

Objective: Compare the behavior of activation functions.

1. Given:

   • Input values $X=-2,-1,0,1,2$.

2. Tasks:

   • For each $x\in X$, compute the outputs of:

     • ReLU: $f(x)=max(0,x)$
     • Leaky ReLU $(\alpha =0.01)$: $f(x)=max(0.01x,x)$
     • Sigmoid: $\sigma (x)=\frac{1}{1+e^{-x}}$
     • Tanh: $\tanh (x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$

   • Sketch the graphs of these functions.

   • Discuss the advantages and disadvantages of each function.

Exercise 5: Gradient Checking

Objective: Verify the correctness of computed gradients.

1. Setup:

   • Prediction: $$\hat{y}=wx+b$$
   • Loss: $$L=\frac{1}{2}(y-\hat{y}{)}^2$$
   • Parameters: $w=0.5$, $x=2$, $b=0.1$, $y=1$.

2. Tasks:

   • Compute the analytical gradient $\frac{\partial L}{\partial w}$.
   • Use numerical approximation to compute: $$\frac{\partial L}{\partial w}\approx \frac{L(w+\epsilon )-L(w-\epsilon )}{2\epsilon }\text{with }\epsilon ={10}^{-4}.$$
   • Compare the two results and explain any differences.

Exercise 6: Regularization

Objective: Understand the effect of L2 regularization on weight updates.

1. Setup:

   • Weights (parameter vector): $$W=[1,-2,0.5]\in {\mathbb{R}}^3.$$
   • L2 regularization with coefficient $\lambda =0.01$.

2. Tasks:

   • Compute the weight penalty term: $$\lambda \sum _j{W}_j^2=\lambda |W{|}_2^2.$$
   • Let $g=\frac{\partial L}{\partial W}$ denote the gradient of the loss without regularization. Derive the gradient descent update rule for $W$ with learning rate $\eta =0.1$ when L2 regularization is added, i.e. express $W_{\text{new}}$ in terms of $W$, $g$, $\lambda$, and $\eta$.
   • Explain qualitatively how this regularization term affects model training (in particular, the magnitude of the weights).

Exercise 7: Neural Network Error Analysis

Objective: Analyze and identify potential issues in a neural network setup.

1. Setup:

   • Input: $$X\in {\mathbb{R}}^{1\times 2},X=[1,2]$$
   • First layer: $$W_1=\left[\begin{array}{ccc}0.5& 0.2-0.3& 0.8\end{array}\right],b_1=[0.1,-0.1]$$ Hidden pre-activation and activation: $$Z_1=X{W}_1+b_1,H=\text{ReLU}(Z_1)$$
   • Output layer: $$W_2=\left[\begin{array}{c}0.7\text{-}0.6\end{array}\right],b_2=0.2$$ Output pre-activation and prediction: $$z_2=H\cdot W_2+b_2,\hat{y}=\sigma (z_2)$$
   • True label: $y=1$.

2. Tasks:

   • Calculate the loss using binary cross-entropy: $$L=-(y\log (\hat{y})+(1-y)\log (1-\hat{y})).$$
   • Based on the scale of the weights and the network depth, discuss whether this initialization is likely to cause vanishing or exploding gradients.
   • Propose changes to the network architecture, initialization scheme, or other hyperparameters to improve training stability and performance.