Neural Networks Exercises

Exercise 1: Forward Pass Calculation

Objective: Understand the propagation of inputs through a neural network.

1. Given:
   • Inputs: $X=[1,0.5]$
   • Weights: $W=[[0.2,0.8],[0.4,0.3]]$
   • Biases: $b=[0.1,0.1]$
2. Tasks:
   • Calculate the weighted sum: $Z=W\cdot X+b$.
   • Apply the ReLU activation function: $A=\text{ReLU}(Z)$.

Exercise 2: Backpropagation

Objective: Compute gradients for a simple neural network.

1. Setup:
   • Two-layer network with:
     • $X=[0.5,-0.2]$
     • $W_1=[[0.1,0.3],[-0.2,0.4]]$
     • $W_2=[0.2,-0.5]$
     • $y=1$
2. Tasks:
   • Perform a forward pass with ReLU for the hidden layer and Sigmoid for the output.
   • Calculate the binary cross-entropy loss.
   • Derive gradients for $W_1$ and $W_2$ using backpropagation.

Exercise 3: Data Preprocessing

Objective: Explore the impact of scaling on neural networks.

1. Given:
   • Dataset: $X=[[5,20,10],[15,5,25],[10,30,15]]$.
2. Tasks:
   • Apply Min-Max scaling to scale values to the range [0, 1].
   • Standardize features to have zero mean and unit variance.
   • Compare the two approaches and explain when each would be preferred.

Exercise 4: Activation Functions

Objective: Compare the behavior of activation functions.

1. Given:
   • Input values: $X=[-2,-1,0,1,2]$.
2. Tasks:
   • Compute outputs for ReLU, LeakyReLU ($\alpha =0.01$), Sigmoid, and Tanh functions.
   • Sketch the graphs of these functions.
   • Discuss the advantages and disadvantages of each function.

Exercise 5: Gradient Checking

Objective: Verify the correctness of computed gradients.

1. Setup:
   • Loss function: $L=\frac{1}{2}(y-\hat{y}{)}^2$, where $y=1$, $\hat{y}=W\cdot x+b$.
   • Parameters: $W=0.5$, $x=2$, $b=0.1$.
2. Tasks:
   • Compute the analytical gradient $\frac{\partial L}{\partial W}$.
   • Use numerical approximation to compute: $\frac{\partial L}{\partial W}\approx \frac{L(W+ϵ)-L(W-ϵ)}{2ϵ}$ for $ϵ={10}^{-4}$.
   • Compare the two results and explain any differences.

Exercise 6: Regularization

Objective: Understand the effect of regularization on weight updates.

1. Setup:
   • Weights: $W=[1,-2,0.5]$.
   • Regularization: L2 with $\lambda =0.01$.
2. Tasks:
   • Compute the weight penalty term $\lambda \sum W^2$.
   • Update the weights using gradient descent with learning rate $\eta =0.1$ and include the regularization term.
   • Discuss how regularization affects model training.

Exercise 7: Neural Network Error Analysis

Objective: Analyze and identify potential issues in a neural network setup.

1. Setup:
   • A neural network has:
     • Inputs: $X=[1,2]$
     • Weights: $W_1=[[0.5,0.2],[-0.3,0.8]]$, $W_2=[0.7,-0.6]$
     • Biases: $b_1=[0.1,-0.1]$, $b_2=0.2$
     • ReLU activation for the hidden layer and Sigmoid for the output.
   • Output: $\hat{y}=0.4$
   • True label: $y=1$.
2. Tasks:
   • Calculate the loss using binary cross-entropy.
   • Determine whether the weight initialization could cause vanishing/exploding gradients.
   • Propose changes to the network architecture or initialization to improve performance.