Deep Dive
Biological neurons and how the brain works
The lecture begins by explaining how the human nervous system processes information to establish foundational concepts for artificial neural networks. When you touch a hot vessel, sensory neurons in your hand detect the stimulus (heat) as chemical signals. These signals travel through sensory neurons to the brain, where they're converted to electrical signals for faster transmission. The brain interprets this information and decides to pull your hand away, sending motor neuron signals back to your hand to execute the response. This entire process demonstrates the basic flow: stimulus detection through dendrites, processing in the cell body (soma), and output transmission through the axon. The axon terminals release chemical signals that the next neuron's dendrites receive, creating a continuous chain of communication. This biological mechanism directly inspired the design of artificial neural networks, which attempt to mimic this input-processing-output workflow in computational form.
Artificial neurons and their structure
An artificial neuron is the basic processing unit of a neural network, designed to mimic but not perfectly replicate biological neurons. Unlike biological neurons, artificial neurons are purely mathematical and computational models. Each artificial neuron receives one or more inputs, assigns weights to them, calculates their weighted sum, adds a bias term, and applies an activation function to produce output. For example, with inputs x1, x2, x3 and corresponding weights w1, w2, w3, the neuron calculates z = (x1×w1) + (x2×w2) + (x3×w3) + b, then applies activation function f to get final output y = f(z). The weights are learnable parameters that determine how strongly each input contributes to the neuron's decision. Just as dendrites receive signals and axons transmit them in biological neurons, artificial neurons gather weighted inputs and pass outputs forward to the next layer. This structure repeats across millions of neurons throughout the entire network, maintaining the unidirectional flow of information through the system.
Neural network layers and architecture
Standard neural networks consist of three types of layers organized in sequence. The input layer is the first layer that receives raw data features from the outside world (like area, bedrooms, location for house price prediction) and passes them forward. One or more hidden layers perform intermediate processing and computations on the input data, analyzing patterns and extracting meaningful representations that the network learns during training. These hidden layers transform raw information into increasingly abstract features; in image recognition, early hidden layers learn basic features like edges while deeper layers learn complex features like faces. Finally, the output layer produces the final prediction or result (spam/not spam, cat/dog/bird, or continuous numerical values depending on the task). Data flows unidirectionally from input through hidden layers to output. Each neuron in one layer connects to neurons in the next layer through weighted connections. The complete architecture shows how input data transforms progressively into meaningful output through layered computation, which is crucial for exams as this diagram needs to be drawn and explained clearly.
Weights and their importance in neural networks
Weights are learnable parameters associated with connections between neurons that determine how strongly each input affects the neuron's output. In a practical example, when predicting house prices using area, bedrooms, location, and age, location may have a much stronger effect on price than house age. Therefore, the network learns a larger weight for location and smaller weight for age. Weights can be positive or negative: positive weights contribute positively to the weighted sum (pushing output higher), while negative weights contribute negatively (pushing output lower). For instance, if input x=5 with weight w=2, the contribution is 10, increasing the sum. But if the same input has weight w=-2, the contribution becomes -10, decreasing the sum. All inputs to a neural network are not equally important, so the network learns different weights for each input during training. These weights get continuously adjusted throughout the training loop to minimize prediction error. While the overall effect on network output depends on the complete network structure and activation function used, not a single weight alone, weights collectively enable the network to learn which features matter most.
Training loops and how networks learn weights
Neural networks learn weights through iterative training loops that repeat many times over the training dataset. Initially, weights are assigned random values that don't yet represent any learned solution. During training, the network follows a specific cycle: first, input data is fed into the network; second, the network processes the data through its neurons and layers to generate predictions; third, the predicted output is compared with the actual expected output to calculate error; finally, if error exists, weights are adjusted according to the error signal. This entire cycle repeats many times. For example, if a network predicts a student will score 60 marks when they actually scored 75, the network recognizes a 15-mark error and adjusts its weights to improve future predictions. After this weight adjustment, the network processes the same or new data again, recalculates predictions, compares to actual values, and adjusts weights again. This repeated process allows the network to gradually learn useful patterns from the data rather than just memorizing a single dataset. Through many iterations, the network learns which weight adjustments reduce prediction error, enabling it to make increasingly accurate predictions. The deeper mechanism underlying this is called backpropagation, though the basic concept of continuous weight adjustment to reduce error is sufficient for this syllabus.
Advantages and limitations of neural networks
Neural networks offer significant advantages that explain their widespread adoption across many domains. They automatically learn patterns from data without requiring manual programming of each pattern. They can model complex and nonlinear relationships between inputs and outputs that simpler models cannot capture, which is why they perform well on difficult real-world problems with large, messy datasets. The network's weights can be continuously adjusted during training, allowing the model to adapt and improve as it sees more data. They have wide applicability across image recognition, face recognition, speech recognition, medical diagnosis, natural language processing, and recommendation systems. However, neural networks have notable limitations. Complex networks require large amounts of data and significant computational resources (powerful processors and GPUs) to train effectively, making them expensive and resource-intensive. Training large networks demands considerable time, especially as the number of layers and neurons increases. Most critically, neural networks suffer from poor interpretability: it is difficult to explain exactly why the network made a particular decision or produced a specific output because the learned weights lack human-readable meaning. This lack of interpretability is one of the biggest cited limitations, particularly in sensitive domains where decision explanations are crucial.