AI Applications · Matrix Mathematics

Neural Network Matrix Simulator

Discover how artificial neural networks use matrix multiplication, activation functions and gradient descent to learn patterns. Train a small neural network, inspect its weight matrices, and follow the calculations behind forward propagation and backpropagation.

Artificial intelligence Weight matrices Forward propagation Backpropagation Gradient descent

Interactive Neural Network Simulator

Train a neural network and inspect the matrices that make it work.

Runs in your browser
This educational model has two input neurons, one configurable hidden layer, and one sigmoid output neuron. It is a small binary-classification example, not a general-purpose large language model or production AI system.

How Neural Networks Use Matrices

A neural network is built from layers of interconnected neurons. Each layer receives numerical values, combines them using weights and biases, and applies an activation function.

Matrix notation makes these calculations compact and efficient. Rather than calculating every neuron separately, a complete layer can be calculated using a matrix multiplication.

\[ \boxed{Z^{[1]}=W^{[1]}X+b^{[1]}} \] \[ \boxed{A^{[1]}=f(Z^{[1]})} \] \[ \boxed{Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]}} \] \[ \boxed{\hat{Y}=\sigma(Z^{[2]})} \]

For this simulator, the dimensions are:

Parameter or valueDimensionsRole
X\(2\times1\)Input vector
W₁\(h\times2\)Input-to-hidden weights
b₁\(h\times1\)Hidden-layer bias
A₁\(h\times1\)Hidden-layer activations
W₂\(1\times h\)Hidden-to-output weights
b₂\(1\times1\)Output bias
Ŷ\(1\times1\)Predicted output

Here, \(h\) is the number of hidden neurons selected in the simulator. Each example is represented as a column vector.

Step 1: Forward Propagation

Forward propagation is the process of moving input data through the network to calculate a prediction.

1. Enter the input vector.

For example, an input pair can be represented as:

\[ X=\begin{bmatrix}1\\0\end{bmatrix} \]
2. Multiply by the first weight matrix.

The hidden layer combines the two inputs using its learned weights.

\[ Z^{[1]}=W^{[1]}X+b^{[1]} \]
3. Apply the hidden activation function.

A nonlinear function transforms the weighted sums into hidden activations.

\[ A^{[1]}=f(Z^{[1]}) \]
4. Calculate the output.

The second weight matrix combines hidden activations to produce the output prediction.

\[ Z^{[2]}=W^{[2]}A^{[1]}+b^{[2]} \] \[ \hat{Y}=\sigma(Z^{[2]}) \]

Step 2: Calculate the Loss

A neural network needs a way to measure how far its predictions are from the desired answers. This measurement is called the loss function.

This simulator uses binary cross-entropy, which is commonly used for binary classification with a sigmoid output.

\[ L=-\frac{1}{m}\sum_{i=1}^{m} \left[ y_i\log(\hat y_i) +(1-y_i)\log(1-\hat y_i) \right] \]

Here, \(m\) is the number of training examples, \(y_i\) is the correct target and \(\hat y_i\) is the network's predicted probability.

A lower loss means that the predictions are generally closer to the training targets. A loss close to zero indicates that the network is making confident and correct predictions on the training data.

Step 3: Backpropagation

Backpropagation calculates how each weight and bias contributes to the prediction error. It applies the chain rule, starting at the output layer and working backward through the hidden layer.

Output-layer error

For a sigmoid output combined with binary cross-entropy, the output error simplifies to:

\[ \delta^{[2]}=\hat Y-Y \]

Output weight gradient

The gradient of the output weight matrix is the outer product of the output error and the hidden activation vector:

\[ \boxed{ \frac{\partial L}{\partial W^{[2]}} =\delta^{[2]}(A^{[1]})^T } \]

Hidden-layer error

The output error is propagated backward through the second weight matrix. The derivative of the hidden activation function determines how much each hidden neuron contributed to the error.

\[ \delta^{[1]}= \left((W^{[2]})^T\delta^{[2]}\right) \circ f'(Z^{[1]}) \]

First-layer weight gradient

\[ \boxed{ \frac{\partial L}{\partial W^{[1]}} =\delta^{[1]}X^T } \]

The symbol \(\circ\) represents element-wise multiplication. The gradients for the bias vectors are the corresponding error terms.

Step 4: Gradient Descent and Weight Updates

Once the gradients have been calculated, gradient descent updates the weights and biases to reduce the loss.

\[ W_{\text{new}}=W_{\text{old}}- \eta\frac{\partial L}{\partial W} \] \[ b_{\text{new}}=b_{\text{old}}- \eta\frac{\partial L}{\partial b} \]

The learning rate \(\eta\) controls the size of each update. A very small learning rate may make learning slow, while an excessively large learning rate can cause unstable updates or prevent convergence.

The simulator uses full-batch gradient descent: it calculates the average gradient over all training examples before updating the parameters once per epoch.

Why XOR Is an Important Neural Network Example

XOR, or exclusive OR, returns 1 when its two inputs are different and 0 when they are equal.

Input x₁Input x₂XOR output
000
011
101
110

XOR is not linearly separable. A single linear classifier cannot separate its two output classes using one straight decision boundary.

A network with a nonlinear hidden layer can learn the pattern by combining multiple transformations. This makes XOR a useful demonstration of why nonlinear activation functions are important.

Training is not guaranteed to converge to the same weights or final loss on every random initialization. If XOR predictions are inaccurate, try another initialization, a different learning rate, or more epochs.

Activation Functions

Activation functions introduce nonlinearity into neural networks. Without nonlinear activation functions, stacking ordinary linear transformations would still produce a single linear transformation.

Sigmoid

\[ \sigma(x)=\frac{1}{1+e^{-x}} \] \[ \sigma'(x)=\sigma(x)(1-\sigma(x)) \]

The sigmoid function maps real values to the interval (0, 1). It is useful for interpreting a single binary output as a probability.

Tanh

\[ \tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}} \] \[ \frac{d}{dx}\tanh(x)=1-\tanh^2(x) \]

Tanh maps values to the interval (-1, 1). It is zero-centered and can be useful in a small hidden layer.

ReLU

\[ \operatorname{ReLU}(x)=\max(0,x) \] \[ \operatorname{ReLU}'(x)= \begin{cases} 0,&x\leq0\\ 1,&x>0 \end{cases} \]

ReLU is widely used in deep learning because of its simple computation. However, a neuron that remains in the negative region can receive zero gradient under ordinary ReLU.

Applications of Neural Networks in Artificial Intelligence

1. Image recognition

Neural networks can learn visual patterns from images. Their weight matrices transform pixel information into increasingly useful representations for recognizing objects and other visual features.

2. Natural language processing

Neural networks are used in language classification, translation, text generation, search and conversational AI. Large language models use much more complex architectures than the small network in this simulator.

3. Recommendation systems

Neural networks can learn relationships between users, products and preferences. Matrix operations help transform user and item representations into recommendation scores.

4. Forecasting

Neural networks can learn patterns from historical observations and use them to estimate future values in areas such as demand planning, energy consumption and financial time series.

5. Anomaly detection

A trained model can identify observations that differ from patterns learned from normal data. Applications include equipment monitoring, network security and quality control.

Frequently Asked Questions

How do matrices work in a neural network?

Matrices store the weights connecting one layer to another. Matrix multiplication combines the incoming activations with these weights, while bias vectors shift the results before activation functions are applied.

What is the difference between forward propagation and backpropagation?

Forward propagation calculates the network's prediction. Backpropagation calculates gradients that indicate how the weights and biases should change to reduce prediction error.

Why does the network need an activation function?

Activation functions introduce nonlinearity. Without them, a sequence of linear layers would be equivalent to one linear transformation, limiting the types of patterns the network could learn.

Can this simulator train a network to solve XOR?

Yes. The network has a nonlinear hidden layer and can learn the XOR pattern. Training results depend on the initialization, learning rate, activation function and number of epochs.

What does the learning rate control?

The learning rate controls the size of the weight and bias updates. A low value can make training slow, while a high value can make optimization unstable.

What does a decreasing loss mean?

A decreasing training loss means that the model is improving its predictions on the training examples according to the selected loss function. It does not automatically guarantee good performance on new, unseen data.

Is this a large language model?

No. This is a small feedforward neural network designed to teach matrix operations and basic learning. Large language models use substantially more parameters, larger datasets and more advanced architectures, including transformer networks.

Does the simulator send my data to a server?

No. The training calculations in this educational simulator run locally in your browser. No external AI API or model service is used.

Related Matrix and Linear Algebra Simulators

Explore the mathematical operations that also appear in machine learning.

Matrix Multiplication Simulator → Eigenvalue & Eigenvector Simulator → SVD Simulator → Matrix Iteration Simulator → Gaussian Elimination Simulator →