In a neural network, a neuron $N$ is a computation unit that takes $n$ input values, applies an affine transformation, then a typically non-linear activation function, and returns a single value.

That is, it is a function of the form

$$N:{\mathbb{R}}^n\to \mathbb{R},N(\mathbf{x})=\varphi (\mathbf{w}\cdot \mathbf{x}+b),$$

where

• $n$ is the number of inputs to the neuron

• $\mathbf{w}\in {\mathbb{R}}^n$ is a vector of weights,

• $b\in \mathbb{R}$ is a bias value, and

• $\varphi :\mathbb{R}\to \mathbb{R}$ is an activation function.

A layer in a neural network is a collections of neurons that operate in parallel on the same input vector.

A layer with $n_{\ell -1}$ inputs and $n_{\ell }$ neurons defines a function

$$F_{\ell }:{\mathbb{R}}^{\ell -1}\to {\mathbb{R}}^{\ell }$$

of the form

$$F_{\ell }(\mathbf{x})={\varphi }_{\ell }(W_{\ell }\mathbf{x}+{\mathbf{b}}_{\ell }),$$

where:

• $W_{\ell }\in {\mathbb{R}}^{n_{\ell }\times n_{\ell -1}}$ is the weight matrix (with the $i$th row representing the weights for the inputs to the $i$th neuron in the layer),

• ${\mathbf{b}}_{\ell }\in {\mathbb{R}}^{n_{\ell }}$ is the bias vector (with the $i$th entry representing the bias on the $i$th neuron in the layer,

• ${\varphi }_{\ell }:\mathbb{R}\to \mathbb{R}$ is an activation function, applied @componentwise,

• each coordinate of $F_{\ell }(\mathbf{x})$ is the output of a single neuron in that layer.

A layer takes an input vector, applies the same affine transformation to all neurons (via a shared weight matrix and bias vector), and then applies an @activation-function to each neuron's output.

Its output is the vector of all neuron outputs in that layer, and in this way, we can view a layer as a vector of neurons.

A neural network is a function obtained by @composing @finitely-many neurons arranged in layers:

$$F(\mathbf{x})=({\varphi }_L\circ A_L)\circ \cdots \circ ({\varphi }_1\circ A_1)(\mathbf{x}),$$

where each

• $A_{\ell }(\mathbf{x})=W_{\ell }\mathbf{x}+{\mathbf{b}}_{\ell }$ is an affine transformation,

• ${\varphi }_{\ell }$ is an @activation-function applied @componentwise,

• and $W_{\ell },{\mathbf{b}}_{\ell }$ are learnable parameters.