Dynamical Systems Theory and Neural Networks
At first glance, neural networks belong to computer science while dynamical systems theory belongs to mathematics and physics. Yet at a deeper level, they speak the same language: the evolution of states under repeated transformation. Once this connection is made explicit, many phenomena in deep learning—training stability, exploding gradients, attractors, memory, and even expressivity—can be reinterpreted through the lens of nonlinear dynamics.
A dynamical system, in its simplest discrete form, is defined by an iteration
\[ x_{t+1} = F(x_t), \]
where \(x_t \in \mathbb{R}^n\) is the state at time \(t\), and \(F\) is a nonlinear map. A feedforward neural network with \(L\) layers is structurally identical to such an iteration. If we denote activations by \(h_\ell\), weights by \(W_\ell\), biases by \(b_\ell\), and nonlinearities by \(\sigma\), then each layer obeys
\[ h_{\ell+1} = \sigma(W_\ell h_\ell + b_\ell). \]
Depth is therefore discrete time. A deep network is a finite-time trajectory of a nonlinear dynamical system in activation space.
This viewpoint becomes even more literal in recurrent neural networks, where the same transformation is applied repeatedly:
\[ h_{t+1} = \sigma(W h_t + U x_t + b). \]
Here, the hidden state evolves exactly like a nonlinear dynamical system driven by an input signal \(x_t\). Questions about memory reduce to questions about stability of trajectories and persistence of information in the state.
Fixed Points, Attractors, and Representation
In dynamical systems theory, a fixed point satisfies
\[ x^* = F(x^*). \]
Stability is determined by the Jacobian \(DF(x^*)\). If all eigenvalues satisfy \(|\lambda_i| < 1\), the fixed point is locally attracting. In neural networks, similar reasoning governs signal propagation. Consider a deep linear network where \(h_{\ell+1} = W h_\ell\). Then
\[ h_L = W^L h_0. \]
The eigenvalues of \(W\) determine whether activations vanish, explode, or remain stable as depth increases. The same spectral radius condition that appears in stability theory appears in deep learning under the name of exploding and vanishing gradients.
Even nonlinear networks can be analyzed by linearizing around trajectories. If we perturb a state \(h_\ell \to h_\ell + \delta h_\ell\), then to first order
\[ \delta h_{\ell+1} = D\sigma(W_\ell h_\ell + b_\ell) \, W_\ell \, \delta h_\ell. \]
The product of these Jacobians across layers determines gradient flow during backpropagation. From a dynamical perspective, training stability is governed by the Lyapunov exponents of this multiplicative process.
Continuous Depth and Neural ODEs
When the number of layers becomes large and each transformation becomes small, the discrete iteration approaches a differential equation. If we write
\[ h_{\ell+1} = h_\ell + \Delta t \, f(h_\ell, \theta_\ell), \]
and take the limit \(\Delta t \to 0\), we obtain
\[ \frac{dh(t)}{dt} = f(h(t), \theta(t)). \]
This is the foundation of neural ordinary differential equations. The network is no longer a stack of layers but a flow in state space. Expressivity corresponds to the geometry of this flow, and training corresponds to adjusting the vector field \(f\).
In this setting, concepts such as trajectories, invariant manifolds, and attractors become literal geometric objects shaping representation learning.
Loss Landscapes as Gradient Flows
Training itself is another dynamical system, this time in parameter space. Gradient descent follows
\[ \theta_{k+1} = \theta_k - \eta \nabla_\theta \mathcal{L}(\theta_k). \]
In continuous time, this becomes the gradient flow equation
\[ \frac{d\theta}{dt} = - \nabla_\theta \mathcal{L}(\theta). \]
Critical points satisfy \(\nabla_\theta \mathcal{L} = 0\). Their stability depends on the Hessian \(H = \nabla^2_\theta \mathcal{L}\). If the eigenvalues of \(H\) are positive, the point is locally attracting; if some are negative, it is a saddle. Modern optimization theory in deep learning can therefore be interpreted as studying high-dimensional nonlinear flows.
Generalization properties are often connected to the geometry of these flows, such as flat versus sharp minima, which correspond to different stability characteristics of the dynamical system in parameter space.
Chaos, Expressivity, and Complexity
Nonlinear dynamical systems exhibit chaos when small perturbations grow exponentially over time. If two nearby states differ by \( \delta_0 \), their separation after \( t \) steps may scale like:
\[ \|\delta_t\| \approx e^{\lambda t}\|\delta_0\| \]
Here \( \lambda > 0 \) is a Lyapunov exponent. When \( \lambda \) is positive, arbitrarily small differences amplify exponentially.
A deep feedforward network with depth \(L\) computes:
\[ h^{(l+1)} = \phi\!\left(W^{(l)} h^{(l)} + b^{(l)}\right) \]
The full network is a composition of maps:
\[ f(x) = F^{(L-1)} \circ \cdots \circ F^{(0)}(x) \]
The Jacobian of the entire network is the product of layer Jacobians:
\[ J_f(x) = J^{(L-1)} \cdots J^{(0)} \]
If the typical singular value grows exponentially with depth:
\[ \sigma_{\text{typ}}(J_f) \sim e^{\alpha L}, \quad \alpha > 0 \]
then perturbations and gradients explode. If singular values decay:
\[ \sigma_{\text{typ}}(J_f) \sim e^{-\alpha L}, \quad \alpha > 0 \]
gradients vanish:
\[ \left\| \frac{\partial \mathcal{L}}{\partial h^{(0)}} \right\| \approx \|J_f^\top\| \left\| \frac{\partial \mathcal{L}}{\partial h^{(L)}} \right\| \to 0 \]
The critical regime corresponds to:
\[ \sigma_{\text{typ}}(J_f) \approx 1 \]
Initialization schemes aim to preserve activation variance:
\[ \mathrm{Var}\!\left(h^{(l)}\right) \approx \mathrm{Var}\!\left(h^{(l-1)}\right) \]
In this near-critical regime, networks balance expressivity and stability. Depth creates complexity through iteration, but usable learning depends on keeping Jacobian spectra close to unity rather than drifting into exponential growth or decay.
Closing Perspective
Neural networks are not merely function approximators. They are dynamical systems in activation space during inference and dynamical systems in parameter space during training. Stability theory explains gradient behavior. Spectral analysis explains depth scaling. Continuous limits lead to differential equations. Even phenomena such as memory, attractors, and chaos translate directly between the two fields.
Viewing deep learning through dynamical systems theory does more than provide elegant mathematics. It offers structural insight into why certain architectures work, why others fail, and how geometry governs learning in high dimensions.