Foundations of the Calculus of Variations and Optimal Control

Introduction: From Points to Curves

In standard differential calculus, one typically seeks a point x that minimizes a function f(x). However, in physics and engineering, the goal is often to find a curve or a function y(x) that minimizes a specific quantity. For example, one might ask: "What is the path of least time for a bead sliding down a frictionless wire between two points?"

This specific historical problem is known as the Brachistochrone problem, posed by Johann Bernoulli in 1696. Its solution marked the birth of the Calculus of Variations. Unlike finding a stationary point, the solution requires finding a stationary function among a family of admissible curves connecting two boundary points.

The Calculus of Variations

The fundamental object of study in this field is the functional. A functional J[y] assigns a real number to a function y(x). In its simplest form, the functional depends on a single independent variable x, the function y, and its derivative y'.

Consider the problem of minimizing the integral:

Here, L is known as the Lagrangian of the functional. We seek a function y*(x) that yields a smaller value for J than any other neighboring function that satisfies the boundary conditions y(x₀) = y₀ and y(x₁) = y₁.

The core concept involves introducing a small perturbation (variation) to the optimal curve and requiring that the change in the functional is zero to the first order.

The Euler-Lagrange Equation

To solve this minimization problem, we consider a varied path y(x) + ε η(x), where η(x) is an arbitrary function vanishing at the endpoints, and ε is a small parameter. By taking the derivative of J with respect to ε and evaluating it at ε = 0, we arrive at the necessary condition for an extremum.

This necessary condition is expressed by the famous Euler-Lagrange equation:

This is a second-order ordinary differential equation (ODE). Solving this ODE with the appropriate boundary conditions yields the extremal function y*(x). The power of this equation lies in its universality; it applies to any Lagrangian L that is sufficiently smooth.

If the Lagangian does not explicitly depend on x, a first integral of the Euler-Lagrange equation exists, often referred to as the Beltrami identity. This simplifies many problems by reducing the order of the differential equation.

Introduction to Optimal Control

While the Calculus of Variations provides the tools to optimize a functional, it generally treats the derivative y' as an independent variable free to vary at every point. Optimal Control Theory introduces a more structured scenario involving dynamical systems.

In Optimal Control, we are concerned with steering a system (defined by state variables) from an initial state to a final state while minimizing a cost function. This process is governed by inputs called control variables.

The standard formulation of an optimal control problem consists of three main components:

1. System Dynamics: A set of differential equations 𝒙(t) = f(x(t), u(t), t), where x(t) is the state vector and u(t) is the control vector.
2. Cost Functional: A scalar quantity to be minimized, often encompassing both integral costs and terminal costs.
   J = Φ(x(t_f)) + ∫_t0^t_f L(x(t), u(t), t) dt
3. Constraints: These can include boundary conditions, state constraints (e.g., obstacles), and control constraints (e.g., |u(t)| ≤ 1).

Pontryagins Maximum Principle

The most celebrated result in optimal control is Pontryagins Maximum Principle (PMP), developed by Lev Pontryagin and his team in the 1950s. PMP provides necessary conditions for optimal control that generalize the Euler-Lagrange equation.

To apply PMP, we define the Hamiltonian H:

Here, λ(t) are the costates or adjoint variables. PMP states that for the optimal control u*(t) and the corresponding state x*(t), there exists a non-zero costate vector λ(t) such that:

• The state equation: 𝒙* = ∂H / ∂λ.
• The costate equation: 𝒙λ = -∂H / ∂x.
• The optimality condition: The Hamiltonian is minimized with respect to the control u at every instant:
  u*(t) = arg min_{u ∈ U} H(x*(t), u, λ(t), t)

This framework allows for the handling of constraints on u, which was difficult in the classical Calculus of Variations. If u is unconstrained, minimizing the Hamiltonian simply implies ∂H / ∂u = 0, which parallels the Euler-Lagrange equation.

The Hamilton-Jacobi-Bellman Equation

While PMP gives necessary conditions for a specific optimal trajectory, the Hamilton-Jacobi-Bellman (HJB) equation offers a sufficient condition for global optimality. It relies on the concept of Dynamic Programming, introduced by Richard Bellman.

The HJB equation is a partial differential equation (PDE) for the value function V(x, t), which represents the minimum cost-to-go from state x at time t.

The HJB equation is stated as:

Solving the HJB equation yields the optimal control law u(x, t) in feedback form. While theoretically powerful, solving the HJB equation can be computationally intractable for high-dimensional systems (the "curse of dimensionality"), making PMP the preferred method for many complex trajectory optimization problems.

Conclusion

The Calculus of Variations and Optimal Control are deeply interconnected mathematical disciplines. The Calculus of Variations provides the historical and theoretical bedrock, treating optimization as the search for extremal functions. Optimal Control builds upon this foundation by explicitly modeling manipulable inputs and dynamical constraints, making it the standard language for modern automation, aerospace trajectory design, and robotics.

From the abstract curves of the Brachistochrone to the rigorous necessary conditions of Pontryagins Maximum Principle, these tools allow engineers and mathematicians to shape the behavior of dynamic systems precisely and optimally. Understanding the transition from the Euler-Lagrange equation to the Hamiltonian formalism is key to mastering the optimization of motion and decision-making in continuous time.