Evolutionary Reinforcement Learning

Reinforcement learning (RL) has achieved remarkable successes in games, robotics, and control, but its reliance on gradientbased optimization can make it vulnerable to local minima, sparse rewards, and brittle hyperparameter settings. Evolutionary reinforcement learning (ERL) combines the strengths of evolutionary algorithms (EAs) with traditional RL, using populations of agents that evolve through selection, mutation, and recombination while also learning from gradientbased updates.

Why Merge Evolutionary Methods with RL?

• Exploration Power: Evolutionary operators can produce large, diverse policy variations, helping agents escape flat reward landscapes where gradient methods stall.
• Robustness to NonDifferentiable Environments: EAs do not require differentiable rewards or policies, allowing them to operate in settings where backpropagation is impossible.
• Parallelism: Populations can be evaluated in parallel, making efficient use of modern multicore and GPU clusters.
• Complementary Learning Signals: Gradient updates provide finegrained adjustments, while evolutionary steps provide coarse, global jumps.

Core Concepts

1. PopulationBased Policy Representation

Each individual in the population encodes a policy usually a neural network with its own set of weights. The population size can range from a few dozen to several thousand depending on computational budget.

2. Fitness Evaluation

Fitness is the cumulative reward obtained by rolling out the policy in the environment. The same environment steps used for RL training are reused for evaluation, ensuring a fair comparison.

3. Evolutionary Operators

• Selection: Common schemes include tournament selection, rankbased selection, or fitnessproportionate (roulettewheel) selection.
• Mutation: Random perturbations of weights (Gaussian noise) or structural changes (adding/removing neurons).
• Crossover (Recombination): Mixing parameters from two parent policies, e.g., by uniform crossover or parameter averaging.

4. GradientBased Learning Within the Population

After a fixed number of environment interactions, each individual may run a few steps of a conventional RL algorithm (e.g., DDPG, PPO, SAC). The gradients refine the policy locally, while the evolutionary loop preserves diversity.

Popular ERL Architectures

Algorithmic Sketch of a Basic ERL Loop

1. Initialise a population P of N policies with random weights.2. For each generation:   a. Evaluate every policy on the environment  fitness.   b. Select a parent subset based on fitness.   c. Apply mutation/crossover to create offspring.   d. Replace the lowestfitness individuals with offspring.   e. Choose the best M individuals and run a few gradientbased RL updates      (e.g., PPO epochs) using their replay buffers.   f. Optionally share experience buffers across the population.3. Repeat until convergence or budget exhausted.

Benefits Observed in Empirical Studies

• Faster Convergence on Sparse Rewards: Evolutionary exploration discovers rewarding states that pure RL misses.
• Improved Stability: Populations smooth out the variance caused by stochastic gradients.
• Better Generalisation: Diverse policies learned through evolution tend to transfer more effectively to slightly altered tasks.

Challenges and Open Questions

• Sample Efficiency: Evolutionary evaluations are expensive; hybrid methods must balance gradient updates with population size.
• Credit Assignment: When mixing gradients and mutations, it can be unclear which operator contributed to performance gains.
• Scalability of Crossover: Defining meaningful recombination for deep networks remains an active research area.
• Hyperparameter Interaction: Learning rates, mutation strengths, and selection pressure interact in nontrivial ways.

Typical Applications

Evolutionary reinforcement learning has been applied to:

• Robotic locomotion and manipulation (e.g., learning quadruped gait).
• Procedurally generated video games where reward signals are sparse.
• Autonomous vehicle control under diverse weather and road conditions.
• Neural architecture search combined with RL for taskspecific policy networks.

Getting Started

If you want to experiment with ERL, consider the following Python ecosystem:

• OpenAI Evolution Strategies a lightweight ES implementation.
• RLpyt modular RL library that can be extended with evolutionary loops.
• NEATPython for evolving network topologies.
• Frameworks such as Ray RLlib provide builtin population based training (PBT) utilities that can be adapted for ERL.

Typical steps for a first prototype:

1. Choose an environment (e.g., Gymnasium Pendulum or MuJoCo HalfCheetah).
2. Implement a simple genetic algorithm that mutates neuralnetwork weights.
3. Add a PPO learner that updates the topk individuals every few generations.
4. Monitor fitness trends and diversity metrics (e.g., pairwise weight cosine similarity).

Future Directions

Research is moving toward tighter integration of evolution and deep RL:

• MetaEvolution: Using metalearning to adapt mutation distributions on the fly.
• Differentiable Evolution: Making crossover and selection differentiable so that they can be optimized jointly with policy gradients.
• Coevolution of Environments: Evolving both agents and curricula to accelerate learning.
• Hybrid Multiobjective Optimization: Balancing reward maximisation with other objectives such as energy consumption or safety.

By exploiting the complementary strengths of evolution and reinforcement learning, ERL offers a versatile toolbox for tackling complex, noisy, and poorly understood decisionmaking problems.

Further reading: Salimans et al., Evolution Strategies as a Scalable Alternative to Reinforcement Learning, 2017; Jaderberg et al., Population Based Training of Neural Networks, 2017; K. V. Ng., Evolutionary Reinforcement Learning, 2022.