RL-Adventure-2: Policy Gradients

PyTorch tutorial of: actor critic / proximal policy optimization / acer / ddpg / twin dueling ddpg / soft actor critic / generative adversarial imitation learning / hindsight experience replay

The deep reinforcement learning community has made several improvements to the policy gradient algorithms. This tutorial presents latest extensions in the following order:

1. Advantage Actor Critic (A2C)

• actor-critic.ipynb
• A3C Paper
• OpenAI blog

2. High-Dimensional Continuous Control Using Generalized Advantage Estimation

• gae.ipynb
• GAE Paper

3. Proximal Policy Optimization Algorithms

• ppo.ipynb
• PPO Paper
• OpenAI blog

4. Sample Efficient Actor-Critic with Experience Replay

• acer.ipynb
• ACER Paper

5. Continuous control with deep reinforcement learning

• ddpg.ipynb
• DDPG Paper

6. Addressing Function Approximation Error in Actor-Critic Methods

• td3.ipynb
• Twin Dueling DDPG Paper

7. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

• soft actor-critic.ipynb
• Soft Actor-Critic Paper

8. Generative Adversarial Imitation Learning

• gail.ipynb
• GAIL Paper

9. Hindsight Experience Replay

• her.ipynb
• HER Paper
• OpenAI Blog

If you get stuck…

• Remember you are not stuck unless you have spent more than a week on a single algorithm. It is perfectly normal if you do not have all the required knowledge of mathematics and CS.
• Carefully go through the paper. Try to see what is the problem the authors are solving. Understand a high-level idea of the approach, then read the code (skipping the proofs), and after go over the mathematical details and proofs.

RL Algorithms

Deep Q Learning tutorial: DQN Adventure: from Zero to State of the Art Awesome RL libs: rlkit @vitchyr, pytorch-a2c-ppo-acktr @ikostrikov, ACER @Kaixhin

Best RL courses

• Berkeley deep RL link
• Deep RL Bootcamp link
• David Silver's course link
• Practical RL link