MearcAll papers

arXiv 1707.06347 · cs.LG

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_aeafd401
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a “surrogate” objective function using stochastic … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_10cc1fd2
Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_df1bead6
Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_76a92720
Q-learning (with function approximation) fails on many simple problems 1 and is poorly understood, vanilla policy gradient methods have poor data effiency and robustness; and trust region policy optimization (TRPO) … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fd6f9282
We propose a novel objective with clipped probability ratios, which forms a pessimistic estimate (i.e., lower bound) of the performance of the policy.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a50b2b1
Hence, to achieve our goal of a first-order algorithm that emulates the monotonic improvement of TRPO, experiments show that it is not sufficient to simply choose a fixed penalty coefficient … (excerpt)

Trust Region Methods · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_504519ca
4 Adaptive KL Penalty Coefficient Another approach, which can be used as an alternative to the clipped surrogate objective, or in addition to it, is to use a penalty on … (excerpt)

Clipped Surrogate Objective · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a3ac0ca3
Note that the score is negative for the setting without clipping or penalties, because for one environment (half cheetah) it leads to a very negative score, which is worse than … (excerpt)

Comparison of Surrogate Objectives · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ab626c59
The three tasks we test on are (1) RoboschoolHumanoid: forward locomotion only, (2) RoboschoolHumanoidFlagrun: position of target is randomly varied every 200 timesteps or whenever the goal is reached, (3) … (excerpt)

Showcase in the Continuous Domain: Humanoid Running and Steering · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1707.06347) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.