This release adds a new algorithm: Soft Actor-Critic (SAC).

Soft Actor-Critic

-implement the original paper: "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" https://arxiv.org/abs/1801.01290 #398

• implement the improvement of SAC paper: "Soft Actor-Critic Algorithms and Applications" https://arxiv.org/abs/1812.05905 #399
• extend SAC to work directly for discrete environment using GumbelSoftmax distribution (custom)

Roboschool (continuous control) Benchmark

Note that the Roboschool reward scales are different from MuJoCo's.

Env. \ Alg.	SAC
RoboschoolAnt	2451.55 graph
RoboschoolHalfCheetah	2004.27 graph
RoboschoolHopper	2090.52 graph
RoboschoolWalker2d	1711.92 graph

LunarLander (discrete control) Benchmark

Trial graph	Moving average

This release corrects and optimizes all the algorithms from benchmarking on Atari. New metrics are introduced. The lab's API is also redesigned for simplicity.

Benchmark

• full algorithm benchmark on 4 core Atari environments #396
• LunarLander benchmark #388 and BipedalWalker benchmark #377

This benchmark table is pulled from PR396. See the full benchmark results here.

Env. \ Alg.	A2C (GAE)	A2C (n-step)	PPO	DQN	DDQN+PER
Breakout graph	389.99 graph	391.32 graph	425.89 graph	65.04 graph	181.72 graph
Pong graph	20.04 graph	19.66 graph	20.09 graph	18.34 graph	20.44 graph
Qbert graph	13,328.32 graph	13,259.19 graph	13,691.89 graph	4,787.79 graph	11,673.52 graph
Seaquest graph	892.68 graph	1,686.08 graph	1,583.04 graph	1,118.50 graph	3,751.34 graph

Algorithms

• correct and optimize all algorithms with benchmarking #315 #327 #328 #361
• introduce "shared" and "synced" Hogwild modes for distributed training #337 #340
• streamline and optimize agent components too

Now, the full list of algorithms are:

• SARSA
• DQN, distributed-DQN
• Double-DQN, Dueling-DQN, PER-DQN
• REINFORCE
• A2C, A3C (N-step & GAE)
• PPO, distributed-PPO
• SIL (A2C, PPO)
  All the algorithms can be ran in distributed mode also; which in some cases they have their special names (mentioned above)

Environments

• implement vector environments #302
• implement more environment wrappers for preprocessing. Some replay memories are retired. #303 #330 #331 #342
• make Lab Env wrapper interface identical to gym #304, #305, #306, #307

API

• all the Space objects (AgentSpace, EnvSpace, AEBSpace, InfoSpace) are retired, to opt for a much simpler interface. #335 #348
• major API simplification throughout

Analysis

• rework analysis, introduce new metrics: strength, sample efficiency, training efficiency, stability, consistency #347 #349
• fast evaluation using vectorized env for rigorous_eval #390 , and using inference for fast eval #391

Search

• update and rework Ray search #350 #351

Improve installation

• #288 split out yarn installation as extra step

Improve functions

• #283 #284 redesign fitness slightly
• #281 simplify PER sample index
• #287 #290 improve DQN polyak and network switching
• #291 refactor advantage functions
• #295 #296 refactor various utils, fix PyTorch inplace ops

Add out layer activation

• #300 add out layer activation

This release corrects and optimizes all the algorithms from benchmarking on Atari. New metrics are introduced. The lab's API is also redesigned for simplicity.

Algorithms

• correct and optimize all algorithms with benchmarking #315 #327 #328
• introduce "shared" and "synced" Hogwild modes for distributed training #337 #340
• streamline and optimize agent components too

Now, the full list of algorithms are:

• SARSA
• DQN, distributed-DQN
• Double-DQN, Dueling-DQN, PER-DQN
• REINFORCE
• A2C, A3C (N-step & GAE)
• PPO, distributed-PPO
• SIL (A2C, PPO)
  All the algorithms can be ran in distributed mode also; which in some cases they have their special names (mentioned above)

Environments

• implement vector environments #302
• implement more environment wrappers for preprocessing. Some replay memories are retired. #303 #330 #331 #342
• make Lab Env wrapper interface identical to gym #304, #305, #306, #307

API

• all the Space objects (AgentSpace, EnvSpace, AEBSpace, InfoSpace) are retired, to opt for a much simpler interface. #335 #348
• major API simplification throughout

Analysis

• rework analysis, introduce new metrics: strength, sample efficiency, training efficiency, stability, consistency #347 #349

Search

• update and rework Ray search #350 #351

Benchmark

• pending upload

This release corrects and optimizes all the algorithms from benchmarking on Atari. New metrics are introduced. The lab's API is also redesigned for simplicity.

Algorithms

• correct and optimize all algorithms with benchmarking #315 #327 #328
• introduce "shared" and "synced" Hogwild modes for distributed training #337 #340
• streamline and optimize agent components too

Now, the full list of algorithms are:

• SARSA
• DQN, distributed-DQN
• Double-DQN, Dueling-DQN, PER-DQN
• REINFORCE
• A2C, A3C (N-step & GAE)
• PPO, distributed-PPO
• SIL (A2C, PPO)
  All the algorithms can be ran in distributed mode also; which in some cases they have their special names (mentioned above)

Environments

• implement vector environments #302
• implement more environment wrappers for preprocessing. Some replay memories are retired. #303 #330 #331 #342
• make Lab Env wrapper interface identical to gym #304, #305, #306, #307

API

• all the Space objects (AgentSpace, EnvSpace, AEBSpace, InfoSpace) are retired, to opt for a much simpler interface. #335 #348
• major API simplification throughout

Analysis

• rework analysis, introduce new metrics: strength, sample efficiency, training efficiency, stability, consistency #347 #349

Search

• update and rework Ray search #350 #351

Benchmark

• pending upload

Improve installation

• #288 split out yarn installation as extra step

Improve functions

• #283 #284 redesign fitness slightly
• #281 simplify PER sample index
• #287 #290 improve DQN polyak and network switching
• #291 refactor advantage functions
• #295 #296 refactor various utils, fix PyTorch inplace ops

Add out layer activation

• #300 add out layer activation

Eval rework

#275 #278 #279 #280

This release adds an eval mode that is the same as OpenAI baseline. Spawn 2 environments, 1 for training and 1 more eval. In the same process (blocking), run training as usual, then at ckpt, run an episode on eval env and update stats.

The logic for the stats are the same as before, except the original body.df is now split into two: body.train_df and body.eval_df. Eval df uses the main env stats except for t, reward to reflect progress on eval env. Correspondingly, session analysis also produces both versions of data.

Data from body.eval_df is used to generate session_df, session_graph, session_fitness_df, whereas the data from body.train_df is used to generate a new set of trainsession_df, trainsession_graph, trainsession_fitness_df for debugging.

The previous process-based eval functionality is kept, but is now considered as parallel_eval. This can be useful for more robust checkpointing and eval.

Refactoring

#279

• purge useless computations
• properly and efficiently gather and organize all update variable computations.

This also speeds up run time by x2. For Atari Beamrider with DQN on V100 GPU, manual benchmark measurement gives 110 FPS for training every 4 frames, while eval achieves 160 FPS. This translates to 10M frames in roughly 24 hours.

Docker image kengz/slm_lab:v3.0.0 released

Add Retro Eval

• #270 add retro eval mode to run fail online eval sessions. Use command yarn retro_eval data/reinforce_cartpole_2018_01_22_211751
• #272 #273 fix eval saving 0 index to eval_session_df causing trial analysis to break; add reset_index for safety

fix Boltzmann spec

• #271 change Boltzmann spec to use Categorical instead of the wrong Argmax

misc

• #273 update colorlover package to proper pip after they fixed division error
• #274 remove unused torchvision package to lighten build

v3.1.0: L1 fitness norm, code and spec refactor, online eval

L1 fitness norm (breaking change)

• change fitness vector norm from L2 to L1 for intuitiveness and non-extreme values

code and spec refactor

• #254 PPO cleanup: remove hack and restore minimization scheme
• #255 remove use_gae and use_nstep param to infer from lam, num_step_returns
• #260 fix decay start_step offset, add unit tests for rate decay methods
• #262 make epi start from 0 instead of 1 for code logic consistency
• #264 switch max_total_t, max_epi to max_tick and max_tick_unit for directness. retire graph_x for the unit above
• #266 add Atari fitness std, fix CUDA coredump issue
• #269 update gym, remove box2d hack

Online Eval mode

#252 #257 #261 #267
Evaluation sessions during training on a subprocess. This does not interfere with the training process, but spawns multiple subprocesses to do independent evaluation, which then adds to an eval file, and at the end a final eval will finish and plot all the graphs and save all the data for eval.

• enabled by meta spec 'training_eval'
• configure NUM_EVAL_EPI in analysis.py
• update enjoy and eval mode syntax. see README.
• change ckpt behavior to use e.g. tag ckpt-epi10-totalt1000
• add new eval mode to lab. runs on a checkpoint file. see below

Eval Session

• add a proper eval Session which loads from the ckpt like above, and does not interfere with existing files. This can be ran on terminal, and it's also used by the internal eval logic, e.g. command python run_lab.py data/dqn_cartpole_2018_12_20_214412/dqn_cartpole_t0_spec.json dqn_cartpole eval@dqn_cartpole_t0_s2_ckpt-epi10-totalt1000
• when eval session is done, it will average all of its ran episodes and append to a row in an eval_session_df.csv
• after that it will delete the ckpt files it had just used (to prevent large storage)
• then, it will run a trial analysis to update eval_trial_graph.png, and an accompanying trial_df as average of all session_dfs

How eval mode works

• checkpoint will save the models using the scheme which records its epi and total_t. This allows one to eval using the ckpt model
• after creating ckpt files, if spec.meta.training_eval in trainmode, a subprocess will launch using the ckpt prepath to run an eval Session, using the same way abovepython run_lab.py data/dqn_cartpole_2018_12_20_214412/dqn_cartpole_t0_spec.json dqn_cartpole eval@dqn_cartpole_t0_s2_ckpt-epi10-totalt1000`
• eval session runs as above. ckpt will now run at the starting timestep, ckpt timestep, and at the end
• the main Session will wait for the final eval session and it's final eval trial to finish before closing, to ensure that other processes like zipping wait for them.

Example eval trial graph:

V3: PyTorch 1.0, faster Neural Network, Variable Scheduler

Docker image kengz/slm_lab:v3.0.0 released

PRs included #240 #241 #239 #238 #244 #248

PyTorch 1.0 and parallel CUDA

• switch to PyTorch 1.0 with various improvements and parallel CUDA fix

new Neural Network API (breaking changes)

To accommodate more advanced features and improvements, all the networks have been improved with better spec and code design, faster operations, and added features

• single-tail networks will now not use list but a single tail for fast output compute (for loop is slow)
• use PyTorch optim.lr_scheduler for learning rate decay. retire old methods.
• more efficient spec format for network, clip_grad, lr_scheduler_spec
• fix and add proper generalization for ConvNet and RecurrentNet
• add full basic network unit tests

DQN

• rewrite DQN loss for 2x speedup and code simplicity. extend to SARSA
• retire MultitaskDQN for HydraDQN

Memory

• add OnpolicyConcatReplay
• standardize preprocess_state logic in onpolicy memories

Variable Scheduler (breaking spec changes)

• implement variable decay class VarScheduler similar to pytorch's LR scheduler. use clock with flexible scheduling units epi or total_t
• unify VarScheduler to use standard clock.max_tick_unit specified from env
• retire action_policy_update, update agent spec to explore_var_spec
• replace entropy_coef with entropy_coef_spec
• replace clip_eps with clip_eps_spec (PPO)
• update all specs

Math util

• move decay methods to math_util.py
• move math_util.py from algorithm/ to lib/

env max tick (breaking spec changes)

• spec/variable renamings:
  • max_episode to max_epi
  • max_timestep to max_t
  • save_epi_frequency to save_frequency
  • traininig_min_timestep to training_start_step
• allow env to stop based on max_epi as well as max_total_t. propagate clock unit usage
• introduce max_tick, max_tick_unit properties to env and clock from above
• allow save_frequency to use the same units accordingly
• update Pong and Beamrider to use max_total_t as end-condition

Update Ray to reenable CUDA in search

• update ray from 0.3.1 to 0.5.3 to address broken GPU with pytorch 1.0.0
• to fix CUDA not discovered in Ray worker, have to manually set CUDA devices at ray remote function due to poor design.

Improved logging and Enjoy mode

#243 #245

• Best models checkpointing measured using the the reward_ma
• Early termination if the environment is solved
• method for logging learning rate to session data frame needed to be updated after move to PyTorch lr_scheduler
• Also removed training_net from the mean learning rate reported in the session dataframe since the learning rate doesn't change
• update naming scheme to work with enjoy mode
• unify and simplify prepath methods
• info_space now uses a ckpt for loading ckpt model. Example usage: yarn start pong.json dqn_pong enjoy@data/dqn_cartpole_2018_12_02_124127/dqn_cartpole_t0_s0_ckptbest
• update agent load and policy to properly set variables to end_val in enjoy mode
• random-seed env as well

Working Atari

#242
Atari benchmark had been failing, but the root cause had finally been discovered and fix: wrong image preprocessing. Image normalization cause the input values to be lowered by ~255, and the resultant loss is too small for optimizer.
PR #242 introduces:

• global environment preprocessor in the form of env wrapper borrowed from OpenAI baselines, in env/wrapper.py
• a TransformImage to do the proper image transform: grayscale, downsize, and shape from w,h,c to PyTorch format c,h,w
• a FrameStack which uses LazyFrames for efficiency to replace the agent-specific Atari stack frame preprocessing. This simplifies the Atari memories
• update convnet to use honest shape (c,h,w) without extra transform, and remove its expensive image axis permutation since input now is in the right shape
• update Vizdoom to produce (c,h,w) shape consistent with convnet input expectation

Tuned parameters will be obtained and released next version.

Attached is a quick training curve on Pong, DQN, where the solution avg is +18:

Add VizDoom environment

#222 #224

• add new OnPolicyImageReplay and ImageReplay memories
• add VizDoom environment, thanks to @joelouismarino

Add NN Weight Initialization functionality

#223 #225

• allow specification of NN weight init function in spec, thanks to @mwcvitkovic

Update Plotly to v3

#221

• move to v3 to allow Python based (instead of bash) image saving for stability

Fixes

• #207 fix PPO loss function broken during refactoring
• #217 fix multi-device CUDA parallelization in grad assignment

This release adds PPOSIL, fixes some small issues with continuous actions, and PPO ratio computation.

Implementations

#145 Implement PPOSIL. Improve debug logging
#143 add Arch installer thanks to @angel-ayala

Bug Fixes

#138 kill hanging processes of Electron for plotting
#145 fix PPO wrong graph update sequence causing ratio to be 1. Fix continuous action output construction. add guards.
#146 fix continuous actions and add full tests

This release adds some new implementations, and fixes some bugs from first benchmark runs.

Implementations

#127 Self-Imitation Learning
#128 Checkpointing for saving models
#129 Dueling Networks

Bug Fixes

#132 GPU test-run fixes
#133 fix ActorCritic family loss compute getting detached, and linux plotting issues, add SHA to generated specs