ReAgent: Applied Reinforcement Learning Platform

---

Overview

ReAgent is an open source end-to-end platform for applied reinforcement learning (RL) developed and used at Facebook. ReAgent is built in Python and uses PyTorch for modeling and training and TorchScript for model serving. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, and optimized serving. For more detailed information about ReAgent see the white paper here: Platform.

The source code is available here: Source code.

The platform was once named “Horizon” but we have adopted the name “ReAgent” recently to emphasize its broader scope in decision making and reasoning.

Algorithms Supported

• Discrete-Action DQN

• Parametric-Action DQN

• Double DQN, Dueling DQN, Dueling Double DQN

• Distributional RL C51, QR-DQN

• Twin Delayed DDPG (TD3)

• Soft Actor-Critic (SAC)

Installation

ReAgent can be installed via. Docker or manually. Detailed instructions on how to install ReAgent can be found here: Installation.

Usage

The ReAgent Serving Platform (RASP) tutorial covers serving and training models and is available here: ReAgent Serving Platform (RASP).

Detailed instructions on how to use ReAgent can be found here: Usage.

License

ReAgent is released under a BSD license. Find out more about it here: License.

Citing

@article{gauci2018horizon,

title={Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform}, author={Gauci, Jason and Conti, Edoardo and Liang, Yitao and Virochsiri, Kittipat and Chen, Zhengxing and He, Yuchen and Kaden, Zachary and Narayanan, Vivek and Ye, Xiaohui}, journal={arXiv preprint arXiv:1811.00260}, year={2018}

}

Table of Contents

Getting Started

• Installation
  • Anaconda
  • Docker
• Tutorial
  • What is RASP?
  • Installing ReAgent
  • Store set-up
  • User simulator
  • Makin’ bacon
  • Straight Outta Context
• Usage
  • 1 - On-Policy RL Training
  • 2- Offline RL Training (Batch RL)
    • Step 1 - Create training data
    • Step 2 - Convert the data to the timeline format
    • Step 3 - Create the normalization parameters
    • Step 4 - Train model
    • Step 5 - Evaluate the Model
    • Step 6 - Visualize Results via Tensorboard

Advanced Topics

• Distributed Training
  • How distributed training works
  • Training on a single node
  • Training on multiple nodes
• Continuous Integration

Package Reference

• Evaluation
  • Submodules
  • ml.rl.evaluation.cpe module
  • ml.rl.evaluation.doubly_robust_estimator module
  • ml.rl.evaluation.evaluation_data_page module
  • ml.rl.evaluation.evaluator module
  • ml.rl.evaluation.ranking_evaluator module
  • ml.rl.evaluation.sequential_doubly_robust_estimator module
  • ml.rl.evaluation.weighted_sequential_doubly_robust_estimator module
  • ml.rl.evaluation.world_model_evaluator module
  • Module contents
• Models
  • Submodules
  • ml.rl.models.actor module
  • ml.rl.models.base module
  • ml.rl.models.bcq module
  • ml.rl.models.categorical_dqn module
  • ml.rl.models.cem_planner module
  • ml.rl.models.convolutional_network module
  • ml.rl.models.dqn module
  • ml.rl.models.dueling_q_network module
  • ml.rl.models.dueling_quantile_dqn module
  • ml.rl.models.example_sequence_model module
  • ml.rl.models.fully_connected_network module
  • ml.rl.models.mdn_rnn module
  • ml.rl.models.no_soft_update_embedding module
  • ml.rl.models.parametric_dqn module
  • ml.rl.models.quantile_dqn module
  • ml.rl.models.seq2slate module
  • ml.rl.models.world_model module
  • Module contents
• Prediction
  • Submodules
  • ml.rl.prediction.dqn_torch_predictor module
  • ml.rl.prediction.predictor_wrapper module
  • Module contents
• Preprocessing
  • Submodules
  • ml.rl.preprocessing.batch_preprocessor module
  • ml.rl.preprocessing.feature_extractor module
  • ml.rl.preprocessing.identify_types module
  • ml.rl.preprocessing.normalization module
  • ml.rl.preprocessing.postprocessor module
  • ml.rl.preprocessing.preprocessor module
  • ml.rl.preprocessing.preprocessor_net module
  • ml.rl.preprocessing.sparse_to_dense module
  • Module contents
• Readers
  • Submodules
  • ml.rl.readers.base module
  • ml.rl.readers.data_streamer module
  • ml.rl.readers.json_dataset_reader module
  • ml.rl.readers.nparray_reader module
  • Module contents
• Simulators
  • Submodules
  • ml.rl.simulators.recsim module
  • Module contents
• Thrift
  • Subpackages
    • ml.rl.thrift.core package
      • Submodules
      • ml.rl.thrift.core.constants module
      • ml.rl.thrift.core.ttypes module
      • Module contents
  • Module contents
• Training
  • Subpackages
    • ml.rl.training.gradient_free package
      • Submodules
      • ml.rl.training.gradient_free.es_worker module
      • ml.rl.training.gradient_free.evolution_pool module
      • Module contents
    • ml.rl.training.ranking package
      • Submodules
      • ml.rl.training.ranking.ranking_trainer module
      • ml.rl.training.ranking.seq2slate_trainer module
      • Module contents
    • ml.rl.training.world_model package
      • Submodules
      • ml.rl.training.world_model.mdnrnn_trainer module
      • Module contents
  • Submodules
  • ml.rl.training.c51_trainer module
  • ml.rl.training.cem_trainer module
  • ml.rl.training.dqn_predictor module
  • ml.rl.training.dqn_trainer module
  • ml.rl.training.dqn_trainer_base module
  • ml.rl.training.imitator_training module
  • ml.rl.training.loss_reporter module
  • ml.rl.training.off_policy_predictor module
  • ml.rl.training.on_policy_predictor module
  • ml.rl.training.parametric_dqn_trainer module
  • ml.rl.training.qrdqn_trainer module
  • ml.rl.training.rl_dataset module
  • ml.rl.training.rl_trainer_pytorch module
  • ml.rl.training.sac_trainer module
  • ml.rl.training.sandboxed_predictor module
  • ml.rl.training.td3_trainer module
  • ml.rl.training.training_data_page module
  • Module contents
• Workflow
  • Submodules
  • ml.rl.workflow.base_workflow module
  • ml.rl.workflow.create_normalization_metadata module
  • ml.rl.workflow.dqn_workflow module
  • ml.rl.workflow.helpers module
  • ml.rl.workflow.page_handler module
  • ml.rl.workflow.parametric_dqn_workflow module
  • ml.rl.workflow.preprocess_handler module
  • ml.rl.workflow.transitional module
  • Module contents
• All Modules
  • ml package
    • Subpackages
      • ml.rl package
        • Subpackages
          • ml.rl.evaluation package
            • Submodules
            • ml.rl.evaluation.cpe module
            • ml.rl.evaluation.doubly_robust_estimator module
            • ml.rl.evaluation.evaluation_data_page module
            • ml.rl.evaluation.evaluator module
            • ml.rl.evaluation.ranking_evaluator module
            • ml.rl.evaluation.sequential_doubly_robust_estimator module
            • ml.rl.evaluation.weighted_sequential_doubly_robust_estimator module
            • ml.rl.evaluation.world_model_evaluator module
            • Module contents
          • ml.rl.models package
            • Submodules
            • ml.rl.models.actor module
            • ml.rl.models.base module
            • ml.rl.models.bcq module
            • ml.rl.models.categorical_dqn module
            • ml.rl.models.cem_planner module
            • ml.rl.models.convolutional_network module
            • ml.rl.models.dqn module
            • ml.rl.models.dueling_q_network module
            • ml.rl.models.dueling_quantile_dqn module
            • ml.rl.models.example_sequence_model module
            • ml.rl.models.fully_connected_network module
            • ml.rl.models.mdn_rnn module
            • ml.rl.models.no_soft_update_embedding module
            • ml.rl.models.parametric_dqn module
            • ml.rl.models.quantile_dqn module
            • ml.rl.models.seq2slate module
            • ml.rl.models.world_model module
            • Module contents
          • ml.rl.prediction package
            • Submodules
            • ml.rl.prediction.dqn_torch_predictor module
            • ml.rl.prediction.predictor_wrapper module
            • Module contents
          • ml.rl.preprocessing package
            • Submodules
            • ml.rl.preprocessing.batch_preprocessor module
            • ml.rl.preprocessing.feature_extractor module
            • ml.rl.preprocessing.identify_types module
            • ml.rl.preprocessing.normalization module
            • ml.rl.preprocessing.postprocessor module
            • ml.rl.preprocessing.preprocessor module
            • ml.rl.preprocessing.preprocessor_net module
            • ml.rl.preprocessing.sparse_to_dense module
            • Module contents
          • ml.rl.readers package
            • Submodules
            • ml.rl.readers.base module
            • ml.rl.readers.data_streamer module
            • ml.rl.readers.json_dataset_reader module
            • ml.rl.readers.nparray_reader module
            • Module contents
          • ml.rl.simulators package
            • Submodules
            • ml.rl.simulators.recsim module
            • Module contents
          • ml.rl.training package
            • Subpackages
              • ml.rl.training.gradient_free package
                • Submodules
                • ml.rl.training.gradient_free.es_worker module
                • ml.rl.training.gradient_free.evolution_pool module
                • Module contents
              • ml.rl.training.ranking package
                • Submodules
                • ml.rl.training.ranking.ranking_trainer module
                • ml.rl.training.ranking.seq2slate_trainer module
                • Module contents
              • ml.rl.training.world_model package
                • Submodules
                • ml.rl.training.world_model.mdnrnn_trainer module
                • Module contents
            • Submodules
            • ml.rl.training.c51_trainer module
            • ml.rl.training.cem_trainer module
            • ml.rl.training.dqn_predictor module
            • ml.rl.training.dqn_trainer module
            • ml.rl.training.dqn_trainer_base module
            • ml.rl.training.imitator_training module
            • ml.rl.training.loss_reporter module
            • ml.rl.training.off_policy_predictor module
            • ml.rl.training.on_policy_predictor module
            • ml.rl.training.parametric_dqn_trainer module
            • ml.rl.training.qrdqn_trainer module
            • ml.rl.training.rl_dataset module
            • ml.rl.training.rl_trainer_pytorch module
            • ml.rl.training.sac_trainer module
            • ml.rl.training.sandboxed_predictor module
            • ml.rl.training.td3_trainer module
            • ml.rl.training.training_data_page module
            • Module contents
          • ml.rl.workflow package
            • Submodules
            • ml.rl.workflow.base_workflow module
            • ml.rl.workflow.create_normalization_metadata module
            • ml.rl.workflow.dqn_workflow module
            • ml.rl.workflow.helpers module
            • ml.rl.workflow.page_handler module
            • ml.rl.workflow.parametric_dqn_workflow module
            • ml.rl.workflow.preprocess_handler module
            • ml.rl.workflow.transitional module
            • Module contents
        • Submodules
        • ml.rl.caffe_utils module
        • ml.rl.debug_on_error module
        • ml.rl.json_serialize module
        • ml.rl.parameters module
        • ml.rl.tensorboardX module
        • ml.rl.torch_utils module
        • ml.rl.types module
        • Module contents
    • Module contents

Others

• Github
• License