From ea002eea003abcf0107f36d5c159db7be22e0374 Mon Sep 17 00:00:00 2001 From: Pranav Gupta Date: Tue, 13 Aug 2024 16:36:30 +0530 Subject: [PATCH 1/4] Info about the new mountain car reward settings --- docs/environments/classical.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/environments/classical.md b/docs/environments/classical.md index 3f80ce85..e4a7a724 100644 --- a/docs/environments/classical.md +++ b/docs/environments/classical.md @@ -9,9 +9,16 @@ Multi-objective versions of classical Gymnasium's environments. | Env | Obs/Action spaces | Objectives | Description | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`mo-mountaincar-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/)
| Continuous / Discrete | `[time_penalty, reverse_penalty, forward_penalty]` | Classic Mountain Car env, but with extra penalties for the forward and reverse actions. From [Vamplew et al. 2011](https://www.researchgate.net/publication/220343783_Empirical_evaluation_methods_for_multiobjective_reinforcement_learning_algorithms). | -| [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | +[`mo-mountaincar-3d-v0`] **
| Continuous / Discrete| `[time_penalty, move_penalty, speed_objective]` | The forward and backward penalties have been merged into the move penalty and a speed objective has been introduced which gives the positive reward equivalent to the car's speed at that time step.* | +[`mo-mountaincar-timemove-v0`] **
| Continuous / Discrete | `[time_penalty, move_penalty]`| Class Mountain Car env but an extra penalty for moving backwards or forwards merged into a move penalty. | +[`mo-mountaincar-timespeed-v0`] **
| Continuous / Discrete| `[time_penalty, speed_penalty]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* +| [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | | [`mo-lunar-lander-v2`](https://mo-gymnasium.farama.org/environments/mo-lunar-lander/)
| Continuous / Discrete or Continuous | `[landed, shaped_reward, main_engine_fuel, side_engine_fuel]` | MO version of the `LunarLander-v2` [environment](https://gymnasium.farama.org/environments/box2d/lunar_lander/). Objectives defined similarly as in [Hung et al. 2022](https://openreview.net/forum?id=AwWaBXLIJE). | +*An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. + +**Read more about these environments and the detailed reasoning behind them at [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing) + ```{toctree} :hidden: :glob: From c8f610becc08ac7ceb93d56a552ebbcfd82273cd Mon Sep 17 00:00:00 2001 From: Pranav Gupta Date: Tue, 13 Aug 2024 16:38:52 +0530 Subject: [PATCH 2/4] grammatical error fixes --- docs/environments/classical.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/environments/classical.md b/docs/environments/classical.md index e4a7a724..b70a9396 100644 --- a/docs/environments/classical.md +++ b/docs/environments/classical.md @@ -11,13 +11,13 @@ Multi-objective versions of classical Gymnasium's environments. | [`mo-mountaincar-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/)
| Continuous / Discrete | `[time_penalty, reverse_penalty, forward_penalty]` | Classic Mountain Car env, but with extra penalties for the forward and reverse actions. From [Vamplew et al. 2011](https://www.researchgate.net/publication/220343783_Empirical_evaluation_methods_for_multiobjective_reinforcement_learning_algorithms). | [`mo-mountaincar-3d-v0`] **
| Continuous / Discrete| `[time_penalty, move_penalty, speed_objective]` | The forward and backward penalties have been merged into the move penalty and a speed objective has been introduced which gives the positive reward equivalent to the car's speed at that time step.* | [`mo-mountaincar-timemove-v0`] **
| Continuous / Discrete | `[time_penalty, move_penalty]`| Class Mountain Car env but an extra penalty for moving backwards or forwards merged into a move penalty. | -[`mo-mountaincar-timespeed-v0`] **
| Continuous / Discrete| `[time_penalty, speed_penalty]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* +[`mo-mountaincar-timespeed-v0`] **
| Continuous / Discrete| `[time_penalty, speed_objective]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* | [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | | [`mo-lunar-lander-v2`](https://mo-gymnasium.farama.org/environments/mo-lunar-lander/)
| Continuous / Discrete or Continuous | `[landed, shaped_reward, main_engine_fuel, side_engine_fuel]` | MO version of the `LunarLander-v2` [environment](https://gymnasium.farama.org/environments/box2d/lunar_lander/). Objectives defined similarly as in [Hung et al. 2022](https://openreview.net/forum?id=AwWaBXLIJE). | *An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. -**Read more about these environments and the detailed reasoning behind them at [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing) +**Read more about these environments and the detailed reasoning behind them in [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing) ```{toctree} :hidden: From e2c14f5d42b5c9c9998a7104074f85d76cd1bca5 Mon Sep 17 00:00:00 2001 From: Pranav Gupta Date: Fri, 23 Aug 2024 09:32:21 +0100 Subject: [PATCH 3/4] Minor Changes --- docs/environments/classical.md | 7 ++++--- mo_gymnasium/envs/mountain_car/mountain_car.py | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/environments/classical.md b/docs/environments/classical.md index b70a9396..c721d67b 100644 --- a/docs/environments/classical.md +++ b/docs/environments/classical.md @@ -9,15 +9,16 @@ Multi-objective versions of classical Gymnasium's environments. | Env | Obs/Action spaces | Objectives | Description | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`mo-mountaincar-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/)
| Continuous / Discrete | `[time_penalty, reverse_penalty, forward_penalty]` | Classic Mountain Car env, but with extra penalties for the forward and reverse actions. From [Vamplew et al. 2011](https://www.researchgate.net/publication/220343783_Empirical_evaluation_methods_for_multiobjective_reinforcement_learning_algorithms). | -[`mo-mountaincar-3d-v0`] **
| Continuous / Discrete| `[time_penalty, move_penalty, speed_objective]` | The forward and backward penalties have been merged into the move penalty and a speed objective has been introduced which gives the positive reward equivalent to the car's speed at that time step.* | -[`mo-mountaincar-timemove-v0`] **
| Continuous / Discrete | `[time_penalty, move_penalty]`| Class Mountain Car env but an extra penalty for moving backwards or forwards merged into a move penalty. | -[`mo-mountaincar-timespeed-v0`] **
| Continuous / Discrete| `[time_penalty, speed_objective]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* +[`mo-mountaincar-3d-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete| `[time_penalty, move_penalty, speed_objective]` | The forward and backward penalties have been merged into the move penalty and a speed objective has been introduced which gives the positive reward equivalent to the car's speed at that time step.* | +[`mo-mountaincar-timemove-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete | `[time_penalty, move_penalty]`| Class Mountain Car env but an extra penalty for moving backwards or forwards merged into a move penalty. | +[`mo-mountaincar-timespeed-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincar/) **
| Continuous / Discrete| `[time_penalty, speed_objective]` | Class Mountain Car env but an extra positive objective of speed which gives the positive reward equivalent to the car's speed at that time step.* | [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | | [`mo-lunar-lander-v2`](https://mo-gymnasium.farama.org/environments/mo-lunar-lander/)
| Continuous / Discrete or Continuous | `[landed, shaped_reward, main_engine_fuel, side_engine_fuel]` | MO version of the `LunarLander-v2` [environment](https://gymnasium.farama.org/environments/box2d/lunar_lander/). Objectives defined similarly as in [Hung et al. 2022](https://openreview.net/forum?id=AwWaBXLIJE). | *An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. **Read more about these environments and the detailed reasoning behind them in [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing) + ```{toctree} :hidden: diff --git a/mo_gymnasium/envs/mountain_car/mountain_car.py b/mo_gymnasium/envs/mountain_car/mountain_car.py index 0c2c5dbb..f49cf5ce 100644 --- a/mo_gymnasium/envs/mountain_car/mountain_car.py +++ b/mo_gymnasium/envs/mountain_car/mountain_car.py @@ -19,7 +19,7 @@ class MOMountainCar(MountainCarEnv, EzPickle): - reverse penalty: -1.0 for each time step the action is 0 (reverse) - forward penalty: -1.0 for each time step the action is 2 (forward) - Alternatively, the reward can be changed with the following options: + #Alternatively, the reward can be changed with the following options: - add_speed_objective: Add an extra objective corresponding to the speed of the car. - remove_move_penalty: Remove the reverse and forward objectives. - merge_move_penalty: Merge reverse and forward penalties into a single penalty. From 5c2662315a9bde818583bfc0ea5d77c8d4359379 Mon Sep 17 00:00:00 2001 From: Lucas Alegre Date: Fri, 23 Aug 2024 10:47:34 +0200 Subject: [PATCH 4/4] pre-commit --- docs/environments/classical.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/environments/classical.md b/docs/environments/classical.md index c721d67b..0d15285e 100644 --- a/docs/environments/classical.md +++ b/docs/environments/classical.md @@ -15,7 +15,7 @@ Multi-objective versions of classical Gymnasium's environments. | [`mo-mountaincarcontinuous-v0`](https://mo-gymnasium.farama.org/environments/mo-mountaincarcontinuous/)
| Continuous / Continuous | `[time_penalty, fuel_consumption_penalty]` | Continuous Mountain Car env, but with penalties for fuel consumption. | | [`mo-lunar-lander-v2`](https://mo-gymnasium.farama.org/environments/mo-lunar-lander/)
| Continuous / Discrete or Continuous | `[landed, shaped_reward, main_engine_fuel, side_engine_fuel]` | MO version of the `LunarLander-v2` [environment](https://gymnasium.farama.org/environments/box2d/lunar_lander/). Objectives defined similarly as in [Hung et al. 2022](https://openreview.net/forum?id=AwWaBXLIJE). | -*An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. +*An additional objective was introduced to prevent the agent from converging to the local maxima due to a lack of reward signal for the static action. **Read more about these environments and the detailed reasoning behind them in [`Pranav Gupta's Dissertation`](https://drive.google.com/file/d/1yT6hlavYZGmoB2phaIBX_5hbibA3Illa/view?usp=sharing)