{ "cells": [ { "cell_type": "markdown", "id": "c030a571", "metadata": {}, "source": [ "## Illustration of PPO agent applied to a farming environment from farm-gym" ] }, { "cell_type": "markdown", "id": "13961cc2", "metadata": {}, "source": [ "### Imports:" ] }, { "cell_type": "code", "execution_count": 1, "id": "4712e8f8", "metadata": {}, "outputs": [], "source": [ "# classical libraries\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "\n", "# farm-gym pre-made environments\n", "import farmgym_games\n", "\n", "# RL library\n", "from rlberry.agents.torch import PPOAgent\n", "from rlberry.manager import AgentManager, evaluate_agents, plot_writer_data\n", "from rlberry.agents.torch.utils.training import model_factory_from_env\n", "from rlberry.envs import gym_make" ] }, { "cell_type": "markdown", "id": "5f10adde", "metadata": {}, "source": [ "### Settings :\n", "We'll use the `Farm1` environment" ] }, { "cell_type": "code", "execution_count": 2, "id": "612d28c7", "metadata": {}, "outputs": [], "source": [ "env_ctor, env_kwargs = gym_make, {\"id\": \"OldV21Farm1-v0\"} # rlberry is gym v0.21 compatible. Use \"id\"=\"Farm1-v0\" for gym v0.26 compatibility" ] }, { "cell_type": "markdown", "id": "ccfc2942-6c55-4f8b-8f4a-ba3063170e52", "metadata": {}, "source": [ "We use an architecture of $256\\times 256$ for both the value and policy neural network of ppo." ] }, { "cell_type": "code", "execution_count": 3, "id": "407c367b-1248-43a4-b970-f18da3c16385", "metadata": {}, "outputs": [], "source": [ "\n", "policy_configs = {\n", " \"type\": \"MultiLayerPerceptron\", # A network architecture\n", " \"layer_sizes\": (256, 256), # Network dimensions\n", " \"reshape\": False,\n", " \"is_policy\": True,\n", "}\n", "\n", "value_configs = {\n", " \"type\": \"MultiLayerPerceptron\",\n", " \"layer_sizes\": (256, 256),\n", " \"reshape\": False,\n", " \"out_size\": 1,\n", "}" ] }, { "cell_type": "markdown", "id": "b8b8127f", "metadata": {}, "source": [ "### Agent code:\n", "We use rlberry's [PPOAgent](https://rlberry.readthedocs.io/en/latest/generated/rlberry.agents.torch.PPOAgent.html#rlberry.agents.torch.PPOAgent). \n", "Remark that 365 days is the maximum lenght of an episode. This helps us to fix some of the parameters." ] }, { "cell_type": "code", "execution_count": 4, "id": "5dab734a", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;21m[INFO] 16:46: Running AgentManager fit() for PPOAgent with n_fit = 1 and max_workers = None. \u001b[0m\n", "/home/frost/.local/lib/python3.10/site-packages/gym/spaces/box.py:127: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", " logger.warn(f\"Box bound precision lowered by casting to {self.dtype}\")\n", "/home/frost/.local/lib/python3.10/site-packages/gym/spaces/box.py:127: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", " logger.warn(f\"Box bound precision lowered by casting to {self.dtype}\")\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:174: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed a `seed` instead of using `Env.seed` for resetting the environment random number generator.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:187: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed `options` to allow the environment initialisation to be passed additional information.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:195: UserWarning: \u001b[33mWARN: The result returned by `env.reset()` was not a tuple of the form `(obs, info)`, where `obs` is a observation and `info` is a dictionary containing additional information. Actual type: ``\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:219: DeprecationWarning: \u001b[33mWARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. \u001b[0m\n", " logger.deprecation(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:141: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method was expecting numpy array dtype to be float32, actual type: float64\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:165: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method is not within the observation space.\u001b[0m\n", " logger.warn(f\"{pre} is not within the observation space.\")\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 2209 | episode_rewards = 134.52428322598146 | total_episodes = 22 | fit/surrogate_loss = -17.20841407775879 | fit/entropy_loss = 1.3652242422103882 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 4267 | episode_rewards = 180.22205261986448 | total_episodes = 45 | fit/surrogate_loss = 3.766744375228882 | fit/entropy_loss = 1.2915725708007812 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 6460 | episode_rewards = -174.0 | total_episodes = 68 | fit/surrogate_loss = 1.7429035902023315 | fit/entropy_loss = 1.4351849555969238 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 8671 | episode_rewards = 0.0 | total_episodes = 85 | fit/surrogate_loss = -4.726207256317139 | fit/entropy_loss = 1.5167763233184814 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 10697 | episode_rewards = -62.0 | total_episodes = 101 | fit/surrogate_loss = 7.040808200836182 | fit/entropy_loss = 0.9549528956413269 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 12417 | episode_rewards = -16.0 | total_episodes = 118 | fit/surrogate_loss = 2.217423915863037 | fit/entropy_loss = 1.255665898323059 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 14796 | episode_rewards = 333.7999433709374 | total_episodes = 137 | fit/surrogate_loss = -4.331498622894287 | fit/entropy_loss = 1.1084470748901367 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 16895 | episode_rewards = -122.0 | total_episodes = 154 | fit/surrogate_loss = -1.8878077268600464 | fit/entropy_loss = 0.8683847784996033 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 18919 | episode_rewards = 0.0 | total_episodes = 167 | fit/surrogate_loss = -6.60181188583374 | fit/entropy_loss = 0.9932242035865784 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 20920 | episode_rewards = 0.0 | total_episodes = 185 | fit/surrogate_loss = -1.0884056091308594 | fit/entropy_loss = 0.9581738710403442 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 22962 | episode_rewards = -160.0 | total_episodes = 201 | fit/surrogate_loss = -6.423931121826172 | fit/entropy_loss = 0.916784405708313 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 25143 | episode_rewards = 0.0 | total_episodes = 214 | fit/surrogate_loss = 2.195204019546509 | fit/entropy_loss = 0.8561437129974365 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 27160 | episode_rewards = -32.0 | total_episodes = 228 | fit/surrogate_loss = -3.6802823543548584 | fit/entropy_loss = 0.576086699962616 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 28953 | episode_rewards = 0.0 | total_episodes = 244 | fit/surrogate_loss = -2.6257214546203613 | fit/entropy_loss = 0.5261974334716797 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:46: [PPOAgent[worker: 0]] | max_global_step = 31158 | episode_rewards = -36.0 | total_episodes = 263 | fit/surrogate_loss = 0.9804189205169678 | fit/entropy_loss = 0.8518242239952087 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 33170 | episode_rewards = 273.2909831213743 | total_episodes = 280 | fit/surrogate_loss = 0.08106783777475357 | fit/entropy_loss = 0.7627907395362854 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 35126 | episode_rewards = -14.0 | total_episodes = 301 | fit/surrogate_loss = 4.697272777557373 | fit/entropy_loss = 1.3077625036239624 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 37032 | episode_rewards = 0.0 | total_episodes = 316 | fit/surrogate_loss = -3.589635133743286 | fit/entropy_loss = 0.7756088376045227 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 39019 | episode_rewards = 123.06599629645945 | total_episodes = 337 | fit/surrogate_loss = 1.0958138704299927 | fit/entropy_loss = 0.9637163281440735 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 41039 | episode_rewards = 186.96929667917848 | total_episodes = 359 | fit/surrogate_loss = 2.217855930328369 | fit/entropy_loss = 1.3185549974441528 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 42864 | episode_rewards = 0.0 | total_episodes = 376 | fit/surrogate_loss = 3.6343650817871094 | fit/entropy_loss = 0.9253543615341187 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 44901 | episode_rewards = 0.0 | total_episodes = 395 | fit/surrogate_loss = 0.038416508585214615 | fit/entropy_loss = 0.9682783484458923 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 46947 | episode_rewards = 0.0 | total_episodes = 412 | fit/surrogate_loss = 0.11422909051179886 | fit/entropy_loss = 0.6158499121665955 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 48993 | episode_rewards = 0.0 | total_episodes = 431 | fit/surrogate_loss = -3.7807295322418213 | fit/entropy_loss = 0.724868893623352 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 50966 | episode_rewards = 0.0 | total_episodes = 450 | fit/surrogate_loss = -0.5481564998626709 | fit/entropy_loss = 0.5342087149620056 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 52844 | episode_rewards = 0.0 | total_episodes = 467 | fit/surrogate_loss = 4.418668270111084 | fit/entropy_loss = 0.8343883752822876 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 54957 | episode_rewards = 0.0 | total_episodes = 486 | fit/surrogate_loss = -2.375662088394165 | fit/entropy_loss = 0.8427497148513794 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 57007 | episode_rewards = 16.0 | total_episodes = 509 | fit/surrogate_loss = 3.9137234687805176 | fit/entropy_loss = 0.9979891180992126 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 59076 | episode_rewards = 0.0 | total_episodes = 530 | fit/surrogate_loss = -0.2222403734922409 | fit/entropy_loss = 1.0436657667160034 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 61089 | episode_rewards = 0.0 | total_episodes = 552 | fit/surrogate_loss = -3.2849655151367188 | fit/entropy_loss = 1.0299031734466553 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 63029 | episode_rewards = 0.0 | total_episodes = 575 | fit/surrogate_loss = 3.238835334777832 | fit/entropy_loss = 1.221884846687317 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 64886 | episode_rewards = 0.0 | total_episodes = 597 | fit/surrogate_loss = -0.7216285467147827 | fit/entropy_loss = 1.2553671598434448 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 66934 | episode_rewards = 112.86767683173309 | total_episodes = 619 | fit/surrogate_loss = 2.7049076557159424 | fit/entropy_loss = 1.272657871246338 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:47: [PPOAgent[worker: 0]] | max_global_step = 68898 | episode_rewards = 18.0 | total_episodes = 641 | fit/surrogate_loss = 4.479696750640869 | fit/entropy_loss = 1.2921851873397827 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 70786 | episode_rewards = 139.0729117340091 | total_episodes = 661 | fit/surrogate_loss = -0.4576759934425354 | fit/entropy_loss = 1.2917993068695068 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 72621 | episode_rewards = -4.0 | total_episodes = 682 | fit/surrogate_loss = -0.2897544205188751 | fit/entropy_loss = 1.2893353700637817 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 74505 | episode_rewards = 0.0 | total_episodes = 704 | fit/surrogate_loss = 3.921590805053711 | fit/entropy_loss = 1.2924174070358276 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 76314 | episode_rewards = 0.0 | total_episodes = 724 | fit/surrogate_loss = 0.5616796016693115 | fit/entropy_loss = 1.3364989757537842 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 78255 | episode_rewards = 0.0 | total_episodes = 746 | fit/surrogate_loss = 1.1299254894256592 | fit/entropy_loss = 1.2764776945114136 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 80013 | episode_rewards = 266.94249821150834 | total_episodes = 763 | fit/surrogate_loss = 4.023199558258057 | fit/entropy_loss = 1.2256656885147095 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 81715 | episode_rewards = 191.68010878323798 | total_episodes = 780 | fit/surrogate_loss = 8.112669944763184 | fit/entropy_loss = 1.2419995069503784 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 83590 | episode_rewards = 276.3426506341879 | total_episodes = 800 | fit/surrogate_loss = -3.397995948791504 | fit/entropy_loss = 1.180037498474121 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 85632 | episode_rewards = 201.806702212911 | total_episodes = 822 | fit/surrogate_loss = 11.917405128479004 | fit/entropy_loss = 1.1466782093048096 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 87484 | episode_rewards = 0.0 | total_episodes = 842 | fit/surrogate_loss = -4.252918243408203 | fit/entropy_loss = 1.2645856142044067 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 89279 | episode_rewards = 190.76893492535956 | total_episodes = 862 | fit/surrogate_loss = -0.5222299695014954 | fit/entropy_loss = 1.1454248428344727 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 91222 | episode_rewards = 259.8129481494122 | total_episodes = 882 | fit/surrogate_loss = -3.2302627563476562 | fit/entropy_loss = 1.1970560550689697 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 93047 | episode_rewards = 275.51577814466606 | total_episodes = 902 | fit/surrogate_loss = 8.772000312805176 | fit/entropy_loss = 1.1550569534301758 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 95024 | episode_rewards = 275.19492424250683 | total_episodes = 922 | fit/surrogate_loss = -2.4215118885040283 | fit/entropy_loss = 1.0462547540664673 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 96894 | episode_rewards = 0.0 | total_episodes = 942 | fit/surrogate_loss = 0.17885783314704895 | fit/entropy_loss = 1.0427830219268799 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 98867 | episode_rewards = 256.3459135016439 | total_episodes = 963 | fit/surrogate_loss = -5.808422565460205 | fit/entropy_loss = 0.9922869801521301 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 100767 | episode_rewards = 261.8378148045977 | total_episodes = 982 | fit/surrogate_loss = -0.07664193212985992 | fit/entropy_loss = 0.9912879467010498 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 102705 | episode_rewards = 250.53306513006976 | total_episodes = 1004 | fit/surrogate_loss = -5.870321273803711 | fit/entropy_loss = 1.03829824924469 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 104646 | episode_rewards = 31.12396309603301 | total_episodes = 1023 | fit/surrogate_loss = 2.2727718353271484 | fit/entropy_loss = 0.9626011848449707 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:48: [PPOAgent[worker: 0]] | max_global_step = 106558 | episode_rewards = 0.0 | total_episodes = 1043 | fit/surrogate_loss = 2.6146774291992188 | fit/entropy_loss = 0.8991783857345581 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 108486 | episode_rewards = 197.42062229739346 | total_episodes = 1063 | fit/surrogate_loss = 5.396429538726807 | fit/entropy_loss = 0.8701432943344116 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 110446 | episode_rewards = 293.10522661518553 | total_episodes = 1083 | fit/surrogate_loss = -4.111895561218262 | fit/entropy_loss = 0.925383448600769 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 112293 | episode_rewards = 0.0 | total_episodes = 1103 | fit/surrogate_loss = 2.3448612689971924 | fit/entropy_loss = 0.9377964735031128 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 114293 | episode_rewards = 270.8051013314283 | total_episodes = 1123 | fit/surrogate_loss = -9.176980972290039 | fit/entropy_loss = 0.9452497363090515 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 116258 | episode_rewards = 283.65709798467174 | total_episodes = 1142 | fit/surrogate_loss = 11.655402183532715 | fit/entropy_loss = 0.8301343321800232 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 118209 | episode_rewards = 0.0 | total_episodes = 1163 | fit/surrogate_loss = -5.4693922996521 | fit/entropy_loss = 0.913402795791626 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 120071 | episode_rewards = 302.01518974606034 | total_episodes = 1183 | fit/surrogate_loss = 7.949192047119141 | fit/entropy_loss = 0.9001861810684204 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 122083 | episode_rewards = 271.50668054340275 | total_episodes = 1204 | fit/surrogate_loss = -0.3852759897708893 | fit/entropy_loss = 0.8573261499404907 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 124048 | episode_rewards = 260.3091520178151 | total_episodes = 1223 | fit/surrogate_loss = -10.70649528503418 | fit/entropy_loss = 0.9148780107498169 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 125843 | episode_rewards = 274.7459547222016 | total_episodes = 1245 | fit/surrogate_loss = -2.958625555038452 | fit/entropy_loss = 0.8772678971290588 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 127866 | episode_rewards = 245.85348013466609 | total_episodes = 1267 | fit/surrogate_loss = -3.8383290767669678 | fit/entropy_loss = 1.009056806564331 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 129823 | episode_rewards = 240.46847748136793 | total_episodes = 1287 | fit/surrogate_loss = -5.578691482543945 | fit/entropy_loss = 1.0033317804336548 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 131699 | episode_rewards = 283.56211757181813 | total_episodes = 1306 | fit/surrogate_loss = 10.772649765014648 | fit/entropy_loss = 1.0038061141967773 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 133625 | episode_rewards = 0.0 | total_episodes = 1326 | fit/surrogate_loss = -2.2552425861358643 | fit/entropy_loss = 1.0329627990722656 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 135498 | episode_rewards = 238.22589699089986 | total_episodes = 1349 | fit/surrogate_loss = 1.1353161334991455 | fit/entropy_loss = 1.0362646579742432 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 137453 | episode_rewards = 273.6662891054696 | total_episodes = 1370 | fit/surrogate_loss = -7.482893943786621 | fit/entropy_loss = 1.1330825090408325 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 139374 | episode_rewards = -10.0 | total_episodes = 1389 | fit/surrogate_loss = 4.843024253845215 | fit/entropy_loss = 1.079302430152893 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 141359 | episode_rewards = 195.0093446034527 | total_episodes = 1409 | fit/surrogate_loss = -0.3425341248512268 | fit/entropy_loss = 1.1312304735183716 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:49: [PPOAgent[worker: 0]] | max_global_step = 143229 | episode_rewards = 0.0 | total_episodes = 1428 | fit/surrogate_loss = -3.96565318107605 | fit/entropy_loss = 1.0328818559646606 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 145122 | episode_rewards = 109.29218472669729 | total_episodes = 1448 | fit/surrogate_loss = 9.060868263244629 | fit/entropy_loss = 1.0830814838409424 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 147111 | episode_rewards = -22.0 | total_episodes = 1471 | fit/surrogate_loss = -2.5419631004333496 | fit/entropy_loss = 1.1042369604110718 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 149006 | episode_rewards = 279.4907295573699 | total_episodes = 1491 | fit/surrogate_loss = 7.033303260803223 | fit/entropy_loss = 0.9747352004051208 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 150915 | episode_rewards = 267.20894531089453 | total_episodes = 1510 | fit/surrogate_loss = 3.960771322250366 | fit/entropy_loss = 0.9919326305389404 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 152908 | episode_rewards = 256.2250103305771 | total_episodes = 1532 | fit/surrogate_loss = -0.007613801397383213 | fit/entropy_loss = 1.0438289642333984 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 154833 | episode_rewards = 245.52128312357308 | total_episodes = 1551 | fit/surrogate_loss = -10.362034797668457 | fit/entropy_loss = 1.1016902923583984 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 156778 | episode_rewards = 278.79854712757714 | total_episodes = 1572 | fit/surrogate_loss = 6.5567522048950195 | fit/entropy_loss = 1.0572316646575928 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 158651 | episode_rewards = 215.68329328843254 | total_episodes = 1591 | fit/surrogate_loss = 2.0005228519439697 | fit/entropy_loss = 1.053226351737976 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 160549 | episode_rewards = 0.0 | total_episodes = 1612 | fit/surrogate_loss = 4.874629974365234 | fit/entropy_loss = 1.0698308944702148 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 162579 | episode_rewards = 273.16079349912553 | total_episodes = 1632 | fit/surrogate_loss = -4.574504375457764 | fit/entropy_loss = 1.1021976470947266 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 164542 | episode_rewards = 312.9760725954338 | total_episodes = 1654 | fit/surrogate_loss = -9.858379364013672 | fit/entropy_loss = 1.090998888015747 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 166422 | episode_rewards = 267.55608565988587 | total_episodes = 1673 | fit/surrogate_loss = 7.012547016143799 | fit/entropy_loss = 1.1158734560012817 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 168419 | episode_rewards = 274.3685780078847 | total_episodes = 1693 | fit/surrogate_loss = -2.038301944732666 | fit/entropy_loss = 1.1137681007385254 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 170362 | episode_rewards = 228.2306574275855 | total_episodes = 1713 | fit/surrogate_loss = 1.7389625310897827 | fit/entropy_loss = 1.0956854820251465 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 172316 | episode_rewards = 241.30185727204105 | total_episodes = 1732 | fit/surrogate_loss = 2.5269970893859863 | fit/entropy_loss = 1.1481139659881592 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 174196 | episode_rewards = -12.0 | total_episodes = 1750 | fit/surrogate_loss = 1.630097508430481 | fit/entropy_loss = 1.1351419687271118 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 176069 | episode_rewards = 189.3297826228422 | total_episodes = 1771 | fit/surrogate_loss = -3.8621702194213867 | fit/entropy_loss = 1.1028456687927246 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 178013 | episode_rewards = 268.1629180735788 | total_episodes = 1789 | fit/surrogate_loss = 5.526572227478027 | fit/entropy_loss = 1.1215077638626099 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 179927 | episode_rewards = 0.0 | total_episodes = 1808 | fit/surrogate_loss = 1.0089154243469238 | fit/entropy_loss = 1.1004434823989868 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:50: [PPOAgent[worker: 0]] | max_global_step = 181719 | episode_rewards = 239.03986857129473 | total_episodes = 1826 | fit/surrogate_loss = 1.3062998056411743 | fit/entropy_loss = 1.0463488101959229 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 183621 | episode_rewards = 265.65874300593913 | total_episodes = 1845 | fit/surrogate_loss = -5.852258682250977 | fit/entropy_loss = 1.133949637413025 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 185565 | episode_rewards = 67.80445637614709 | total_episodes = 1865 | fit/surrogate_loss = 9.158913612365723 | fit/entropy_loss = 1.1922465562820435 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 187451 | episode_rewards = 282.8416307910926 | total_episodes = 1883 | fit/surrogate_loss = 2.0727455615997314 | fit/entropy_loss = 1.1585074663162231 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 189332 | episode_rewards = 205.79572679668 | total_episodes = 1902 | fit/surrogate_loss = 2.384558916091919 | fit/entropy_loss = 1.1495928764343262 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 191211 | episode_rewards = 0.0 | total_episodes = 1922 | fit/surrogate_loss = 0.5820152163505554 | fit/entropy_loss = 1.1726787090301514 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 193215 | episode_rewards = 271.4559898725673 | total_episodes = 1943 | fit/surrogate_loss = -2.57812237739563 | fit/entropy_loss = 1.0842591524124146 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 195103 | episode_rewards = 0.0 | total_episodes = 1962 | fit/surrogate_loss = 1.2391142845153809 | fit/entropy_loss = 1.194376826286316 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 196935 | episode_rewards = 171.46015329686514 | total_episodes = 1981 | fit/surrogate_loss = 1.2422070503234863 | fit/entropy_loss = 1.1019363403320312 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 198911 | episode_rewards = 170.06682650489532 | total_episodes = 2000 | fit/surrogate_loss = 0.9598288536071777 | fit/entropy_loss = 1.1226402521133423 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 200645 | episode_rewards = 16.0 | total_episodes = 2018 | fit/surrogate_loss = -3.168940305709839 | fit/entropy_loss = 1.0989958047866821 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 202747 | episode_rewards = 284.3230790190264 | total_episodes = 2037 | fit/surrogate_loss = 12.86429214477539 | fit/entropy_loss = 1.0760250091552734 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 204687 | episode_rewards = 110.60583113983654 | total_episodes = 2056 | fit/surrogate_loss = -10.055243492126465 | fit/entropy_loss = 1.0619055032730103 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 206614 | episode_rewards = 233.9282157160054 | total_episodes = 2075 | fit/surrogate_loss = -2.6406538486480713 | fit/entropy_loss = 1.0568320751190186 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 208579 | episode_rewards = 278.2330085934469 | total_episodes = 2096 | fit/surrogate_loss = 7.980196475982666 | fit/entropy_loss = 1.0911810398101807 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 210508 | episode_rewards = 260.32258285052814 | total_episodes = 2115 | fit/surrogate_loss = 5.786709308624268 | fit/entropy_loss = 1.0824633836746216 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 212373 | episode_rewards = 218.04632652768822 | total_episodes = 2134 | fit/surrogate_loss = -8.352408409118652 | fit/entropy_loss = 1.0934839248657227 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 214329 | episode_rewards = 271.3465157977871 | total_episodes = 2154 | fit/surrogate_loss = -4.062629222869873 | fit/entropy_loss = 1.0969864130020142 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 216267 | episode_rewards = 221.73785563376867 | total_episodes = 2172 | fit/surrogate_loss = 5.777106761932373 | fit/entropy_loss = 1.024115800857544 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:51: [PPOAgent[worker: 0]] | max_global_step = 218162 | episode_rewards = 216.0446248254139 | total_episodes = 2192 | fit/surrogate_loss = -3.4470815658569336 | fit/entropy_loss = 1.118396282196045 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 219938 | episode_rewards = 132.59513162358 | total_episodes = 2210 | fit/surrogate_loss = 3.124840021133423 | fit/entropy_loss = 1.1320301294326782 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 221842 | episode_rewards = 184.9587281019406 | total_episodes = 2229 | fit/surrogate_loss = -1.0659230947494507 | fit/entropy_loss = 1.031003475189209 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 223804 | episode_rewards = 0.0 | total_episodes = 2249 | fit/surrogate_loss = -2.4230310916900635 | fit/entropy_loss = 1.0751807689666748 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 225768 | episode_rewards = 244.20561467772313 | total_episodes = 2269 | fit/surrogate_loss = -3.263524055480957 | fit/entropy_loss = 1.0651490688323975 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 227599 | episode_rewards = 22.675146305186402 | total_episodes = 2287 | fit/surrogate_loss = 8.311609268188477 | fit/entropy_loss = 1.0777026414871216 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 229551 | episode_rewards = 0.0 | total_episodes = 2306 | fit/surrogate_loss = 2.6351635456085205 | fit/entropy_loss = 1.0248957872390747 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 231495 | episode_rewards = 285.339815879762 | total_episodes = 2326 | fit/surrogate_loss = 1.2256782054901123 | fit/entropy_loss = 1.0415252447128296 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 233411 | episode_rewards = 171.14700593599062 | total_episodes = 2346 | fit/surrogate_loss = -4.267335891723633 | fit/entropy_loss = 1.0866059064865112 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 235322 | episode_rewards = 270.56492642385905 | total_episodes = 2366 | fit/surrogate_loss = -3.5790865421295166 | fit/entropy_loss = 1.0592182874679565 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 237167 | episode_rewards = 215.45273470694696 | total_episodes = 2384 | fit/surrogate_loss = -7.527778625488281 | fit/entropy_loss = 1.069936752319336 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 239003 | episode_rewards = -20.0 | total_episodes = 2403 | fit/surrogate_loss = 5.44546365737915 | fit/entropy_loss = 1.1499285697937012 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 240983 | episode_rewards = 269.9111065055036 | total_episodes = 2422 | fit/surrogate_loss = 2.35046124458313 | fit/entropy_loss = 1.0309354066848755 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 242875 | episode_rewards = 318.0889483706464 | total_episodes = 2442 | fit/surrogate_loss = -4.4732279777526855 | fit/entropy_loss = 0.9553772211074829 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 244771 | episode_rewards = 104.58798503468927 | total_episodes = 2461 | fit/surrogate_loss = -1.5414801836013794 | fit/entropy_loss = 1.0260636806488037 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 246707 | episode_rewards = 0.0 | total_episodes = 2480 | fit/surrogate_loss = 3.820281982421875 | fit/entropy_loss = 1.0317150354385376 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 248705 | episode_rewards = 162.6760645376208 | total_episodes = 2500 | fit/surrogate_loss = 0.21251605451107025 | fit/entropy_loss = 1.0387043952941895 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 250589 | episode_rewards = 90.02615020038215 | total_episodes = 2519 | fit/surrogate_loss = 6.289228439331055 | fit/entropy_loss = 1.0504704713821411 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 252467 | episode_rewards = 214.89130183644414 | total_episodes = 2537 | fit/surrogate_loss = -6.966522693634033 | fit/entropy_loss = 1.026532530784607 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 254358 | episode_rewards = 308.76341924834617 | total_episodes = 2557 | fit/surrogate_loss = 12.154333114624023 | fit/entropy_loss = 1.0158194303512573 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:52: [PPOAgent[worker: 0]] | max_global_step = 256267 | episode_rewards = 283.83298564334217 | total_episodes = 2576 | fit/surrogate_loss = 2.852245807647705 | fit/entropy_loss = 1.0434391498565674 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 258172 | episode_rewards = 84.42354537626872 | total_episodes = 2597 | fit/surrogate_loss = -9.431618690490723 | fit/entropy_loss = 1.0137786865234375 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 260093 | episode_rewards = 276.9943537693276 | total_episodes = 2618 | fit/surrogate_loss = 0.1441255807876587 | fit/entropy_loss = 0.963430643081665 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 262045 | episode_rewards = 241.20177508175692 | total_episodes = 2641 | fit/surrogate_loss = 0.2748934328556061 | fit/entropy_loss = 0.990574836730957 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 264026 | episode_rewards = 268.78084189097274 | total_episodes = 2663 | fit/surrogate_loss = 3.5319406986236572 | fit/entropy_loss = 1.0552963018417358 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 265932 | episode_rewards = 242.44137571994833 | total_episodes = 2686 | fit/surrogate_loss = -9.579716682434082 | fit/entropy_loss = 0.9850586652755737 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 267875 | episode_rewards = 245.39531372725043 | total_episodes = 2708 | fit/surrogate_loss = 1.6980942487716675 | fit/entropy_loss = 1.0098484754562378 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 269777 | episode_rewards = 205.25879645323425 | total_episodes = 2726 | fit/surrogate_loss = 6.197882175445557 | fit/entropy_loss = 0.9618978500366211 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 271704 | episode_rewards = 283.7851303751837 | total_episodes = 2747 | fit/surrogate_loss = 0.685973048210144 | fit/entropy_loss = 0.9942080974578857 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 273541 | episode_rewards = 291.6241996708527 | total_episodes = 2766 | fit/surrogate_loss = -1.1453349590301514 | fit/entropy_loss = 1.101591944694519 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 275517 | episode_rewards = 282.02527238406714 | total_episodes = 2786 | fit/surrogate_loss = -6.049954414367676 | fit/entropy_loss = 1.0179861783981323 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 277360 | episode_rewards = 16.0 | total_episodes = 2806 | fit/surrogate_loss = 11.253918647766113 | fit/entropy_loss = 1.049414038658142 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 279367 | episode_rewards = 281.1290978810243 | total_episodes = 2825 | fit/surrogate_loss = -2.6344029903411865 | fit/entropy_loss = 1.0347164869308472 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 281167 | episode_rewards = 0.0 | total_episodes = 2844 | fit/surrogate_loss = 1.1064000129699707 | fit/entropy_loss = 0.9371924996376038 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 283179 | episode_rewards = 250.78104080966858 | total_episodes = 2863 | fit/surrogate_loss = 12.849495887756348 | fit/entropy_loss = 1.0081018209457397 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 285144 | episode_rewards = 287.55088389857764 | total_episodes = 2883 | fit/surrogate_loss = -3.789557695388794 | fit/entropy_loss = 0.9457067847251892 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 287026 | episode_rewards = 254.9618615560583 | total_episodes = 2901 | fit/surrogate_loss = 0.6840089559555054 | fit/entropy_loss = 0.9657329320907593 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 288904 | episode_rewards = 239.0461878378266 | total_episodes = 2919 | fit/surrogate_loss = -0.8061223030090332 | fit/entropy_loss = 1.0255134105682373 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 290812 | episode_rewards = 232.09967697204786 | total_episodes = 2938 | fit/surrogate_loss = -1.9753848314285278 | fit/entropy_loss = 0.9870120286941528 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:53: [PPOAgent[worker: 0]] | max_global_step = 292779 | episode_rewards = 293.1587391022088 | total_episodes = 2958 | fit/surrogate_loss = -3.4968020915985107 | fit/entropy_loss = 1.0315067768096924 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 294745 | episode_rewards = 186.26363476045046 | total_episodes = 2977 | fit/surrogate_loss = 7.592077255249023 | fit/entropy_loss = 1.0064972639083862 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 296645 | episode_rewards = 320.222622397055 | total_episodes = 2995 | fit/surrogate_loss = -1.5015491247177124 | fit/entropy_loss = 0.944429337978363 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 298664 | episode_rewards = 187.139482215965 | total_episodes = 3015 | fit/surrogate_loss = 3.7631969451904297 | fit/entropy_loss = 0.9813820123672485 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 300548 | episode_rewards = 328.55860880531134 | total_episodes = 3033 | fit/surrogate_loss = 1.5286048650741577 | fit/entropy_loss = 0.8878442049026489 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 302499 | episode_rewards = 190.13890473281583 | total_episodes = 3052 | fit/surrogate_loss = -2.894129991531372 | fit/entropy_loss = 0.9722768068313599 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 304394 | episode_rewards = 263.70690042441134 | total_episodes = 3071 | fit/surrogate_loss = 2.445435047149658 | fit/entropy_loss = 0.9777105450630188 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 306318 | episode_rewards = -36.0 | total_episodes = 3089 | fit/surrogate_loss = 3.277486801147461 | fit/entropy_loss = 0.9544088244438171 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 308290 | episode_rewards = 140.11722203309503 | total_episodes = 3108 | fit/surrogate_loss = -0.8213484287261963 | fit/entropy_loss = 0.925036609172821 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 310244 | episode_rewards = 291.42358804221857 | total_episodes = 3128 | fit/surrogate_loss = -0.5326124429702759 | fit/entropy_loss = 0.9467383027076721 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 312007 | episode_rewards = -2.0 | total_episodes = 3147 | fit/surrogate_loss = 1.535997748374939 | fit/entropy_loss = 0.8873388171195984 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 313877 | episode_rewards = 296.1952368101264 | total_episodes = 3167 | fit/surrogate_loss = -0.6225532293319702 | fit/entropy_loss = 0.8994657397270203 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 315845 | episode_rewards = 213.33218784048097 | total_episodes = 3186 | fit/surrogate_loss = 3.5437350273132324 | fit/entropy_loss = 0.8863654732704163 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 317720 | episode_rewards = 281.0779023134249 | total_episodes = 3205 | fit/surrogate_loss = -0.6437132954597473 | fit/entropy_loss = 0.8195433020591736 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 319621 | episode_rewards = 80.51307731884518 | total_episodes = 3223 | fit/surrogate_loss = 7.503363132476807 | fit/entropy_loss = 0.8818414807319641 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 321499 | episode_rewards = 263.8596860259897 | total_episodes = 3243 | fit/surrogate_loss = -13.773136138916016 | fit/entropy_loss = 0.8752599358558655 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 323401 | episode_rewards = 322.76747419685483 | total_episodes = 3262 | fit/surrogate_loss = 10.615867614746094 | fit/entropy_loss = 0.883058488368988 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 325272 | episode_rewards = 299.6771221698585 | total_episodes = 3280 | fit/surrogate_loss = 8.855615615844727 | fit/entropy_loss = 0.8485798835754395 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 327263 | episode_rewards = 277.88398165026155 | total_episodes = 3300 | fit/surrogate_loss = -7.257081985473633 | fit/entropy_loss = 0.8347086310386658 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 329167 | episode_rewards = 286.43851646392847 | total_episodes = 3320 | fit/surrogate_loss = 3.605395793914795 | fit/entropy_loss = 0.8551650643348694 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:54: [PPOAgent[worker: 0]] | max_global_step = 331089 | episode_rewards = 132.27738500552374 | total_episodes = 3339 | fit/surrogate_loss = 8.149479866027832 | fit/entropy_loss = 0.8589387536048889 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 333046 | episode_rewards = 294.103572316817 | total_episodes = 3358 | fit/surrogate_loss = -14.957820892333984 | fit/entropy_loss = 0.9307618737220764 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 334946 | episode_rewards = 359.3958632660398 | total_episodes = 3378 | fit/surrogate_loss = 12.715492248535156 | fit/entropy_loss = 0.9235902428627014 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 336786 | episode_rewards = 312.18158983836196 | total_episodes = 3397 | fit/surrogate_loss = 3.759201765060425 | fit/entropy_loss = 0.9244174361228943 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 338763 | episode_rewards = 0.0 | total_episodes = 3417 | fit/surrogate_loss = -3.8107359409332275 | fit/entropy_loss = 1.0092012882232666 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 340757 | episode_rewards = 297.4810194857149 | total_episodes = 3439 | fit/surrogate_loss = -3.7150909900665283 | fit/entropy_loss = 0.9435456395149231 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 342703 | episode_rewards = 325.8883162072191 | total_episodes = 3458 | fit/surrogate_loss = 8.093222618103027 | fit/entropy_loss = 0.9791096448898315 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 344588 | episode_rewards = 325.63559877100266 | total_episodes = 3477 | fit/surrogate_loss = 2.5148143768310547 | fit/entropy_loss = 0.9678980708122253 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 346553 | episode_rewards = 264.9606710241308 | total_episodes = 3500 | fit/surrogate_loss = -6.9905500411987305 | fit/entropy_loss = 0.9897512793540955 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 348515 | episode_rewards = 0.0 | total_episodes = 3521 | fit/surrogate_loss = 3.7423617839813232 | fit/entropy_loss = 1.0184897184371948 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 350317 | episode_rewards = 98.06277900045299 | total_episodes = 3541 | fit/surrogate_loss = 1.9921005964279175 | fit/entropy_loss = 1.0237897634506226 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 352120 | episode_rewards = 151.4683927095074 | total_episodes = 3559 | fit/surrogate_loss = 1.9690021276474 | fit/entropy_loss = 1.0069987773895264 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 354120 | episode_rewards = 202.49230749307205 | total_episodes = 3579 | fit/surrogate_loss = -0.41003695130348206 | fit/entropy_loss = 0.9336974024772644 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 356081 | episode_rewards = 301.713127133677 | total_episodes = 3599 | fit/surrogate_loss = 0.08953665941953659 | fit/entropy_loss = 0.9974294304847717 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 358059 | episode_rewards = 0.0 | total_episodes = 3619 | fit/surrogate_loss = 1.450573444366455 | fit/entropy_loss = 1.018861174583435 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 359913 | episode_rewards = 0.0 | total_episodes = 3638 | fit/surrogate_loss = -4.925795078277588 | fit/entropy_loss = 0.966366171836853 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 361833 | episode_rewards = 32.18952697529554 | total_episodes = 3660 | fit/surrogate_loss = -0.42690742015838623 | fit/entropy_loss = 0.9651669859886169 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 363767 | episode_rewards = 226.74993878617212 | total_episodes = 3679 | fit/surrogate_loss = -3.410787582397461 | fit/entropy_loss = 1.0196833610534668 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 365648 | episode_rewards = 0.0 | total_episodes = 3699 | fit/surrogate_loss = -1.4755240678787231 | fit/entropy_loss = 1.0036985874176025 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:55: [PPOAgent[worker: 0]] | max_global_step = 367506 | episode_rewards = 241.05188779928602 | total_episodes = 3720 | fit/surrogate_loss = 3.930341958999634 | fit/entropy_loss = 1.0016478300094604 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 369452 | episode_rewards = 284.35806858077245 | total_episodes = 3740 | fit/surrogate_loss = 10.060277938842773 | fit/entropy_loss = 0.9490146636962891 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 371410 | episode_rewards = 246.99344154863303 | total_episodes = 3762 | fit/surrogate_loss = -1.8628233671188354 | fit/entropy_loss = 0.9763739705085754 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 373356 | episode_rewards = 242.28637067749025 | total_episodes = 3781 | fit/surrogate_loss = -1.496168613433838 | fit/entropy_loss = 1.0062780380249023 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 375157 | episode_rewards = 292.2258261309967 | total_episodes = 3800 | fit/surrogate_loss = 4.843698978424072 | fit/entropy_loss = 1.0262356996536255 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 377121 | episode_rewards = 14.0 | total_episodes = 3821 | fit/surrogate_loss = -5.91718864440918 | fit/entropy_loss = 1.0802301168441772 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 379066 | episode_rewards = 218.7146904838953 | total_episodes = 3840 | fit/surrogate_loss = -3.436647415161133 | fit/entropy_loss = 1.0132417678833008 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 380992 | episode_rewards = 234.36918883589595 | total_episodes = 3860 | fit/surrogate_loss = 4.775086402893066 | fit/entropy_loss = 0.9952309131622314 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 382892 | episode_rewards = 284.07871555046415 | total_episodes = 3879 | fit/surrogate_loss = -5.776726245880127 | fit/entropy_loss = 1.0385886430740356 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 384869 | episode_rewards = 324.0644475643733 | total_episodes = 3899 | fit/surrogate_loss = 8.264321327209473 | fit/entropy_loss = 1.0189669132232666 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 386712 | episode_rewards = 15.0 | total_episodes = 3919 | fit/surrogate_loss = 5.558108329772949 | fit/entropy_loss = 0.9657159447669983 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 388612 | episode_rewards = 148.0068600827308 | total_episodes = 3938 | fit/surrogate_loss = -2.7608399391174316 | fit/entropy_loss = 0.9457671046257019 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 390497 | episode_rewards = 292.24682707125083 | total_episodes = 3958 | fit/surrogate_loss = 4.495976448059082 | fit/entropy_loss = 0.9412776827812195 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 392297 | episode_rewards = -12.0 | total_episodes = 3976 | fit/surrogate_loss = -6.394156455993652 | fit/entropy_loss = 0.8748922944068909 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 394354 | episode_rewards = 244.19843768782306 | total_episodes = 3994 | fit/surrogate_loss = 1.8958444595336914 | fit/entropy_loss = 0.9547006487846375 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 396337 | episode_rewards = 252.92778983197093 | total_episodes = 4013 | fit/surrogate_loss = 6.616912364959717 | fit/entropy_loss = 0.9458012580871582 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 398203 | episode_rewards = 286.96804348546357 | total_episodes = 4032 | fit/surrogate_loss = -8.990714073181152 | fit/entropy_loss = 0.9437640309333801 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 400107 | episode_rewards = 0.0 | total_episodes = 4050 | fit/surrogate_loss = 1.1266694068908691 | fit/entropy_loss = 0.9508050084114075 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 402072 | episode_rewards = 0.0 | total_episodes = 4069 | fit/surrogate_loss = 4.978913307189941 | fit/entropy_loss = 0.941914439201355 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 403943 | episode_rewards = 150.6610338132491 | total_episodes = 4087 | fit/surrogate_loss = -7.1340250968933105 | fit/entropy_loss = 1.004734992980957 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:56: [PPOAgent[worker: 0]] | max_global_step = 405719 | episode_rewards = 286.1085754713148 | total_episodes = 4105 | fit/surrogate_loss = -3.3456854820251465 | fit/entropy_loss = 0.9755523204803467 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 407602 | episode_rewards = 0.0 | total_episodes = 4124 | fit/surrogate_loss = 2.1803932189941406 | fit/entropy_loss = 0.949565589427948 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 409551 | episode_rewards = 156.19653705120353 | total_episodes = 4142 | fit/surrogate_loss = 1.3640680313110352 | fit/entropy_loss = 0.9335667490959167 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 411380 | episode_rewards = 350.7678222447295 | total_episodes = 4160 | fit/surrogate_loss = 4.900801181793213 | fit/entropy_loss = 0.9382438659667969 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 413195 | episode_rewards = 285.3994499195645 | total_episodes = 4177 | fit/surrogate_loss = -2.779069662094116 | fit/entropy_loss = 0.9222322702407837 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 415163 | episode_rewards = 288.03400255951937 | total_episodes = 4197 | fit/surrogate_loss = 2.873145818710327 | fit/entropy_loss = 0.9784928560256958 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 417081 | episode_rewards = 271.8003946442427 | total_episodes = 4216 | fit/surrogate_loss = 0.8303982019424438 | fit/entropy_loss = 0.9632890820503235 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 419070 | episode_rewards = 274.3610669586073 | total_episodes = 4235 | fit/surrogate_loss = -0.3378711938858032 | fit/entropy_loss = 0.9332031011581421 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 420935 | episode_rewards = 285.698652336808 | total_episodes = 4255 | fit/surrogate_loss = -1.6555614471435547 | fit/entropy_loss = 0.928404688835144 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 422869 | episode_rewards = 321.4248764564302 | total_episodes = 4276 | fit/surrogate_loss = 2.801839828491211 | fit/entropy_loss = 0.9713084697723389 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 424801 | episode_rewards = 347.5848249806207 | total_episodes = 4298 | fit/surrogate_loss = -5.931520462036133 | fit/entropy_loss = 0.9765328764915466 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 426753 | episode_rewards = 235.22405365927392 | total_episodes = 4319 | fit/surrogate_loss = 4.456506252288818 | fit/entropy_loss = 0.9539737105369568 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 428589 | episode_rewards = 293.8601487388092 | total_episodes = 4339 | fit/surrogate_loss = -1.527997374534607 | fit/entropy_loss = 1.0273936986923218 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 430491 | episode_rewards = 238.40763901062212 | total_episodes = 4360 | fit/surrogate_loss = 5.263881683349609 | fit/entropy_loss = 0.9276028275489807 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 432489 | episode_rewards = 298.29358894142774 | total_episodes = 4380 | fit/surrogate_loss = 1.954353928565979 | fit/entropy_loss = 1.0460476875305176 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 434335 | episode_rewards = 249.3733569086184 | total_episodes = 4400 | fit/surrogate_loss = 1.5861694812774658 | fit/entropy_loss = 0.9824934601783752 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 436158 | episode_rewards = 238.96283738341737 | total_episodes = 4417 | fit/surrogate_loss = -0.17691625654697418 | fit/entropy_loss = 0.9926157593727112 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 438063 | episode_rewards = 260.36137471838583 | total_episodes = 4435 | fit/surrogate_loss = 3.9432201385498047 | fit/entropy_loss = 0.9440750479698181 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 440029 | episode_rewards = 281.4619657222258 | total_episodes = 4455 | fit/surrogate_loss = -11.234806060791016 | fit/entropy_loss = 0.9728662967681885 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:57: [PPOAgent[worker: 0]] | max_global_step = 441900 | episode_rewards = 303.9872611084485 | total_episodes = 4473 | fit/surrogate_loss = 10.437104225158691 | fit/entropy_loss = 0.9096306562423706 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 443753 | episode_rewards = 315.03343422340197 | total_episodes = 4491 | fit/surrogate_loss = -4.030556678771973 | fit/entropy_loss = 0.9464774131774902 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 445673 | episode_rewards = 0.0 | total_episodes = 4511 | fit/surrogate_loss = -6.509860515594482 | fit/entropy_loss = 0.9715564250946045 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 447532 | episode_rewards = 291.69420461798654 | total_episodes = 4530 | fit/surrogate_loss = 4.072037696838379 | fit/entropy_loss = 0.9845868945121765 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 449527 | episode_rewards = 218.99857107201902 | total_episodes = 4549 | fit/surrogate_loss = -5.0280632972717285 | fit/entropy_loss = 0.8980158567428589 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 451450 | episode_rewards = 323.45601542226564 | total_episodes = 4569 | fit/surrogate_loss = 12.949789047241211 | fit/entropy_loss = 0.9183349609375 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 453436 | episode_rewards = 331.10574235011643 | total_episodes = 4588 | fit/surrogate_loss = -4.701037406921387 | fit/entropy_loss = 0.8513623476028442 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 455331 | episode_rewards = 298.16808547838895 | total_episodes = 4607 | fit/surrogate_loss = 3.3393399715423584 | fit/entropy_loss = 0.9371728301048279 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 457258 | episode_rewards = 305.20445123022546 | total_episodes = 4625 | fit/surrogate_loss = -1.4828935861587524 | fit/entropy_loss = 0.9189344644546509 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 459204 | episode_rewards = 296.97104745527434 | total_episodes = 4646 | fit/surrogate_loss = -6.455825328826904 | fit/entropy_loss = 0.8192564249038696 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 461093 | episode_rewards = 326.2481563297774 | total_episodes = 4665 | fit/surrogate_loss = 8.854265213012695 | fit/entropy_loss = 0.8241569399833679 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 462970 | episode_rewards = 310.69696450029136 | total_episodes = 4683 | fit/surrogate_loss = -1.0951286554336548 | fit/entropy_loss = 0.9279043674468994 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 464888 | episode_rewards = 0.0 | total_episodes = 4702 | fit/surrogate_loss = 7.324929714202881 | fit/entropy_loss = 0.9040438532829285 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 466803 | episode_rewards = 284.0750483620409 | total_episodes = 4720 | fit/surrogate_loss = -10.986502647399902 | fit/entropy_loss = 0.8823921084403992 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 468636 | episode_rewards = -24.0 | total_episodes = 4737 | fit/surrogate_loss = -9.095108985900879 | fit/entropy_loss = 0.9000943303108215 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 470528 | episode_rewards = 0.0 | total_episodes = 4756 | fit/surrogate_loss = 0.49896976351737976 | fit/entropy_loss = 0.9501118063926697 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 472417 | episode_rewards = -36.0 | total_episodes = 4774 | fit/surrogate_loss = 11.732644081115723 | fit/entropy_loss = 0.9509857892990112 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 474413 | episode_rewards = 157.45800940040132 | total_episodes = 4795 | fit/surrogate_loss = -11.044668197631836 | fit/entropy_loss = 0.9405302405357361 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 476233 | episode_rewards = 304.95351220285875 | total_episodes = 4815 | fit/surrogate_loss = 3.230506181716919 | fit/entropy_loss = 0.8895562887191772 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:58: [PPOAgent[worker: 0]] | max_global_step = 478088 | episode_rewards = 273.65990527820844 | total_episodes = 4836 | fit/surrogate_loss = -2.057007074356079 | fit/entropy_loss = 0.8450247049331665 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 480096 | episode_rewards = 225.393869417009 | total_episodes = 4854 | fit/surrogate_loss = -6.749125003814697 | fit/entropy_loss = 0.7920497059822083 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 482010 | episode_rewards = 284.2181490104804 | total_episodes = 4874 | fit/surrogate_loss = 11.958834648132324 | fit/entropy_loss = 0.7524480819702148 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 483903 | episode_rewards = 118.52480329809106 | total_episodes = 4892 | fit/surrogate_loss = 1.5171358585357666 | fit/entropy_loss = 0.8042746186256409 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 485814 | episode_rewards = 287.68503833467213 | total_episodes = 4910 | fit/surrogate_loss = 0.09560095518827438 | fit/entropy_loss = 0.7662050724029541 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 487756 | episode_rewards = 294.12818602515716 | total_episodes = 4928 | fit/surrogate_loss = -5.227457523345947 | fit/entropy_loss = 0.8396679759025574 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 489715 | episode_rewards = 212.56486168613156 | total_episodes = 4948 | fit/surrogate_loss = 1.5038007497787476 | fit/entropy_loss = 0.9010404348373413 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 491574 | episode_rewards = 248.34553919585076 | total_episodes = 4967 | fit/surrogate_loss = 3.363309383392334 | fit/entropy_loss = 0.8638811707496643 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 493465 | episode_rewards = 0.0 | total_episodes = 4986 | fit/surrogate_loss = 8.634424209594727 | fit/entropy_loss = 0.8884937167167664 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 495356 | episode_rewards = 151.93969389847663 | total_episodes = 5005 | fit/surrogate_loss = -1.5574429035186768 | fit/entropy_loss = 0.8366995453834534 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 497239 | episode_rewards = 302.49554099930833 | total_episodes = 5024 | fit/surrogate_loss = 1.633874535560608 | fit/entropy_loss = 0.8804383277893066 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: [PPOAgent[worker: 0]] | max_global_step = 499103 | episode_rewards = 167.51614314661649 | total_episodes = 5044 | fit/surrogate_loss = 1.513819694519043 | fit/entropy_loss = 0.8617693185806274 | \u001b[0m\n", "\u001b[38;21m[INFO] 16:59: ... trained! \u001b[0m\n", "/home/frost/.local/lib/python3.10/site-packages/gym/spaces/box.py:127: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", " logger.warn(f\"Box bound precision lowered by casting to {self.dtype}\")\n", "/home/frost/.local/lib/python3.10/site-packages/gym/spaces/box.py:127: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", " logger.warn(f\"Box bound precision lowered by casting to {self.dtype}\")\n" ] } ], "source": [ "manager = AgentManager(\n", " PPOAgent,\n", " (env_ctor, env_kwargs),\n", " agent_name=\"PPOAgent\",\n", " init_kwargs=dict(\n", " policy_net_fn=model_factory_from_env,\n", " policy_net_kwargs=policy_configs,\n", " value_net_fn=model_factory_from_env,\n", " value_net_kwargs=value_configs,\n", " learning_rate=9e-5,\n", " n_steps=5 * 365,\n", " batch_size=365,\n", " eps_clip=0.2,\n", " ),\n", " fit_budget=5e5,\n", " eval_kwargs=dict(eval_horizon=365),\n", " n_fit=1,\n", " seed = 42, # Important: as farm-gym is very stochastic, for some choice of seed PPO does not train and the final reward is 0 !\n", " output_dir=\"ppo1_results\", # results/trained agents are kept in this directory\n", " )\n", "manager.fit()" ] }, { "cell_type": "markdown", "id": "857b148f-6769-4b7c-8961-5208b3edff2f", "metadata": {}, "source": [ "### Training reward curve" ] }, { "cell_type": "code", "execution_count": 5, "id": "a6770442-c396-4f05-acaf-072a1fda2c90", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEXCAYAAABsyHmSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABdtElEQVR4nO2dd3gc1bm432+lVZclW5J7kWzcjavANNPBhGZCSUwg4BScegmQ0BIS8sslCUlID4EQci9wIWBqKAGCTTfg3nuVu9Vs9a49vz9mZjW7mq3alVbyeZ9Hj3bPzM6c2Z053/nK+T5RSqHRaDQaTTS4eroDGo1Go+m9aCGi0Wg0mqjRQkSj0Wg0UaOFiEaj0WiiRgsRjUaj0USNFiIajUajiRotRDQnHCKyWUTOjfExnxCRB2J5zERBRBaIyNKe7ocmMUnu6Q5oNN2NUmpyT/dBo+kraE1Eo0lARKRHJng9dV5N70ULEU2vRUSGishLIlIuIntF5Faz/aci8qKILBKRWhFZIyLTbJ8rEZELzdenisgqEakRkVIR+Z1tvytN01eViHwgIhNt22aYx60VkUVAml/fLheRdeZnPxWRqWFcT4mI3C0iG4B6EUkWkdPMz1eJyHrLDCci54nIRttnF4vIStv7j0XkKvP1PSKy2+zrFhH5vG2/BSLyiYj8XkQqgZ+KSJ6IvGZ+JyuAMbb9xdy3zNy+UUSmhPyxNH0XpZT+03+97g9jArQa+AmQAowG9gBzgZ8CrcC1gBv4AbAXcJufLQEuNF9/BnzZfJ0FnGa+HgfUAxeZx7gL2GWeKwXYB9xubrvWPN8D5mdnAGXAbCAJuNk8Z2qIayoB1gEjgHRgGFAJXGpe70Xm+wJzexOQb/ahFDgEZJvbGoE887jXAUPNY3zRvK4h5rYFQBvwXxjm7XTgOeB5IBOYYh53qbn/XPN7zwUEmGgdS/+dmH9aE9H0Vk4BCpRSP1NKtSil9gB/B+ab21crpV5USrUCv8PQFE5zOE4rcJKI5Cul6pRSy8z2LwL/VkotNo/xEMYAe4Z5HDfwB6VUq1LqRWCl7ZgLgb8ppZYrpdqVUk8CzQHO78+flFIHlFKNwI3Am0qpN5VSHqXUYmAVcKm5fSVwNjALWA98ApxpnmenUqoSQCn1glLqsHmMRcBO4FTbOQ8rpf6slGoDWoBrgJ8opeqVUpuAJ/2+r2xgAiBKqa1KqSNhXJemj6KFiKa3MgoYapp5qkSkCvghMMjcfsDaUSnlAQ5izMb9+RqG1rFNRFaKyOVm+1AMbcN+jAMY2sFQ4JBSyp69dJ/t9Sjg+359GxHg/P4csL0eBVznd5yzgCHm9g+BczEEyYfAB8A55t+H1kFE5Cabaa0KQ7vID3DOAgyNxN5m/x7eA/4CPAyUichjItIvjOvS9FG0ENH0Vg4Ae5VSuba/bKXUpeb2EdaOIuIChgOH/Q+ilNqplLoeGAj8CnhRRDLNfUfZjiHmMQ8BR4BhZpvFSL++/dyvbxlKqWfDuC67YDoA/J/fcTKVUg+a2/2FyIf4CRERGYWhoX0Xw7yVC2zCMEU5nbMcw7w1wtZmvzaUUn9SSs0CJmEI4DvDuC5NH0ULEU1vZQVQazqi00UkSUSmiMgp5vZZInK1GW10G4Y5aZn/QUTkRhEpMDWNKrPZg+ETuExELhARN/B98xifYvhR2oBbRcQtIlfjax76O/BNEZltOqIzReQyEcmO8BqfBq4Qkbnm9aWJyLkiMtzc/ikw3jz3CqXUZgzBNxv4yNwnE0NIlJvX+xUMTcQRpVQ78DKGgz1DRCZh+HSs7+sU87rcGL6VJvP70pygaCGi6ZWYg93lwHQMp3kF8DiQY+7yKoZf4zjwZeBq07fhzyXAZhGpA/4IzFdKNSqltmP4JP5sHvsK4ArT/9ICXI3hlD5mnudlW99WAbdgmH2OYzjkF0RxjQeAeRhmunIMzeROzOdWKVUPrAE2m30CQ8DtU0qVmftsAX5rtpcCJ2P4ToLxXYwgg6PAE8D/2rb1wxCSxzHMXJXAbyK9Nk3fQXzNuhpN70dEfgqcpJS6saf7otH0dbQmotFoNJqo0atTNZpuQkRGAlsCbJ6klNrfnf3RaGKBNmdpNBqNJmq0OUuj0Wg0UdPnzVn5+fmqsLCwp7uh0Wg0vYrVq1dXKKUKQu3X54VIYWEhq1at6uluaDQaTa9CRPaF3kubszQajUbTBbQQ0Wg0Gk3UaCGi0Wg0mqjp8z4RJ1pbWzl48CBNTU093RVNgpKWlsbw4cNxu9093RWNJqE5IYXIwYMHyc7OprCwEN9ErBqNUaitsrKSgwcPUlRU1NPd0WgSmhPSnNXU1EReXp4WIBpHRIS8vDytqWo0YXBCChFACxBNUPT9odGExwkrRDQajaY309TaTm2TU3WD7kULkT7ET3/6U4YNG8b06dOZMmUKr732WtB2gMcee4wJEyYwYcIETj31VJYuXepzzHXr1iEivP3223Hpc0lJCf/85z/jcmyNpi+zdGcFy/cc6+luaCHS0yil8HhiVxju9ttvZ926dbzwwgt89atf9R7bqf2NN97gb3/7G0uXLmXbtm08+uijfOlLX+Lo0aPe4z377LOcddZZPPtsOJVdI0cLEY2md6OFSA9QUlLC+PHjuemmm5gyZQoHDhzgW9/6FsXFxUyePJn7778fgJUrV3L11VcD8Oqrr5Kenk5LSwtNTU2MHj066DkmTpxIcnIyFRUVAdt/9atf8Zvf/Ib8/HwAZs6cyc0338zDDz8MGALuhRde4IknnmDx4sU+jub//u//Zvz48Zx11llcf/31PPTQQwDs3r2bSy65hFmzZjFnzhy2bdsGwIIFC7j11ls544wzGD16NC+++CIA99xzDx9//DHTp0/n97//fVe/Wo1G082ckCG+dnaU1sbcrpid5mbcoODltHfu3MmTTz7JaaedBsDPf/5zBgwYQHt7OxdccAEbNmxgxowZrFu3DoCPP/6YKVOmsHLlStra2pg9e3bQ4y9fvhyXy0VBQUHA9s2bNzNr1iyf7cXFxTz55JMAfPrppxQVFTFmzBjOPfdc/v3vf3PNNdewcuVKXnrpJdavX09rayszZ870HmfhwoU8+uijjB07luXLl/Ptb3+b9957D4AjR454tZ4rr7ySa6+9lgcffJCHHnqIN954I7wvV6PRJBQnvBDpKUaNGuUVIADPP/88jz32GG1tbRw5coQtW7YwdepUxowZw9atW1mxYgV33HEHH330Ee3t7cyZM8fxuL///e95+umnyc7OZtGiRd4oo0DtwXj22WeZP38+APPnz+epp57immuu4ZNPPmHevHmkpaWRlpbGFVdcAUBdXR2ffvop1113nfcYzc3N3tdXXXUVLpeLSZMmUVpaGvmXptFoEo4TXoiE0hjiRWZmpvf13r17eeihh1i5ciX9+/dnwYIFXtPR2WefzVtvvYXb7ebCCy9kwYIFtLe385vf/MbxuLfffjs/+MEPwmqfNGkSq1ev5vzzz/e2rV69msmTJ9Pe3s5LL73Eq6++ys9//nPvArza2tqA1+TxeMjNzfVqT/6kpqZ6X+tiaBpN30D7RBKAmpoaMjMzycnJobS0lLfeesu7bc6cOfzhD3/g9NNPp6CggMrKSrZv386UKVO6fN677rqLu+++m8rKSsCIxHriiSf49re/zbvvvsvUqVM5cOAAJSUl7Nu3j2uuuYZXXnmFM888k9dff52mpibq6uq8pqh+/fpRVFTECy+8ABiCYv369UH7kJ2dHVQwaTSaxOaE10QSgWnTpjFjxgwmTJjAiBEjOPPMM73bZs+eTWlpKWeffTYAU6dO5ejRozFZDHfllVdy6NAhzjjjDESE7Oxsnn76aYYMGcIPf/hDPv/5z/vsf8011/DII4/w1ltvceWVVzJ16lQGDRrEySefTE5ODgDPPPMM3/rWt3jggQdobW1l/vz5TJs2LWAfpk6dSlJSEtOmTWPBggXcfvvtXb4ujUbTffT5GuvFxcXKvyjV1q1bmThxYg/1qG9QV1dHVlYWDQ0NnH322Tz22GPMnDmzp7sVU/R9oklklmwx/IoXThoUl+OLyGqlVHGo/bQmoomKhQsXsmXLFpqamrj55pv7nADRaDThoYWIJir0AkGNRgPasa7RaDSaLqCFiEaj0WiiRgsRjUaj0USNFiIajUajiRrtWAfW7D9OTWPs8mf1S3czc2T/mB2vK5SUlPDpp5/ypS99CYAnnniCVatW8Ze//CWq433wwQeOua4++OAD5s2bR1FREc3NzcyfP5/7778/YDvA0qVLueOOO6ipqQHgjjvuYOHChd5jtrW1MWTIEL72ta/x4IMPRtXfUPziF7/ghz/8YVyOrdGcCGghAtQ0tpKXmRp6xzCprG8OvVM3YaVat4RIPJkzZw5vvPEG9fX1TJ8+3ZtTy6l96NChfOlLX+Jf//oXM2fOpKKigrlz5zJs2DAuu+wyABYvXsy4ceN44YUX+OUvfxmXaoNaiGg0XUObs3qA+vp6LrvsMqZNm8aUKVNYtGgRAIWFhdx7771Mnz6d4uJi1qxZw9y5cxkzZgyPPvooYKQSufPOO5kyZQonn3yy97OB2p1SrR8+fJhLLrmEsWPHctddd3n79c4773D66aczc+ZMrrvuOurq6gB4++23mTBhAjNnzuTll18OeX2ZmZnMmjWLXbt2BWx/+OGHWbBggXd9SX5+Pr/+9a99NI5nn32W733ve4wcOZLPPvvM2/7mm28yYcIEZs2axa233srll1/u/V6/+tWvcuqppzJjxgxeffVVwNC+rr766k7XfM8999DY2Mj06dO54YYbwvrtNBqNL1qI9ABvv/02Q4cOZf369WzatIlLLrnEu23kyJGsW7eOOXPmsGDBAl588UWWLVvmNQG9/PLLrFu3jvXr17NkyRLuvPNOjhw5ErD9wQcfZM6cOaxbt86bUmTdunUsWrSIjRs3smjRIg4cOEBFRQUPPPAAS5YsYc2aNRQXF/O73/2OpqYmbrnlFl5//XVWr17tU7AqEJWVlSxbtozJkycHbA+Uhn7z5s0ANDU1sWTJEq644gquv/56b1GspqYmvvGNb/DWW2+xevVqysvLvZ//+c9/zvnnn8+KFSt4//33ufPOO6mvrw94zQ8++CDp6emsW7eOZ555JtKfUaPpVmqaWlm97xgeT2JlGdFCpAc4+eSTWbx4MXfffTcff/yxN+8UGPmsrH1mz55NdnY2BQUFpKamUlVVxdKlS7n++utJSkpi0KBBnHPOOaxcuTJguxMXXHABOTk5pKWlMWnSJPbt28eyZcvYsmULZ555JtOnT+fJJ59k3759bNu2jaKiIsaOHYuIcOONNwa8ro8//pgZM2Zw8cUXc88993iFSKD2YLzxxhucd955pKenc8011/Cvf/2L9vZ2tm3bxujRoykqKgLg+uuv937mnXfe4cEHH2T69Omce+65NDU1sX///oDXrNH0JrYfreV4fSu1TW093RUftE+kBxg3bhxr1qzhzTff5L777uOCCy7gJz/5CdCRLt3lcvmkTne5XLS1xebmsR83KSmJtrY2lFJcdNFFncrgBkrr7oTl+win3UpDP2/ePG+blYYeDFPW0qVLKSwsBAwt5r333utUZMuOUoqXXnqJ8ePH+7QvX77c8Zo1Gk3X0ZpID3D48GEyMjK48cYbufPOO1mzZk3Yn50zZw6LFi2ivb2d8vJyPvroI0499dSA7eGmWj/ttNP45JNPvH6M+vp6duzYwYQJEygpKWH37t0AMau1/p3vfIcnnnjCK6QqKyu5++67ueuuu6ipqeHjjz9m//79lJSUUFJSwsMPP8yzzz7L+PHj2bNnDyUlJQBe3w/A3Llz+fOf/+ytVbJ27dqQ/XC73bS2xraypUZzIqE1EYyQ3FhGVPVLdwfdvnHjRu68805cLhdut5tHHnkk7GN//vOf57PPPmPatGmICL/+9a8ZPHhwwPa8vDyfVOv9+zuHHhcUFPDEE09w/fXXe6sRPvDAA4wbN47HHnuMyy67jIyMDObMmROT+h9Dhgzh6aef5pZbbqG2thalFLfddhtXXHEFTz75JOeff76P9jBv3jzuuusuHnnkEf76179yySWXkJmZySmnnOLd58c//jG33XYbU6dOxePxUFRUFLLs7sKFC5k6dSozZ87UfhGNJgp0KnhNr8NKQ6+U4jvf+Q5jx46NSx0SfZ9oEomVJceobmjllMIB5GS4EyYVfI+bs0QkV0ReFJFtIrJVRE4XkQEislhEdpr/+5v7ioj8SUR2icgGEdH5x09A/v73vzN9+nQmT55MdXU13/jGN3q6SxrNCUuPCxHgj8DbSqkJwDRgK3AP8K5Saizwrvke4HPAWPNvIRC+HUjTZ7j99ttZt24dW7Zs4ZlnniEjI6Onu6TRnLD0qBARkRzgbOAfAEqpFqVUFTAPeNLc7UngKvP1POApZbAMyBWRIdGcu6+b8TRdQ98fGk149LQmUgSUA/8rImtF5HERyQQGKaWOmPscBSyj3zDggO3zB802H0RkoYisEpFV9sVoFmlpaVRWVuqBQuOIUorKykrS0tJ6uisaTcLT09FZycBM4L+UUstF5I90mK4AUEopEYlotFdKPQY8BoZj3X/78OHDOXjwIE4CRqMBY6IxfPjwnu6GRpPw9LQQOQgcVEotN9+/iCFESkVkiFLqiGmuKjO3HwJG2D4/3GyLCLfb7V3xrNFoNJro6VFzllLqKHBARKwlxhcAW4DXgJvNtpuBV83XrwE3mVFapwHVNrOXRqPRaLqZntZEAP4LeEZEUoA9wFcwhNvzIvI1YB/wBXPfN4FLgV1Ag7mvRqPRaHqIHhciSql1gNOClgsc9lXAd+LdJ41Go9GER09HZ2k0Go2mF6OFiEaj0WiiRgsRjUaj0USNFiIajUajiRotRDQajUYTNVqIaDQajSZqtBDRaDSaHkQpxZItpRw83tDTXYkKLUQ0Go2mB2nzGOn9dpXV9XBPokMLEY1Go9FEjRYiGo1Go4kaLUQ0Go1GEzVaiGg0Go0marQQ0Wg0Gk3UaCGi0Wg0mqjRQkSj0WgSlMq6ZlbvO45RBSMx6fF6IhqNRqNxZsOhatrbFe2exBUiWhPRaDQaTdRoIaLRaDSaqNFCRKPRaDRRo4WIRqPRaKJGCxGNRqPRRI0WIhqNRtNF2j2Ko9VNPd2NHkGH+Go0Gk0X2X60lsNVjaS5XeRmpPR0d7oVrYloNBpNF2hua6es1tBC2hJ4PUe80EJEo9FousDyPcdoa49eeFTWtQB06Rg9iRYiGo1G0wVa2jxd+mxja3sMe9P9aJ+IRqPRxIjmNg+t7R7cSaHn51UNLawqOU5WWu8ehsPWREQkU0Rc5utxInKliLjj1zWNRqPpXWw9XMPSnRVh7VvT2AZAXVNbPLsUdyIxZ30EpInIMOAd4MvAE/HolEaj0fRWEjlZYjyIRIiIUqoBuBr4q1LqOmByfLql0Wg0mt5AREJERE4HbgD+bbYlxb5LGo1GEx8SuS5HbyUSIXIbcC/wilJqs4iMBt6PS680Go0mxqzdf5x3t5b1dDf6HGELEaXUh0qpK5VSvzLf71FK3RqLTohIkoisFZE3zPdFIrJcRHaJyCIRSTHbU833u8zthbE4v0aj6ftY6zE0sSVkbJmIvA4E1AGVUlfGoB/fA7YC/cz3vwJ+r5R6TkQeBb4GPGL+P66UOklE5pv7fTEG59fECI9HIQIi0tNd0Wj6JCrwcNwjhKOJPAT8FtgLNAJ/N//qgN1d7YCIDAcuAx433wtwPvCiucuTwFXm63nme8ztF4gerRKK97aVsfZAVU93Q9NDVDe2smRLKcfq9aw/GhpaQof7bjlc0w09CZ+QmohS6kMAEfmtUqrYtul1EVkVgz78AbgLyDbf5wFVSinr2zwIDDNfDwMOmP1qE5Fqc//wArM13cIxbTY4YalqMH77yrpmBmSGTkTY0uYhJVknzrD4dFclF04aFHSfhpbEWuEeya+XaTrTAcNvAWR25eQicjlQppRa3ZXjOBx3oYisEpFV5eXlMTtuU2s7S7aUclzPsjSaLnOoqpGPdpRT29Tao/0oq23SmlMXiDQ66wMR+UBEPsSIzPpeF89/JnCliJQAz2GYsf4I5IqIpSUNBw6Zrw8BIwDM7TlApf9BlVKPKaWKlVLFBQUFXexiB8fNWdahqsaYHVOjOVGxNNb65p6dWW84UM2afcfjfp5Pd1Ww9YhhilJKUV7XN+qPhCVEzHQnOcBYDMFxKzBeKfVOV06ulLpXKTVcKVUIzAfeU0rdgCGgrjV3uxl41Xz9mvkec/t7Sgd+9wgbDlax/WgtHo/qUgK6aGhp8+A5wVYFa3oXS7aUdlqT0tDSzqHjxgR0X2UDx+t7VgOLFWEJEaWUB7hLKdWslFpv/jXHsV93A3eIyC4Mn8c/zPZ/AHlm+x3APXHsAwAlFfVeO6+mg7KaZg4ca2DjoWo+2hE7k2E4fLSjnLUHjJljbVMrB483dOv5NYHRU7oOgn0XTW2J5dfoCpGkj1wiIj8AFgH1VqNS6lgsOqKU+gD4wHy9BzjVYZ8m4LpYnC8cWto87CqrAwjp7DpRKa+N51wiMNYsbvke4/Yb3j+jR/qhiZ4TOa6yrKZnnpt4EIkQsdZjfMfWpoDRDvv2CTx6WqXR9DmUUlQ3tvZYGduy2qZuNwHHk7CFiFKqKJ4d6Q0IJ/DUqZeglKKmsY2cDF2lQOPMvsoGdpXVMXNU/7DCkKMl0BS0tZdWMAxERNVQRGQKMAlIs9qUUk/FulMaTbRYA8SsUf3pH8cBQuNLu0exr7K+V2QqOGg6t5vj7JfYV1nP6IKsrh2kF8ibsIWIiNwPnIshRN4EPgcsBbQQ0SQMdc3GGtXmPmQu6A3srainpKKe9JTET+zdFGU52oaWNlwipLk7rrExyMK/lvbo78H65rZeswgzkl5eC1wAHFVKfQWYhhH222fRLpHeS6LlF+rrWIWYrP+RKCQ99VuFa55WStHuUXy6q7JT1cJggSUHj0W/nuyz3ZWs3HsMexcTdTyKRIg0mqG+bSLSDyjDXPin6d1U1DX3aBizk5PR41FsPVLTpxyQiU6oJVdt7R4OB1hoe+CYb5h1og54dhpb26lvDp2rav3Bat7f5pxCfkdpbcTnDXeFfqKlNwlEJEJklYjkYiRfXA2sAT6LR6cShV5g3o0J6/ZXsaqkayt2o13zWV7bzEc7yjulnThS08Sh443eEGtNfCmraeLdrWVec6AT247WsuVwDdWNxiDY1u7pJDx6E7vL6vhsd6eEF52oiHEYe7Qain082niwOka96TqRRGd923z5qIi8DfRTSm2IT7cSG/t4WdvUSnaajgQC2F8Z+YBS3WgIj5rGVp9IGZ2IoHsoq2kiLyuVMnOgrG1qJSvVGBYaW9pp9XjoZ97flp/JMlltPVJLaU2To+DZV9nA2EFGTtW2dg9JLgnodN98qIZ9lQ2cNjrPcbtSipZ2D6nJieNvKa9tZn0Y2apb2z1xqbleWpM4KVPC1kRE5P9E5BYRmaCUKjkRBEigcay0pom2dg9ltU0s33OMo9WJ84PGk02HqlmypdRxm1LRqfaJzN6K+oSa8cWS5rZ2qhtb2XCwmu1HnX+3T3ZVsGJPx1piSwZYPgIr95OVygN8TZMtbR6UUnywvZztIe6Nuqa2gA7vPRX1fLyjIqTW0NTazsqSY52irvZXNsT83jwQZpaEDQerOvlRIiIM+dPS5uH97WXsKqvj/W1lcRFawYjEnPU/wBDgzyKyR0ReEpGuJmDstRj2VONmrWvueg6cuua2bsskuqe8jl1lkT9UPSEsg5kUK+oCmxla21SXH6bdZXUJNeOLlqbWdlptkULH6lv4eEcFR6qNwf9wVSNtft9VoMkCGOPa+oNVeEK4q2qbWtldXuc9R7tHBQ2r9U+CWFHXTGNLu9ecFMh/0dTaTlu7h4PHG6luaOVwVcdv1truYUdpbdha8tr9x2NqorMyK0SycNmexse6h483BB5jjje00N6uKKmop92jeH9bGTXdmBk5EnPW+yLyEXAKcB7wTWAyRtbdE5JYukyWmbOs7kivsqfcyFpz0sDsEHvGn0DPVjjP3Lr9VQG37Sit5XB1o9dEYg2i7qTEDZtUSsVlncXSnRWIwBlj8mlua/c6du22eWug3nyohkHZaY7HsXqmVHg1Y9bafh+PB69zeuLQfgzLTe+0f6OfJhLs97WzdGcFae4kBud09HtXWR0lFfVBPuVMZV0LlXUtNLa2M25QtqMG0+5REQcOfLortO/FYtuRzucMZjqrchAw1Q2tXjNkvInEnPUu8AlG+pPtwClKqQnx6lgi4P88B3q+E9V8X1bbxL7KjgfJ41FsOhTYPNMaIK69rLbJx9Swp7yzs/vg8cichZFqCTVNrSzbUxm0lsuusjpqGjseqLqmjpnrh9vL+XB79Iki1+w/zt4oBiWLstomlmwp9X6Pdc1ttPl93+9uLWPDwaqozxEMpQzz1KqS4yHvV3+txCJWAu54fQut7Z5Omq1SRmXEfZX1Xue9P9UBZuRNre0+QiMaAfL+9o4IrP2VDZTVNjlqMKu7IW18JDhpTtuP1nabWSuSadkGoAWYAkwFpohI5+mEJiJ2ldVRVhsfk8mGA9XsLO0Y8I83tAQ1SX24vdwxxfqGA9WsLOmwjVuajJ3SANdQWdfc6cFfsqWU97eVBV2oZWHZ2zcfqqGuqS3gA2wNIqHCImuajPKtlUFMYU4cq2thd1kd7R5FWRQmLus6ak3Btmx3JescZpfRJOazStLazSANLW0B0+VHuhDTEnyWAK+s71q00tHqpoACeeXeY+wsrTPWSDhtt92HR6obeW9bZ7Pb/ijNUe1+6UgC3Z/GRCU2A/Rnuyt59MMuVxl3ZNme8LWfrhCJOet2ABHJBhYA/wsMBlLj0rMEINSMLRYTs2hmTOHgP8sNF49SuBwMdc2t0R3PMmmcM76gkylp37F6kl3GuXaV1TFiQAZJrs7nDsdZGOyBsUcPWUJo7f4qr+nQ41E0tLZ7o5KCsfVIDUermxiQ1cjMkf1D7m9hdd9+z1hmiBV7j/loUI0t7WGt/C6taWJAZop3wN12pJaqhlYyUpK8gj7V3XmeGMrm76+RNrca0VXWb9CVRXQW0UTy+WPUs+nc3toNa4tiVQvkH5/sjclxnAhnkhYLIjFnfVdEFgFrgXkYjvbPxatjmq5hd8TVNrWG7bQ/WtOEx6OCOq2d8Nc2Pttd6ePcc3KKHjzW6COodzuYySA801dbgKR2mw5Ve/1N0Hm2CbD1aA3LdldyqKqRJVtKg2oa1qzcySdgzfyb29rZdKja2+/qhlbvLF7wDV+ub27zESAAG4OYHC0aW9rZeLC6075Hq5t8NMVohL9/FFRpbVPcJjvR4PGuio/vQq59MRB0dqobWwOucI/XgN8dEZORJGBMA34HrFZKhV7meQKRaC6RI9WNNNkGD6vmxoyRuT77NbS0dXL4tbYr9ph5kGaMzCUvKzpFs765zSc8VClDI1jmN0DZ7e8tbR42HqymtKaJcYNi4/QPZr7bV1nPqLxM76zSmqFvOFjN7NFJPvtZODkx65rbSE128eH2ctLcSQzITOFodRO5GW6G5KT7mGCq/dbDOPmhahoNoZOUJJ2co+0ehUs6tKto80BFwuGqxh5Lm+7EewFWj8eaWGdL+P4L6wF4/KbiTtt+/Z9t3H/F5JieDwyNL1bPUiAiMWc9JCJnAV8G/ldECoAspVT89DFNRHg8CpdL2HyoxnH7Tr/V34FSWFgDUyxTVitgy+HO/bKvMbAP+LvK4z+D2llaR2u7x3EgXm4TgHa/kp2m1nZKa5rYWVrnfVB9AxDqyUjxfcT2lNczqF9HFFGgTAGW2c0yuX26q8Lr78nLSqHS1IS6I6ijrV3FfNW2xpcDEQamJBKRmLPuxyhbe6/Z5AaejkenEoVQ2rJTAreKuua4OcqDsbO0lve2lQWtPW6PVgIoqeisru+2CZpj9S0xU7NLKus7mW2CEWoNQqxw+g7CZenOCq+AqW/p+G6t+6alzdNp7QP4hr6GQ1ltk0/AQKXNlNZddu/eTH1zW8AJU0+hlGJnH1mcG0l01ueBKzFL4yqlDgM9v9AgjvjP8sKxwK7bX8WGA92/ytkKsY1FNUZLIzhc1cgnu7qw2tZGsNDcRMFfyEZCre2zoa41EhNUVUNLj9xPfYkH/r2Vn7y2uVvO5W8C21NRx9efWuXjU2pp8/DK2kP86j/bu6VP8SYSIdKiDI+gAhCRzPh0qXcQ6bqIeNEdNvFYkKhraWKFXcuKZfbVSBJjdkUIhotSyqv9eJTinyv2czRBV/V/tqeSb/9zDeURBolEy3LzfEttE681+6oAeMQWxvvT1zfz5qaj3dKn7iASIfK8iPwNyBWRW4AlGBl9Twj8zQaHohAiu8rqwko9HS5HqhtZurOiR9O49zRtHg8NLTrOY9meSm57fp1PEIA/D761ja8/tcrH5Hm0uonfL9kRdvTem5uO8l/PraWmsZVDxxt5b1sZ9/1rE19/alXCpe1/dsV+nz55PCqouber/H2p4R62R7dZaV4qbd9vmYN/ySm0vbcQlhARI5ZuEfAi8BIwHviJUurPcexbQhHMrBPOLLu5zVgMF8vVrlakULD03X2dRz/Yw63PrevpbvQ4m82ghWAa8i4zhNoeEffH93ay+XANd70UXj5VK5lgQ0t7p8gy/zoZx+pbYjppihR/obbw6dUsfHq1474NLW0cj3IydvdLG7ht0Trve3v6FqdoPidivbo8nsLSn7CEiGnGelMptVgpdadS6gdKqcVx7lufJN4/bbxj57cdrUmorMXrbGlCAkVaJTq7yuoCpvkIF8sX5gpjRvvmxiPe15FqD5ZpKMklVPitlfEouOP5dXz9qVUcrW7irpc28L1F6wL66fZXNvD1p1ZxKE5O70DpW+zXbJkA7/vXJu58cQMvrj7I+jBSzzS3tvPmxiO0exSV9S0+E7n9xxpobfewp7yOtWGki7dYf7CK3y/e0WW/5qp9x1j49Opue04jMWetEZFT4taTPk64pTgTnYfe2cF9r27q1O5RvqaCl9Yc5JEP4pPOIRAP/Hsr3312rU+blcU1UsprmwNWs4uUYLNMpRQPvr2Ne1/Z2KVzWPW8U8JIMPnGxiM0e8O4QwuRuuY26prbfBZJtnsUj328p9O+Neag/Id3d3jbPralQt9TXuf9PlaY62deXH0wZB9iSWNrO6+uO0RJRT23Pb+O/3p2rbffb28+yp/f2xXyGG9uOsrLaw8FtFAcqW7iF29ti6hff35vF5uP1HR5ImT50fYd654FopEIkdnAZyKyW0Q2iMhGEenzNUXs+A8FiVTHu7v74r+6/M4XN/C7JR0Dx1ubjrJ6f3imu7rmti7PxAHHGe0zy/fz6/9sjzil+72vbOSZFfu95pgj1Y1R9dHjUXzrmdW8s8XZkWrNllvaPLQFiWtWSgWdoVqpPtxJzpMV/2N/59m1fPufazoFAVTUNbPTr0zAbYvWcduidT6L/Nod+mJfPW8/7v8t2+c99i/e2sY/V+wHOrSnjYeqg87+PUrx+vrDYZeVDcX/LN3L6xuO8Or6w0Dn7MGhaGxp59+mNvfSGmcBGI4wD0RXw9utn6ahOcHSngBzgTHA+cAVwOXmfwBEJPxEQr2UQAWKIhnA41Gxz37I7qoI+Mu3tvk4tKsbW9kWoLhRKG5btM67mjcSwtEwrGR8oWZ3W4/UOGbQtayDP351M/e+HLm20NzmwaPgjQ1HHLfb07U0BUlR8sraQyz8v9UBBY2libhEKK1p6iTwnI7tZMq65+WN/Opt59DTPfbUJw63mSUcwFnDsUxHJabz336r+qcDaW338MgHu/n6U6vYdqSWV9cf5vbnO98jq/YdY+sR58W1gdhs7h8sv9zBIEWnnl3ZcZ31ASLxuuIov+35dd7vyIn65jbe314W8Fm3CoA9Y/s94knYQkQptc/pz7bLu3HonyYMAtl+E4Fnlu8LvZMDHqV4b1uZ40C35XAN720r49dhxNlbs127OXHN/uM8vtTXFPPbxTv4UwgzRksUSS2tzwQyGwXTPuy8ZYaEBpojWNkFFPCjf23i+y+s9xEkzWHMtv01EPDNamDXhELN3p2yHVjay77KBm59bi2Lt3Zk4E22DbqHqhr58aubvJqsvZCV/8D56Id7+O3iDg1406FqymubWRuGFrw1yKTnp69vYU+FoW03tLT5aEqfhlGXvavsdciUbfHEZyU8s3w/JbbcXrvL67yCJ1jhr3gQywo9fcPoHwGRTPrj6e+2rzIPNDOKBf4PsDWIB4tce397edgDpZ2VJcf454r9fPufa7yO4NKaJl5Ze4jfLdnhM+sNhiWEkm1mnr9+sJtle46FpbV1VbGzhEeyy/dRW7H3GJV1zWFH5Vh7BSrLah3Hfk2WdlfV0ML/e2NLyHM4aSCPL+3IamSvGPjg25HZ+yGwuQt8AwLuf22zj9Pe/h2FCkX+w7s7ufeVjTwcA39cTaOhOT3y4W7+/N6uiELpu2pedtIiPEpR1dDiXZNkf65++dY2Hvj3VmO/bo60jqUQSdzpcC+kLcJII+vGCVSHIRb428EPHm/kfz7Zy/9+WuJtu+ulDZ3MC4Ey7Ppj1w7s9tyX1x4C4OH3d3lt0f7YB89fvLnV+9qaPTv5E6yxKZgwWR7B91nf3MbXn1rl45C3hFhjazv/WLqX1naj7vhjH+/hR//aFJYQsfuffvHmNlY49Mm6Pqc1CO9uK4t6AaRdg+pKFNUzy/cFNOmFwq5pBzMTRav1BuIv7+9ixd5j3iin8rrmsJ/JYJpEtPx7wxF+8OIGb9qbsppmvv7Uqk6p/ccMNNaBzxiRG/M+OJG4tUJ7AdatHclsNdxdV+w95o3JD0RXtZvaplZeWH0g7P2dhIG/an+svqWTs9FpAD9wrKGTD2KZLemh02cOBwlZtAs4y3a/+XC1d0brNFZbs8Vg62z+uWJ/WJpUaU0TS0zzzNLdHb+b3QT22Z5Kthyp8favzaNYH8DPZtHQ0sYv/aJ8/KOiqhpavOtDnlvp+3s+t3K/1xQWCR6l2HioOmYh0++HqCr51GeGAHAqFmYXIskuF5/tqeT/ve6bxmTz4eqQ54iGxz7e4y2r8Ku3t/OLt7aG+ITB320aXFdobm33mqcsTa7K1ESs8GH/ic5gM8HnlGE5MelDKCJJBR+KE86cFU9imTojEJajcnC/NOaMLQi5f7gahf9s0UnIWuYV/7TYu8vrGJSdxrN+g+Gtz/mG7vrjVOTo90t2el//+b2dFOZn8p1zT/K2/fHdnRyrb+Hm0wt9PmevOCj41iD5+lOrWDhnNFuO1DBzZC5Th+fiUYof/asj7NltM1399p0Oez0YObbsIaTBzHLNbe1hJVi888XAQZJLtkYXpvzB9vKwTYaxoqm1nbsdghd8Ur8L/MNhgLb/1vHEbtLrDr5jhqw/flNxp0mjU2XMqoYWPjInn901IEckRMxU8GOVUk6p4C+Iee803YKTCcSfZXsqw6r8B85CRCmFUqEXw/nPui1CCVX/mHx/Ff94QyvH/bLnbj1iOFbtDvqtR2r4y/sdg3ySSzoFLliawNJdFTx+U7HPOgjwtdv7O6CDhak2tnRUV1RK8Z1/rmXKsH4B97eIhx25uwUI0GmNj0Wgcrdd1ZKyUpP7RLYHu3D5oW0y013+hZilgldKxc8Yn6CEkvTVja09khY+Hjy+dC9/eDe82d4Ov/obHqV4ac0hFj7dOUR1UxhV/KIhHEeyE/7p6l0OQsQf/1DjiUMCJ7cOVFscDGf13op6vv/CetaYAm9TgNowGrzrPKKlu8Lho8U/oi9QdgEJsM/mw9WOEXexpkdTwYvICBF5X0S2iMhmEfme2T5ARBaLyE7zf3+zXUTkTyKyy1zwOLMr548Vge7FlXuPedN4e/dJwPu2vLa5U9TPmv3HvbbYrub1UcC720odj7UngcquQufgASF0vfojfr6aT0w/0f0O6cfXBKklUt3Yysc7y6lubPXJ+urEir3HuPW5tXz9qVVB9+tr2IMKupq1OAEfRR++9cwan/eBCldVBohYW7O/iv9sLnXcFksiMWe1KKWUiMQyFXwb8H2l1BoRyQZWi8hiYAHwrlLqQRG5B7gHQwv6HDDW/JsNPGL+13SBVfuOs2rfcR6/qZi7XtxAc1s79S3tnD02n7PHFvDAm+E5EwNhmbLAKMj09uYOR293Zy8NlZjOf3Nzmydo1blX1h5yNLeUVNRHFc20JcyFc04pR04E7Ga2cGbZw3LTA/4O3eF37A6C1YJP7obnq0dTwSuljiil1piva4GtwDBgHvCkuduTwFXm63nAU8pgmdmXIV3pQ28mkgypO0pr+cQWNeSkyq/Zf5xjDS3etSYf7azosgABY2C2TEKPL93rk2k2Kc4JI/154rOSoNudhIzdR+JPoJDj2iht7f5JDQMx2FZi90Ql3O+qO5kxMjcux3XKpmARrJ5LcoA0OLEk0hrrFwE1dKSCj1kmXxEpBGYAy4FBSinr6TwKDDJfDwPsYTsHzTafJ1lEFgILAUaOHBmrLjr0Obz92j2KI9Wxz1R6vD78XEKW8/jMMflAR2oEO3+NU8LEYLbnbpYhIVcbO+WEioZQJrCukqiFoBKN7s4pNzQnnbVUxfy4obIpBMJ/kWs8iCg6yxQaMU8BLyJZGHVKblNK1djTmdtNaOGilHoMeAyguLg4bneRFT8e6kbdWVbLwWPdXwnxnS1HGTcw2yf54KMf7uab54zxMSnFG6vWhRMvdHMG11DEqg5DLFZMa2JANzs+untSFIruMBeHFCIiUkuQn0IpFToGMfjx3RgC5Bml1Mtmc6mIDFFKHTHNVVag+CFghO3jw822HqHCDI2tamilpc1DSrKz1G8OkFgvnDTcXeH5VcYAPW5Qlrdt1b7jrD9Y1a1RP08ti+1K4njivz6lr3Pz6aN48rPe8/tESjjh67EkwWRIt/QnpK6jlMo2BcUfMRzcwzAG77uBP3Tl5GbFxH8AW5VSv7Nteg242Xx9M/Cqrf0mM0rrNKDaZvbqMRpb2vl0d/DV5U4Ec4h1FXuon3++oXDqJWhODIbkpPd0F+JKIicn7Q4+2BH7Vfz+RGIwu1Ip9VelVK1SqkYp9QiGo7srnAl8GThfRNaZf5cCDwIXichO4ELzPcCbwB5gF4ZT/9tdPH/MCHc1ty/xu8Hti9wS0QGpSQwidbyOysuIU09ODHLS3T3dhZgTiU+kXkRuAJ7DGP2ux1wzEi1KqaUE1rg6rYA3y/R+pyvnjCfVYdZTPni8gfys1C5niA1GV0tsak4MIg0B7e5out5GqPLUgYqG9WYi0US+BHwBKMXwUVxntmlMNh0Ovfq63aPYdqSWpTsr4ppyIVYOYk3fxp3k4itnFjJrZHg15VLdvSdn6xeLR4TeKQY8ekPHmudQMjY1OcnnfUF2ajy61K1EUpSqRCk1TymVb/5dpZQqiWPf+gyBhvPKOJqZtAzRhOKMMXkUZKVy5ph8vnXumLA+c8nkwTHtw/xT4jPQ/3n+DC6aNMin7etnFcXs+FOHd2TITbaVwg2lZ8yd3NGnBacX8svPnxyzPvUUkeTOGi4ir4hImfn3kogMj2fnNNFzIpqzpg/P7dR2/viB3d+RXsJXzywKmRAT4PKTO9bzZqTEMvE3XDhxUOidoiA9xXfGf/Ppo5gwuHOWpmh9FDefXsi54wqYGcbiwots12gl/YTECweOlkh00//FiI4aav69brZpEpATT4RAm+ocMn3u+NAp7mNBQVbXzBLnxbCfToNlujspYAh6KOyfi6VPZJo5m59dNCBmxwyEwtlfYS8hfNfc8RSGGTiQk+7mxtNG8W1baQECnCPNZgK0J/g8EYVIgVLqf5VSbebfE0D3PKGaiImnT6R/RmJGmKQkdb6dXd3wpA7LTedHl06M+3nC5fQxeXzlzEKftpMGZnHWSUa2gpEDMrj/iklhH8/+FQZavHbOuPCHgosmDuKRG2Z6a7tcPjW8zEVpXfHHKAildI0blM3nphh96ZcWWOO6L8hvHepu89imd6Gc8L2FSH6VShG5UUSSzL8bgfhXrO8D9ETK6XBP+a1zwrOF20nU2Ht/pyXAgMyUuJ83ySVBZ5W5IUwmN5w6ktNH54U8zxUhBtuRAzK4ZU4RZ4zO86a3sRDpiAw6Z1wBI/qHH6prF8QihmnIn8zUJDJTOn//luCy88VTRuBOcnlNaU7C34mifOecr/4RZnMczhnqjh1oOrit7BOBBvjHbyqmMEA/wsH+XHZHcsSrpg+N+zkiESJfxYjOOmr+XQt8JR6d6gv4L/DrbsL1iUwZGnnCgejWxMSHy2z2+osnd7avR2vCiYRklwTVeEL9FtNG5DI4J3RCxUkhfqsklzC7KM9xABydn8kVU4dy+clDOPMkZ4H1l+tnOLbbD6cUzBlbwEPXTvWZkbtEvIk77Ywc4CusHrp2atBrsGP/bQFcAeb5I8xzfPPs0cydPIgv24ScJZyDTeR+/4Vp3H+5oZmNG5hNskv4xtmjO+13zcxhIfscSrmw3wszw4yI6wrjHUybsSaS6Kx9SqkrlVIF5t9VSqnuL3/WSzgcRRrwWBKuEAnHsepPl8wKAbhq+lCKR4V+qO67dCKP2EIqzx7bMesc0T+DsQOznD4WV0JqIhnBtSGXhKc5hsqDFGzrpScPIc2dxFUzhgVMypfmTnL07dgF5JBcQ9jlZqRQmJ/p/c2czp3mdvkM3vddOjHkd2FnRP8OM+GFEweGHKAHZKZw3awRPv21JhGKwBGL2WluUt2GFtUv3c2jN85i3KBsHy39vPEFXlNXMEKZT+2/c6zzWo0pyGTeNF/NozvMuZFEZ/1aRPqJiFtE3hWRctOkpTFJJAtnuBancG6yH1w8zuf9pCHGjDgaLSYQl08dGnA19H2Xdcx4+2em4LaZP/wfxNsuHBuzPoVLqMFgSAgtwyUSlhApysvk2pnRBUSGO5g4+UqW2wpB+R8nN4h/rKnVw4QhHfeIJYD8CXTp00fmUpSfyX2XTeQLs0Y4alhhm4RU5GblabYwXgnz6Q7kG/rKGYXMOSnfURDeceE4vnJGYUR9c2J4/wyumBZ/85U/kUwpL1ZK1QCXAyXAScCd8eiUpuuErYmEeDbmTR/KhMG+wiLftB93hzoOUJjXYYP276//oObkFwmGk/08FBl+tv8LJw7yChIngTHXXFthmXbcScLDNtORyyVhpSwXES6ZEv46jatnhDa/+GM3/402bf/BtOolW43cqK9vcE5hNyy3IzdXoN8m0K1qaUyFeZm4/LS9X1/TYRYb0d84R0ZqZ2f4KPM7H9gvtVPYbyiSk1wdCxZDPCcL54xm5shc0tzO5zjzpHxuPqOQG2cbpjb7bzNpaD/OjOI+9MfpWe4Od2wkQsT6hS4DXlBKxac4dh+kK7/jrrI6NoexEt6fcIWIiHTSNHy2O7RdMnkwXywe4eg0tTM0DDu/nYNBKgha+AuNaMxxdk4Lw6Htj/9XO6J/Ou4kF/dfPolvOyzaGzkgg8dvKub7Fxnfs0vEaz4BI2w2FrEK/rPcS0+OvF6b/RB3zh3PH744PSyzi/3c2UEim8LlljmdFwbaf/vM1CTvea8/dSR3zx3vWKhrzth8fnrFJCYPzSE1OYkxBYZgtI4Ujp/DOHfw7acWDegU7uvE0Nx0/jx/Bp+LYDIQLl1x+HeFSITIGyKyDZgFvCsiBYCujBNjDlc10tDSkQ6lpKKeI1UdX3NjgJKeq/cd55b/W0WzmXgxkhmIv6Zhx3pwL5zYsWjPneTiokmDQg7gp4+JbIC2m00C4T+gOa1bsKrLhRN2GguztNWnEQMygmpCgUxKhk8kMYIV7CYjd5KLrNTksExhdqe3v0M8FE5amNM57VFulnkpO9WNO8nF2EHODmQRYbgtEs36mq1Ir1ACMh5FrdJTkhxNc4ECG5zwD7P/8WUTOcOcEH35tI7Agu6IIo7EsX4PcAZQrJRqxUi+2NUsvic8b2866qNp7C6v49NdnSOnNxysoqKumU92VXDAoab3K+sOoRRUmlFhsVqxHuomfPDqk8lISeLm00f5CBqAMQW+Tu6vnVXEnRePj2l/nMaAk8zzhhOZFW6svt0X4T+whOtv6Agf9W13uYQsv9n7AD8H9E8uc17XcdX0od7Zdaw4fXQet9t8S2Fdnm2fSJ25Treq0xHm2cJVU5JdfPm0Udw5t4v3Uwg7ldW3aNZ0hBMoYieQKcz52L4LNEflZXr7eM64An5z7VQumTw4YFh0LAn5lInI+eb/q4FzgXnm60swhMoJR1VDC79dvD0mCRRfXHOQ3y/Z6X1vrwNip6ymmU2Hqs3zd84WbM1kXSJUN7byRgAbdaSEGhDys1L50/wZzBlbwPxTRvo4x/0/OXFwtmPI4e+um+Z47C8UD+fKENEmTtqQ9eD739yz/B7qM8bkhT1Ts89Y/Qe9cE1q1nqIi/xSfbhEOkVMnVLk29eRAYIOLp86lGtMAReu8zcUXzuriMlDO5zK4QgFoWN2HG0vUpNd3gJqToO22289yTnjCiJOYKj8/oe6NK8QiegsBqECKqLhlELjvnD5CO3O+/XPSOHaWcO7JTorHOPlOcB7wBUO2xTwskN7n+btzUfZeqSWT3dXcPGk6G2bR6sjswba12fYV6S/tOYgpTVGBbcNh6q8FQ1jQaT34B0XjuN7i9aZH/bdFsh00M9hMd754wc6frf+5isnc5ZXU7BtykhJ8oZsLt1ZwROflSB0XN/QnDQ8GCuVDxxr9KnHAnTSFAL1wV/7sdu+k5NcPH5TcafP+38tf5o/nTR3Ev/ZXBrwnE44/VY/vmxil/NdBTO1ZaUmU9fcximFA9hTURewH8GPb/zPTXdTlJfJjtI6R79KLMZD61qs/6EOGUh7DIdYr0j/5tmjKa1tZmXJcZ9jP3St8ySsuwh5dyml7jf/64WFJt6Zbhdvks/2RL/gf5/NpPXWpo566bEUIBD5NWamJjNhcDbbjtZ2mhlH8lB9afZIx3b/Q4RzyD98YbpPHQe7Ocr6LcvrmnnkhlmA4ZfacLCaF9cY3+VXzihkdtEA/rF0r/l5X+xKRJZfhNDnw4iQsr6Xn105maa29ogH/WCWy1F5XTdnBDOM9kszhMjcyYN45MM6n/3HB/BT+GOFCV88eTBnjslj3OBsxjl8Nhazav9rCXVPdmgikZ/b+kQ0hmVx+Ny4QdmO5X6dJmHdSSTrRPJE5E8iskZEVovIH0Uk8tCWXsYv3tzKE5+W+LRFquIGesidnHbhPCcKxe6yOto9iseX7gmzF9FhdSeS0NmvnlnEJZMHM9rPVm/N2LuSTtz/oXcaBCYPMUwxVlbfrLRkn2io6SNyyctMYe7kwewqMwa+VpuWNzQ33SeU9syT8oMOYMGSEgb73Gg/e/XQ3HRG54deLOmfysVa7R4qWi5agtnqLc3L5zq9foTwj//4TcWcM66A5CQX0xyyMUdyvGB4tR7T5xRuJFlU5+6CFLE+cufF47l6xjAev6m4x4VFICKZ8jwHfARcY76/AViEUb62z7Knop49FfUssC0GspzWIkJjS3vY8edbj9QwNDednHQ333h6Ne0OcZ2R+MP/s/koy/aEjmjqCtYgfdnJQzh4vIF500LPrAdkGvZYMGbXP3ltM9Bhtrl21nDmTh7E7c+vj0ufR+ZlOJqNLLLT3PzKXGfgb7YKxs+unExpTRN//3ivT3u0Yca3XziOqsbQ6XH8VyH/6uqTafMomsy+56S7g15vV/nCrOE8/MFux2wA3zpnDJ/srmRIThqzi/J4bf1hMk1tLNbW+FhoIrOLBrD/WAM3zB7JqYUDQjq/PWGavZywPuPpQoTXmIGZAVOXXD51CFOH5Thu604iESJDlFL/bXv/gIh8MdYdSmTKa5u595WN5GcZs5jd5XX8c8V+bj3/JKYGmD3Z+e3iHeRlpvCra6Y6CpBwsWbNTqptrLFW7aYku/iv8yNfDT7UttjMPthmp3WeVeWku31Sc3cHVtioU2biOy8eT2t7R6DD0Nx0huamk5m6n5aGjvZoB7f0lCTSU9IDbj9//EDe217WyaEsIriTpFN7vLA0ESefVl5Wqjf44fKpQ7h40iCSk4TTD+V1CoroKoKhbYWTrDIQF08axHnjB5KS7OLUCFLQR2KKHZWXwb5KWwRlFwIlA+ULA7hqeuSLSeNBJHfhOyIyX0Rc5t8XgP/Eq2OJyIHjxo1RYVYk3G/6JTYcDL4Y8LgtGWNliMSM4Wgi1vGW7qoIua+9VsN3z/NdDOWUPdW+wvjxm4rJ62KdDDuhBlsr8qQ7sRzmTgvzxg/OZorDTO+uuRO4IYDPBvBGGHUVy48Tj7UKkRDubNwlQpo7iWSXi6+dVRTz0q8iwoIzCruUVFBEIkrKGU101p0Xj+fBq0/2Cp5Ifr1b5hRx2wVjGW6uwk+oXEoBiEQTuQX4HvB/5vskoF5EvgEopVTsEiklILvK6jqlu0g3Z2iRmEQCUV7bHJd6y5bGc9roAUwfkcv9V0wiJcnFj/61yWdw+sllk9hRVsuGg9UcilPyyFBWn8wIHMqfmzKYD7aXd7FHhq8nUlNQQXYq540fyDPLnfOPWj6SS0/u4qrkBBlALOGfGsE6Bjt5mSkhJ0+JivcJieC3SHMnkeZO6nCJRCBFZhcZWtaovAwOVTV2mnhZY5D/WNSTRCJEcjD8IEVKqZ+JyEgME9fy+HQtsXjw7W2ditFEIkRCrUhetOoAN84eGVGW03CYO3kwq/Yd55RCQyMZ0T/Da6IZktOhdYzMy2BkXganFA7g+y/Ex1cRyiRgaQX2zLwW9102kXrbupxrZg73ro9IOMzLDDc6KfBhzJlsDy9mHz84myumDuH8CdGVGv7556f02lKb4YYCOzF5aD9eXnvIJ5FjuGSnuZkwuLOJdc7YAto9KqIiYPEmEiHyMOABzgd+BtQCLwGnxKFfvQJLLW4NsEDQTqiBYN2BKtYdqIq5g7QoP7PTMd1JLm67YKxj1txoa05Hg/8KdwsnYVMYg1DVWFOQlUp5XWe/1BeLR/D0sv3elfO9HZcI87pgfw+Uer430LEoMXIxMiqv87PXVZJcwgVxqksfLZEIkdlKqZkishZAKXVcROJfNi6B6BRlYd5X4fjII43Q2H+sgbKaJooL41N/2snWHy9OLRzAipLOUWTzT/H1K3So/5F9V18/q4iB/WJvCgzFjy6bSG1T50CA4f0zuOdzE7p8fGvc6qWT+D6BFZHWVyYE8SASIdIqIkmY97SZgDH0FLwv4f80m+/DyVMVqUniZ29sAeDxOAmR7uTrZxV1qvntxESz9kSkmXWjycQbC7JSkzstLowl0QpVTeyYPDSHP82f3uVV/32ZSL6ZPwGvAANF5OcY5XHvi0uvEhT/R9lyFoYK121p8/Dp7tCRVI7nVMpRle5NA4vLJUFDFS0G9UuL63qHXkeCONZPdLQACU7Y345S6hkRWQ1cgHF7X6WU2hq3niUg/uO2FeK7p6Le21Ze28zxhhafmtkvrz3oLd4T+hy+QqO6sZU2jyI/K5U2j4fHP97LFVOHsmpffBcZanoer2O9h/uh0QQjIhGrlNoGbItTXxKecGb/976yEYCzbdETtU2+2X49QTSXlnaPT4qRH7y4ATAc3i1tHhpb28NeZBjtrP6+yybSz2ExoKZ7OX10Hm9uPMKpfcCkqem7aD0tAqKdEfqv9F349OqA+248WO3oTLev5N7vUE/En3u74NhNxEioE5HBOWn8XZv3NAlO74296wEicUPY/SSRpKd4duUBb06kSPjKmYWkuY3zLJwzulNBKI1Go4kHWhOJgEjST5TVNtHuUbS1q7BqVFtUN7by3WfXRtSvLxQP58wx+cwa2Z8lW0s7FV/SaDSaeKGFSAREkjTx450VrNl3nE2HazplYY01VvGmNHcSl0+N77k0Go3GjjZnRYC/DLGy+Vo0tHSYoT7YXs6mwzVAfPPc3HfZxNA7aTQaTZzolUJERC4Rke0isktE7umu8/prIh7bUkv/yC37vu1xXNOhneAajaYn6XVCxFw1/zDwOWAScL2ITIrHucpqfWugby+t9XlvFw7+cqLN4/GmrYhFyVr7uhOLuZMSK4eORqM58eh1QgQ4FdillNqjlGrBqLg4Lx4nqvNb37F4S6nPe3vYrX/qk7Z2xagBnRMcRsMphf355tmjve+/dOpIbjh1JNcVj4jJ8TUajSZaeqMQGQYcsL0/aLZ5EZGFIrJKRFaVl0dfc+JgVej1GBb1Le0+gsSjFLnp4eWnPH+8kc1WxKgOlyTCN88Zzd9unMXjNxXzjbPHeOtqXzdrOOdPGMh5UablPtGYMy4+dcftuFzQP7NruUiH5HbWNMPF0lIH9TP+p7pdJCUFjgi09guXXFvVx6ICw3wabWLeSGrmDOsfuOpjvIn0O0pUznIoqxBr+mR0llLqMeAxgOLi4m7JGvH9F9Zz6ZSOIkTHG1rJy2wL8okOMlMNx/vlJw9h3vRhXDltqEMxmmQevWEmyWGuOcnNcFPVELzU7CmFA8jJcLPET8OyOGd8AR+GWfgpOUnol+7mWF1H8aELJw3ik10VNLZ0rWjXhZMGBexjKOyr/4MxJDeNI1VNAbePHZTFztI673t3sstbAuD8CR1mxWD9nDEyl7X7qzq1Z6clM3lojs/509xJTBuRw/I9HeltLpw0iKU7K3zWEV1omjStrMwn45ud2ak/hfkZlNY0eT8frM8F2akM6pdGVUM1g3PSGFOQ5bgGyTrG0Nx0Dlc1MmFINkrB9qOGCTgjNYnU5CSmDs/plAtuxd5j1DS2ctqYPLYfreV4fQuThvZjaG46E4f0o6m1naU7jdxzg3PSOFrd0Xcn1uw/7r0PR+ZlsN8sVRvOM2Fx8vAcBtWmsuFAR9XS/OxUpo/IZVdZLSUVzhNM6/rtXDhpELvK6iixpUfy73+g36B/ppvj9R19tkrvnjQwi8L8TO/nAn0X3UFv1EQOAXY7znCzLeZEKn0+9itXu6u8LsCeBiKw4PTCTjULApWRtQuQUDOMGSM71orMHNWfs8bmk2+bBV44aRA5DnXF7X1zJ7mYPTq8lBsiwskO6eVnjMzt1HbSwO5ZCFmYH37QQajYh65qGoX5mfQLUKtl2ohcx3anDMGnjR7A1CiKHHUX9lt3xIAMbynbvMxUZo3q75hMdOrwHArzM8lKTWby0H4U5mcyxOYDtP824dS7sX+2KMj33lUK8zubq8O9t0eEaeqePDTH8T62f41O/tLupDcKkZXAWBEpMuuZzAdei8eJIl05rhQUOhR6CsR1s4Zz1th8bwhwpkMocP/MFKY7DMSpIepEJ7mEvKwURuZlMCAzhTR3UsjytHYmDTXSsmenuR0XS/qPBYJz0lmnDKihFl8OHxAbM0a4izxdLmPG6s/ogthFvvkPLlYVRzC0DiecBFtykiuiGuEWp4Y5GegJ0txJ3u/Hem0XNvZFvulhhMvbK3a6k1wxq3lv9ci6p53yy/n/NoHMiuH+hvbvxokLJw3q1tpATvQ6c5ZSqk1Evgv8B6PO+/8opTbH41yeCK0wdc1tJEcwUltmgfMnDMSd5OLssZ1LXopAflZnO3I4ldbs2kgkDB+Q7vMghkMUhd8YPiCdY3UtPutrwAhbLsrP5OMd0aXP9ycl2UWLrfpkqttFc2vHe7s5yk6wdDXJLiE8wwje2bidWaP6BzUVuoP4NKKhX5qbzNRknxLDA7ICa1dJLqEgO5WTBmZxvCH8+uijCzJp9yjv/WMNtP2DaL2hENv0xHoWgvl84s3Q3HQyU5JNTb466L5njIltrZtEzOjcGzURlFJvKqXGKaXGKKV+HrfzBNn2DVu0lJ2qxvCGlunDc71CJNnl4rzxA31mzpbpKdOc9YQzA+tJBPF+Xy4XQR3/lhaVlpzEaaPzHM0N/r6McMxJgRZ12h/kJJcEXFuTm+Gmf6bbKxBd5u/hby44aWBWQBOUP4X5GZ1MF0lJElRAjR+cHfT4loaSG8bAnJwkjprdueMLmD488DlyMtxMGZYTUEsKRGpyElOG5Xjv5ZwMN+eML2BgFxzV6SlJjB+c7TXhnjdhIOc4TLi6ir8zPcX8jazvwD5R8jcFn1I4wPE3i2aJWFdNp91Nr9NEEoXpQR7yYbnpHPJzrvlT4FDONdd8cFvaPfRLc1NR18yADOOGKi7sT1OLh5UOZWbtM8xB/dICRsCM6J9BWU3nNPLFhf3ZWVZHdZhORydcrg5fzsDstKCmpIH90pg2QsjPSkFECCdW4KSBWazcG7yGiuEz6qw+2rW21GQXg/qlcbiqkbysVJrbOva3sidX1DVz4FhDwJpQwXwtxYX9WVVy3NbvzlpIKCyhYy8ZkO/wm4aj/Z073lmYBwvQmDmqP/3SYjc0RJKANBB2QRyOmfL0MXlhfT+Th/VjUHYaxxpayM9KJfeYmzrzWcrNMEzJ7iQXK/cec4y2nDysH5V1LUH9i3aG9U+nprHVx2/jz/D+6Ryv99X+Zo8egDvJ5c3gLQlUsUwLkSA41Q+ZOCSbuZMGB30wggmQ88cPZMKQbEcndF5WKmnuJO/Mx27GSk1O6jQ7nzmqP5lm1IsVpXFyEKdrqtu5z7kZKRGZ4ZyYMaI/SS7hrLH53hmcxcnDjZmpPQQ6nFDPNHeSdyDISXdz5kn5bDhY1ak+i0W4JrWUZBezg5TUzc9KJT8r1bvY1O5/ygoxuEY6cw+Xqbb7JVqTxvD+6Ww/Wtupj5mpybS0e7zRZgOimAmfOnpAlyYhsSYzQNniNHeSj6/TowyN03rW/LVGq/3Mk/IdrQFDcjqbfk8pGkBpTRPuJFen7zrNneQtA23HHgk5qF8aBzIafCLJshO4vo8WIhGSneqO2JGVl5mCS4RhuelcMmVwVA+phT1MsSvHCcaw3PD9ITNG5pLmTvI+YE6DaLQx9/4RaOkpSQF9QSnJLvqlualuaKWoIJO95fVe85b9E2MHha8ZDMxOY9JQxeB+adS1dBZcowsyI/IXRItIh2kNjJDg5CRhdH5kDuMRAzqb1sCYtYMRGhttMbJ+ae6ELWSWlWp+XwVZZKQkUdvURlNruyFQw3RwR2JOzkl3d4oiG5qbxv5j9QwO8Cy4k1zMGJnrddo71RSC6Mxj8UYLkSA4/V51zeGt/bCYf8oILpwYuxju6SNyabY5iSMhnFrR00bkRjTrGZCZEpaTP1Ykmw7VmWa6+9KaJkYOyMCd5CLZJQzJTaNfmpv8rFTvg+xyCaeNySPdnRRRWn4wnKjQYaqza6CjI6zZYp051TxG/8yUTmaLcHAnuQKaqbrCzCgDMRKdZL/vy5ro9HMY7ONFRkpywAAOizyHABp/+me6OXAsvFDn7kILkSA4Sf02T3gD+A8uHkdbu4p5+F1ykivsBYdOnN/Fle4zR/XHJbB63/HQO8eBKUNzKK1p8mph/tqYNRv2f8ic1lxEQlZqMuMHZzPQwZcVLslJLiYO7Uee2eeZI3M7ZYbWdB+JNBCHy8DsNM4dn9KlMSDWaCESIeGqkxMGd7Z7JgKuADPxsYOyaffUhDSRxcuEBoZ5qKqxNegamJRkV9gLtWJNOOdNTXYxICvFZ+W+HbupUEQIFqnqcgn52amM6MH0H5CYJpQTmUQSINBLQ3y7CyfHun+ixWgpyE6NKI9QvMlKTaa4cEBQc8+EIZFHGkVC/4wUZo50XtXcWxCRqM1CGalJnWzv00fkhmXm0Gh6Ci1EIiRWs7LcDHfYaw16GmvFcKQLEMNldH4W7mQX2TEMK+1pwg35tHPGmHzOPCn+CfM0mliihUgQLHlxw+yR3janAlO3XTCWq2cM69QejNyM2JuFunOR0tAIIrhC0T8zhXPGFSScmt4VZozI9UY99RV6sYKoiSN9Z+oXB6zMoXvKO7JvOmkiU4blMGVYDi+vNfJAnjMu+Grac8fHfsA8b8LAiHJjRcLgfkZmUvvhJwzOZtyg7F5teoonXQ2A0Gh6C1qIBGGvmbrZSpsNeFeMAtx0+ij2VXZOCX351CFBjxuPwSXS0NVImDgkm7GDsnyc8qGcwpq+Q0F2Kv0z3RGvS9GcGGghEgxzkAzkBjl7bAGM7Xj/62umsu5AFf2DmKpiaQbqLkQk5gkBNb0Hd5KLWaMSNwuwpmfR+nYQrGHTHqU12SFlgcWAzJSQ6zCiSeOt0Wg0iYoe0YLhMPmOV4EbjUaj6Y1oIRIEK1Om3Zz1pVNHOu+s0Wg0JyDaJxIEq/Z5fmYqQ3LSGDkgI+Hremg0Gk13ooVIEMYUZLFszzG+UDxcrxrWaDQaB7Q5KwhWUSD/Oh4ajUajMdBCJAjW6nSX/pY0Go3GET08BsHK+p6kV2VrNBqNI1qIBMHj1US0ENFoNBontBAJwrajtYDWRDQajSYQWogEYcuRGkBnL9VoNJpAaCESBvHMVDtxaD9vHQ2nIlgajUaTyGgh0k2kuZ3DhIflpus1KBqNpteihUg3kJGSxNDctJ7uhkaj0cQcLUQCsKO0NuC2gf0i0xxmj+5bFe40Go3GQguRALiC+EFyIszkG6pglHUqXSVQo9H0NnTurAAUZMfWT+E2qxmmOFQ1HDUgg7Z2xcgBGTE9p0aj0cQbLUQCkJmSxOM3FTtuiyaIanj/dEQMR7o/yUkuxg/OjvygGo1G08NoIRKAQHXQM1KTGN4/nYaWdpJcwoFjnWuszxmXj1JQ39xGZqrxFYsIw/trTUOj0fQttBCJgAsnDfK+njTUKJN7uLqR9nZf1cTK+hsorFej0Wj6Cj3mWBeR34jINhHZICKviEiubdu9IrJLRLaLyFxb+yVm2y4Ruac7+3vy8Jyg24c6mKk0Go2mr9OT0VmLgSlKqanADuBeABGZBMwHJgOXAH8VkSQRSQIeBj4HTAKuN/ftFgb1C77OY+ygLMYPzuaUwgHd1CONRqPpeXpMiCil3lFKtZlvlwHDzdfzgOeUUs1Kqb3ALuBU82+XUmqPUqoFeM7cNyEQYMSADHIyIgv/1Wg0mt5MoqwT+Srwlvl6GHDAtu2g2RaovRMislBEVonIqvLy8jh0V6PRaDQQZ8e6iCwBBjts+pFS6lVznx8BbcAzsTqvUuox4DGA4uJindVQo9Fo4kRchYhS6sJg20VkAXA5cIHqSGF7CBhh22242UaQ9rgye7T2c2g0Go0TPRmddQlwF3ClUsq+2OI1YL6IpIpIETAWWAGsBMaKSJGIpGA431/rjr5mp2k/h0aj0TjRk+tE/gKkAovNnFHLlFLfVEptFpHngS0YZq7vKKXaAUTku8B/gCTgf5RSm3um6x3kpruprGvRea80Gs0JifT1QkjFxcVq1apVUX22rKaJplYPI/MCrzRv9ygaWtq0tqLRaPoUIrJaKeWc+8mGXrEehIEh1oaAkaFXCxCNRnOikighvhqNRqPphWghotFoNJqo0UJEo9FoNFGjhYhGo9FookYLEY1Go9FEjRYiGo1Go4kaLUQ0Go1GEzVaiGg0Go0mavr8inURKQf2deEQ+UBFjLrTGzjRrhf0NZ8o6GuOjFFKqYJQO/V5IdJVRGRVOEv/+won2vWCvuYTBX3N8UGbszQajUYTNVqIaDQajSZqtBAJzWM93YFu5kS7XtDXfKKgrzkOaJ+IRqPRaKJGayIajUajiRotRDQajUYTNVqIBEBELhGR7SKyS0Tu6en+hIOI/I+IlInIJlvbABFZLCI7zf/9zXYRkT+Z17dBRGbaPnOzuf9OEbnZ1j5LRDaan/mTmDWBA52jG653hIi8LyJbRGSziHzvBLjmNBFZISLrzWv+f2Z7kYgsN/u5SERSzPZU8/0uc3uh7Vj3mu3bRWSurd3x3g90ju5CRJJEZK2IvBGsP33lmkWkxLz31onIKrMt8e5tpZT+8/vDqOG+GxgNpADrgUk93a8w+n02MBPYZGv7NXCP+foe4Ffm60uBtwABTgOWm+0DgD3m//7m6/7mthXmvmJ+9nPBztEN1zsEmGm+zgZ2AJP6+DULkGW+dgPLzf49D8w32x8FvmW+/jbwqPl6PrDIfD3JvK9TgSLzfk8Kdu8HOkc33t93AP8E3gjWn75yzUAJkO/XlnD3drfdAL3pDzgd+I/t/b3AvT3drzD7XoivENkODDFfDwG2m6//Blzvvx9wPfA3W/vfzLYhwDZbu3e/QOfogWt/FbjoRLlmIANYA8zGWJWc7H//Av8BTjdfJ5v7if89be0X6N43P+N4jm661uHAu8D5wBvB+tOHrrmEzkIk4e5tbc5yZhhwwPb+oNnWGxmklDpivj4KDDJfB7rGYO0HHdqDnaPbME0WMzBm5n36mk2zzjqgDFiMMYuuUkq1OfTTe23m9mogj8i/i7wg5+gO/gDcBXjM98H601euWQHviMhqEVlotiXcvZ0c9uVoej1KKSUicY3p7o5z+CMiWcBLwG1KqRrTtNtt/enua1ZKtQPTRSQXeAWY0F3n7glE5HKgTCm1WkTO7eHudCdnKaUOichAYLGIbLNvTJR7W2sizhwCRtjeDzfbeiOlIjIEwPxfZrYHusZg7cMd2oOdI+6IiBtDgDyjlHo5RH/6xDVbKKWqgPcxzCy5ImJNCu399F6buT0HqCTy76IyyDnizZnAlSJSAjyHYdL6Y5D+9IVrRil1yPxfhjFZOJUEvLe1EHFmJTDWjMxIwXDOvdbDfYqW1wArIuNmDL+B1X6TGdVxGlBtqrD/AS4Wkf5mVMbFGHbgI0CNiJxmRnHc5Hcsp3PEFbMf/wC2KqV+Z9vUl6+5wNRAEJF0DB/QVgxhcq1Df+z9vBZ4TxnG7teA+WYkUxEwFsPR6njvm58JdI64opS6Vyk1XClVaPbnPaXUDUH60+uvWUQyRSTbeo1xT24iEe/t7nIS9bY/jGiHHRj25h/1dH/C7POzwBGgFcPG+TUMu+67wE5gCTDA3FeAh83r2wgU247zVWCX+fcVW3uxeSPvBv5CR8YDx3N0w/WehWE33gCsM/8u7ePXPBVYa17zJuAnZvtojAFxF/ACkGq2p5nvd5nbR9uO9SPzurZjRuYEu/cDnaOb7/Fz6YjO6rPXbJ53vfm32epTIt7bOu2JRqPRaKJGm7M0Go1GEzVaiGg0Go0marQQ0Wg0Gk3UaCGi0Wg0mqjRQkSj0Wg0UaOFiEaj0WiiRgsRjSYMROQJEbk2xD4lIpIfwTEXiMhfIuzHdBG5NJLPaDTxRAsRjaZ3MR1jYZxGkxBoIaLR+CEiPxajQNFSEXlWRH7gt/0CMYojbRSjEFiqbfNdZvsKETnJ3P8KMQobrRWRJSISVsZfEblORDaJUYDqIzMlx8+AL4pRqOiLZnqM/zHPt1ZE5pmfXSAir4rIB2IUF7o/Rl+PRuODFiIajQ0ROQW4BpgGfA4jNYR9exrwBPBFpdTJGJmwv2Xbpdps/wtG+nKApcBpSqkZGAkE7wqzOz8B5iqlpgFXKqVazLZFSqnpSqlFGGk83lNKnQqcB/zGzLUERsK+azBSpVwnIsWdT6HRdA0tRDQaX84EXlVKNSmlaoHX/baPB/YqpXaY75/EqChp8azt/+nm6+HAf0RkI3AnMDnMvnwCPCEit2BU33PiYuAeMeqLfICRN2qkuW2xUqpSKdUIvIyRa0yjiSlaiGg0sUU5vP4z8BdTQ/kGxkAf+kBKfRO4DyOV92oRyXPYTYBrTM1kulJqpFJqq0NfnN5rNF1GCxGNxpdPgCtEJE2MYleX+23fDhRa/g7gy8CHtu1ftP3/zHydQ0ethpsJExEZo5RarpT6CVCOIUxqMerJW/wH+C8znTciMsO27SIRGWCmjL/KvDaNJqboyoYajQ2l1EoReQ0j1XopRlrtatv2JhH5CvCCWaxoJfCo7RD9RWQD0IxRtxrgp+b+x4H3gKIwu/MbERmLoW28i5EWfD8d5qtfAv+N4XvZICIuYC8dgm8FRsGu4cDTSqlVYZ5XowkbnQpeo/FDRLKUUnUikgF8BCxUSq3p6X5FgogswKgp8d2e7oumb6M1EY2mM4+JyCQM38WTvU2AaDTdidZENJoeRkR+BFzn1/yCUurnPdEfjSYStBDRaDQaTdTo6CyNRqPRRI0WIhqNRqOJGi1ENBqNRhM1WohoNBqNJmr+PwhGb9/gqTMWAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "data = plot_writer_data(manager, tag=\"episode_rewards\", smooth_weight=0.8) # smoothing tensorboard-style" ] }, { "cell_type": "markdown", "id": "d6d1bc02-d647-48e8-9c17-e9fbf79f297b", "metadata": {}, "source": [ "### Evaluation of the trained agent" ] }, { "cell_type": "code", "execution_count": 6, "id": "2a5dd296", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[38;21m[INFO] 17:00: Evaluating PPOAgent... \u001b[0m\n", "[INFO] Evaluation:/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:174: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed a `seed` instead of using `Env.seed` for resetting the environment random number generator.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:187: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed `options` to allow the environment initialisation to be passed additional information.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:195: UserWarning: \u001b[33mWARN: The result returned by `env.reset()` was not a tuple of the form `(obs, info)`, where `obs` is a observation and `info` is a dictionary containing additional information. Actual type: ``\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:219: DeprecationWarning: \u001b[33mWARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. \u001b[0m\n", " logger.deprecation(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:141: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method was expecting numpy array dtype to be float32, actual type: float64\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:165: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method is not within the observation space.\u001b[0m\n", " logger.warn(f\"{pre} is not within the observation space.\")\n", "................................................................................................................................ Evaluation finished \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PPOAgent
count128.000000
mean185.779274
std36.115669
min82.613569
25%158.785278
50%184.637599
75%218.280214
max258.224123
\n", "
" ], "text/plain": [ " PPOAgent\n", "count 128.000000\n", "mean 185.779274\n", "std 36.115669\n", "min 82.613569\n", "25% 158.785278\n", "50% 184.637599\n", "75% 218.280214\n", "max 258.224123" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "evaluation = evaluate_agents([manager], n_simulations=128, plot=False)\n", "evaluation.describe()" ] }, { "cell_type": "markdown", "id": "c7a2362e-2633-4851-a1b8-248fe3df0534", "metadata": {}, "source": [ "### Small peek into the agents policy" ] }, { "cell_type": "code", "execution_count": 7, "id": "4cb1cb69-2847-4f88-a631-914716a15e4c", "metadata": {}, "outputs": [], "source": [ "agent = manager.agent_handlers[0] # select the agent from the manager" ] }, { "cell_type": "code", "execution_count": 9, "id": "f8e915f0-72c4-4ba1-bc2f-1fad5ada10ac", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/frost/.local/lib/python3.10/site-packages/gym/spaces/box.py:127: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", " logger.warn(f\"Box bound precision lowered by casting to {self.dtype}\")\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:174: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed a `seed` instead of using `Env.seed` for resetting the environment random number generator.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:187: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed `options` to allow the environment initialisation to be passed additional information.\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:195: UserWarning: \u001b[33mWARN: The result returned by `env.reset()` was not a tuple of the form `(obs, info)`, where `obs` is a observation and `info` is a dictionary containing additional information. Actual type: ``\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:219: DeprecationWarning: \u001b[33mWARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. \u001b[0m\n", " logger.deprecation(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:141: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method was expecting numpy array dtype to be float32, actual type: float64\u001b[0m\n", " logger.warn(\n", "/home/frost/.local/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:165: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method is not within the observation space.\u001b[0m\n", " logger.warn(f\"{pre} is not within the observation space.\")\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Day: 1.0, Mean temp: 4.66, stage: 0, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 2.0, Mean temp: 8.107, stage: 0, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 3.0, Mean temp: 4.794, stage: 1, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 4.0, Mean temp: 6.229, stage: 1, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 5.0, Mean temp: 5.328, stage: 1, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 6.0, Mean temp: 6.831, stage: 1, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 7.0, Mean temp: 11.42, stage: 2, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 8.0, Mean temp: 11.808, stage: 3, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 9.0, Mean temp: 7.065, stage: 3, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 10.0, Mean temp: 5.824, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 11.0, Mean temp: 7.3, stage: 3, weight of fruit: 0.0\n", "Action is sow some seeds\n", "\n", "Day: 12.0, Mean temp: 7.8, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 13.0, Mean temp: 10.459, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 14.0, Mean temp: 8.196, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 15.0, Mean temp: 6.916, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 16.0, Mean temp: 8.3, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 17.0, Mean temp: 3.395, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 18.0, Mean temp: 3.507, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 19.0, Mean temp: 1.572, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 20.0, Mean temp: -0.618, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 21.0, Mean temp: 1.371, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 22.0, Mean temp: 2.858, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 23.0, Mean temp: 0.966, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 24.0, Mean temp: 1.87, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 25.0, Mean temp: 5.66, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 26.0, Mean temp: 7.941, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 27.0, Mean temp: 5.144, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 28.0, Mean temp: 5.272, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 29.0, Mean temp: 7.686, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 30.0, Mean temp: 11.335, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 31.0, Mean temp: 10.712, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 32.0, Mean temp: 10.321, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 33.0, Mean temp: 9.554, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 34.0, Mean temp: 6.328, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 35.0, Mean temp: 6.586, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 36.0, Mean temp: 4.039, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 37.0, Mean temp: 4.962, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 38.0, Mean temp: 8.219, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 39.0, Mean temp: 9.835, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 40.0, Mean temp: 7.665, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 41.0, Mean temp: 6.337, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 42.0, Mean temp: 5.62, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 43.0, Mean temp: 7.099, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 44.0, Mean temp: 7.855, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 45.0, Mean temp: 10.274, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 46.0, Mean temp: 12.223, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 47.0, Mean temp: 7.699, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 48.0, Mean temp: 7.065, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 49.0, Mean temp: 6.367, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 50.0, Mean temp: 8.157, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 51.0, Mean temp: 6.388, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 52.0, Mean temp: 8.593, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 53.0, Mean temp: 10.491, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 54.0, Mean temp: 9.339, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 55.0, Mean temp: 6.897, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 56.0, Mean temp: 4.347, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 57.0, Mean temp: 2.649, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 58.0, Mean temp: 4.304, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 59.0, Mean temp: 8.054, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 60.0, Mean temp: 6.459, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 61.0, Mean temp: 5.813, stage: 3, weight of fruit: 0.0\n", "Action is doing nothing\n", "\n", "Day: 62.0, Mean temp: 4.74, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 63.0, Mean temp: 5.463, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 64.0, Mean temp: 7.361, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 65.0, Mean temp: 6.06, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 66.0, Mean temp: 6.398, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 67.0, Mean temp: 8.256, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 68.0, Mean temp: 7.85, stage: 3, weight of fruit: 0.0\n", "Action is remove weeds by hand\n", "\n", "Day: 69.0, Mean temp: 10.868, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 70.0, Mean temp: 12.469, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 71.0, Mean temp: 8.836, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 72.0, Mean temp: 7.698, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 73.0, Mean temp: 7.344, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 74.0, Mean temp: 9.662, stage: 3, weight of fruit: 0.0\n", "Action is harvesting\n", "\n", "Day: 75.0, Mean temp: 7.742, stage: 3, weight of fruit: 0.0\n", "Action is scatter pesticide\n", "\n", "Day: 76.0, Mean temp: 7.807, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 77.0, Mean temp: 9.912, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 78.0, Mean temp: 9.564, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 79.0, Mean temp: 7.413, stage: 3, weight of fruit: 0.0\n", "Action is harvesting\n", "\n", "Day: 80.0, Mean temp: 6.346, stage: 3, weight of fruit: 0.0\n", "Action is doing nothing\n", "\n", "Day: 81.0, Mean temp: 5.97, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 82.0, Mean temp: 5.594, stage: 3, weight of fruit: 0.0\n", "Action is doing nothing\n", "\n", "Day: 83.0, Mean temp: 6.003, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 84.0, Mean temp: 6.43, stage: 3, weight of fruit: 0.0\n", "Action is scatter fertilizer\n", "\n", "Day: 85.0, Mean temp: 6.677, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 86.0, Mean temp: 7.663, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 87.0, Mean temp: 8.407, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 88.0, Mean temp: 5.038, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 89.0, Mean temp: 4.676, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 90.0, Mean temp: 5.745, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 91.0, Mean temp: 5.488, stage: 3, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 92.0, Mean temp: 7.189, stage: 4, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 93.0, Mean temp: 8.946, stage: 5, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 94.0, Mean temp: 10.253, stage: 5, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 95.0, Mean temp: 13.488, stage: 5, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 96.0, Mean temp: 12.475, stage: 5, weight of fruit: 0.0\n", "Action is 5L of water\n", "\n", "Day: 97.0, Mean temp: 13.701, stage: 5, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 98.0, Mean temp: 17.629, stage: 6, weight of fruit: 0.0\n", "Action is 1L of water\n", "\n", "Day: 99.0, Mean temp: 16.655, stage: 6, weight of fruit: 1.0\n", "Action is 1L of water\n", "\n", "Day: 100.0, Mean temp: 16.302, stage: 6, weight of fruit: 1.8462644348974382\n", "Action is 1L of water\n", "\n", "Day: 101.0, Mean temp: 16.295, stage: 6, weight of fruit: 2.9439341468885543\n", "Action is 1L of water\n", "\n", "Day: 102.0, Mean temp: 16.228, stage: 6, weight of fruit: 4.277763800353128\n", "Action is doing nothing\n", "\n", "Day: 103.0, Mean temp: 9.44, stage: 6, weight of fruit: 4.277763800353128\n", "Action is 1L of water\n", "\n", "Day: 104.0, Mean temp: 6.957, stage: 6, weight of fruit: 4.277763800353128\n", "Action is 5L of water\n", "\n", "Day: 105.0, Mean temp: 10.949, stage: 6, weight of fruit: 4.277763800353128\n", "Action is doing nothing\n", "\n", "Day: 106.0, Mean temp: 14.965, stage: 6, weight of fruit: 5.864047649310217\n", "Action is 1L of water\n", "\n", "Day: 107.0, Mean temp: 15.352, stage: 6, weight of fruit: 7.669518508245714\n", "Action is harvesting\n", "\n", "Plant is Dead\n" ] } ], "source": [ "env = env_ctor(**env_kwargs)\n", "obs = env.reset()\n", "\n", "actions_txt = [\"doing nothing\", \n", " \"1L of water\", \n", " \"5L of water\", \n", " \"harvesting\",\n", " \"sow some seeds\",\n", " \"scatter fertilizer\",\n", " \"scatter herbicide\",\n", " \"scatter pesticide\",\n", " \"remove weeds by hand\",]\n", "episode = pd.DataFrame()\n", "for day in range(365):\n", " action = agent.policy(obs) \n", " print(\"Day: {}, Mean temp: {}, stage: {}, weight of fruit: {}\".format(obs[0], np.round(obs[1],3), \n", " int(obs[7]), obs[15]))\n", " obs,reward, is_done,_ = env.step(action)\n", " print(\"Action is\", actions_txt[action])\n", " episode = pd.concat([episode, pd.DataFrame({'action':[actions_txt[action]],\n", " 'reward':[reward]})], ignore_index=True)\n", " print('')\n", " if is_done:\n", " print('Plant is Dead')\n", " break" ] }, { "cell_type": "code", "execution_count": 15, "id": "9707f129-5d05-480d-9e72-68258fb56584", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs0AAAFzCAYAAADFZzQZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAisElEQVR4nO3deZhlZ10n8O8POpgAgYCpYcISmwEUGZYgTUTCEtkGGZGgoIOCYRkjI8vgODziMhgYcBkFVHDAICFBkZ2wqQiGJSFAoEN2QgCToDCBNJsElWDCO3+ct5Kbzq1+qzt1q6qbz+d56qlzzz3L7557lu9977nnVGstAADAym6w0QUAAMBmJzQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMDAlo0uYDUOPvjgtnXr1o0uAwCAfdwZZ5zx5dba0s7994rQvHXr1mzfvn2jywAAYB9XVZ+b19/pGQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAwMJCc1XtX1Ufq6qzq+r8qnpe739CVV1cVWf1v8MWVQMAAKyFLQuc9hVJHtRa+2ZV7ZfkQ1X1N/25Z7fW3rzAeQMAwJpZWGhurbUk3+wP9+t/bVHzAwCARVlkS3Oq6oZJzkhyxyR/0lo7var+W5IXVtVzk5yc5DmttSvmjHtMkmOS5NBDD93lfO717Nesdel7rTN+/+c3ugQAgH3OQn8I2Fq7qrV2WJLbJjm8qu6a5NeS3DnJvZPcMsmvrjDuca21ba21bUtLS4ssEwAAdmldrp7RWvt6kvcneXhr7dI2uSLJq5Mcvh41AADAnlrk1TOWquqg3n1Akocm+VRVHdL7VZKjkpy3qBoAAGAtLPKc5kOSnNjPa75Bkje21t5VVe+rqqUkleSsJE9dYA0AAHC9LfLqGeckueec/g9a1DwBAGAR3BEQAAAGhGYAABgQmgEAYGChNzdh7/QPz7/bRpewaRz63HM3ugQAYBPQ0gwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwIDQDAMCA0AwAAANCMwAADAjNAAAwsLDQXFX7V9XHqursqjq/qp7X+9++qk6vqs9W1Ruq6kaLqgEAANbCIluar0jyoNbaPZIcluThVXWfJL+X5CWttTsm+VqSpyywBgAAuN4WFprb5Jv94X79ryV5UJI39/4nJjlqUTUAAMBaWOg5zVV1w6o6K8llSd6b5O+TfL21dmUf5PNJbrPIGgAA4PpaaGhurV3VWjssyW2THJ7kzqsdt6qOqartVbV9x44diyoRAACG1uXqGa21ryd5f5IfSXJQVW3pT902yRdWGOe41tq21tq2paWl9SgTAADmWuTVM5aq6qDefUCShya5IFN4fkwf7Ogkb19UDQAAsBa2jAfZY4ckObGqbpgpnL+xtfauqvpkktdX1QuSnJnkVQusAQAArreFhebW2jlJ7jmn/0WZzm8GAIC9gjsCAgDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwIzQAAMCA0AwDAgNAMAAADQjMAAAwsLDRX1e2q6v1V9cmqOr+q/nvvf2xVfaGqzup/j1hUDQAAsBa2LHDaVyb5ldbaJ6rqwCRnVNV7+3Mvaa39wQLnDQAAa2Zhobm1dmmSS3v35VV1QZLbLGp+AACwKOtyTnNVbU1yzySn915Pr6pzqur4qrrFetQAAAB7auGhuapumuQtSZ7VWvtGkpcnuUOSwzK1RL9ohfGOqartVbV9x44diy4TAABWtNDQXFX7ZQrMr22tvTVJWmtfaq1d1Vr7TpJXJjl83ritteNaa9taa9uWlpYWWSYAAOzSIq+eUUleleSC1tqLZ/ofMjPYo5Oct6gaAABgLSzy6hlHJHlCknOr6qze79eTPK6qDkvSklyS5BcXWAMAAFxvi7x6xoeS1Jyn/npR8wQAgEVwR0AAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGFhaaq+p2VfX+qvpkVZ1fVf+9979lVb23qj7T/99iUTUAAMBaWGRL85VJfqW1dpck90nytKq6S5LnJDm5tXanJCf3xwAAsGktLDS31i5trX2id1+e5IIkt0nyqCQn9sFOTHLUomoAAIC1sC7nNFfV1iT3THJ6klu11i7tT30xya1WGOeYqtpeVdt37NixHmUCAMBcCw/NVXXTJG9J8qzW2jdmn2uttSRt3nitteNaa9taa9uWlpYWXSYAAKxooaG5qvbLFJhf21p7a+/9pao6pD9/SJLLFlkDAABcX6sKzVV18mr67fR8JXlVkgtaay+eeeodSY7u3UcnefvqSgUAgI2xZVdPVtX+SW6c5OB+abjqT90s04/6duWIJE9Icm5VndX7/XqS303yxqp6SpLPJfnpPSsdAADWxy5Dc5JfTPKsJLdOckauCc3fSPKyXY3YWvvQzPA7e/DqSwQAgI21y9DcWvujJH9UVc9orb10nWoCAIBNZdTSnCRprb20qu6bZOvsOK211yyoLgAA2DRWFZqr6s+T3CHJWUmu6r1bEqEZAIB93qpCc5JtSe7Sr6sMAADfVVZ7nebzkvz7RRYCAACb1Wpbmg9O8smq+liSK5Z7ttZ+YiFVAQDAJrLa0HzsIosAAIDNbLVXz/jgogsBAIDNarVXz7g809UykuRGSfZL8s+ttZstqjAAANgsVtvSfOByd1VVkkcluc+iigIAgM1ktVfPuFqbvC3Jf1r7cgAAYPNZ7ekZPznz8AaZrtv8rYVUBAAAm8xqr57xyJnuK5NckukUDQAA2Oet9pzmJy26EAAA2KxWdU5zVd22qk6qqsv631uq6raLLg4AADaD1f4Q8NVJ3pHk1v3vnb0fAADs81Ybmpdaa69urV3Z/05IsrTAugAAYNNYbWj+SlU9vqpu2P8en+QriywMAAA2i9WG5icn+ekkX0xyaZLHJHnigmoCAIBNZbWXnHt+kqNba19Lkqq6ZZI/yBSmAQBgn7balua7LwfmJGmtfTXJPRdTEgAAbC6rDc03qKpbLD/oLc2rbaUGAIC92mqD74uSfKSq3tQfPzbJCxdTEgAAbC6rvSPga6pqe5IH9V4/2Vr75OLKAgCAzWPVp1j0kCwoAwDwXWe15zQDAMB3LaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGFhaaq+r4qrqsqs6b6XdsVX2hqs7qf49Y1PwBAGCtLLKl+YQkD5/T/yWttcP6318vcP4AALAmFhaaW2unJPnqoqYPAADrZSPOaX56VZ3TT9+4xQbMHwAAdst6h+aXJ7lDksOSXJrkRSsNWFXHVNX2qtq+Y8eOdSoPAACua11Dc2vtS621q1pr30nyyiSH72LY41pr21pr25aWltavSAAA2Mm6huaqOmTm4aOTnLfSsAAAsFlsWdSEq+p1SY5McnBVfT7JbyU5sqoOS9KSXJLkFxc1fwAAWCsLC82ttcfN6f2qRc0PAAAWxR0BAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAICBLRtdAOzrjnjpERtdwqZx2jNO2+gSAGCPaGkGAIABoRkAAAaEZgAAGBCaAQBgQGgGAICBhYXmqjq+qi6rqvNm+t2yqt5bVZ/p/2+xqPkDAMBaWWRL8wlJHr5Tv+ckObm1dqckJ/fHAACwqS0sNLfWTkny1Z16PyrJib37xCRHLWr+AACwVtb7nOZbtdYu7d1fTHKrlQasqmOqantVbd+xY8f6VAcAAHNs2A8BW2stSdvF88e11ra11rYtLS2tY2UAAHBt6x2av1RVhyRJ/3/ZOs8fAAB223qH5nckObp3H53k7es8fwAA2G2LvOTc65J8JMkPVNXnq+opSX43yUOr6jNJHtIfAwDAprZlURNurT1uhacevKh5AgDAIrgjIAAADAjNAAAwIDQDAMDAws5pBlhrH3zAAze6hE3jgad8cKNLAPiuoqUZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAATc3Afgu9bJfeedGl7BpPP1Fj9zoEoBNTkszAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADCwZSNmWlWXJLk8yVVJrmytbduIOgAAYDU2JDR3P9pa+/IGzh8AAFbF6RkAADCwUS3NLcl7qqol+dPW2nE7D1BVxyQ5JkkOPfTQdS4PAHbPCx//mI0uYdP4jb9480aXAGtuo1qa79da+6EkP5bkaVX1gJ0HaK0d11rb1lrbtrS0tP4VAgBAtyGhubX2hf7/siQnJTl8I+oAAIDVWPfQXFU3qaoDl7uTPCzJeetdBwAArNZGnNN8qyQnVdXy/P+ytfbuDagDAABWZd1Dc2vtoiT3WO/5AgDAnnLJOQAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAaEZgAAGBCaAQBgQGgGAIABoRkAAAa2bHQBAAA7u+CF79voEjaFH/yNB210CXRamgEAYEBoBgCAAaEZAAAGhGYAABgQmgEAYEBoBgCAAaEZAAAGhGYAABhwcxMAgH3Yscceu9ElbBrXZ1loaQYAgAGhGQAABoRmAAAYEJoBAGBAaAYAgAGhGQAABoRmAAAYEJoBAGBAaAYAgAGhGQAABoRmAAAYEJoBAGBAaAYAgIENCc1V9fCqurCqPltVz9mIGgAAYLXWPTRX1Q2T/EmSH0tylySPq6q7rHcdAACwWhvR0nx4ks+21i5qrX07yeuTPGoD6gAAgFXZiNB8myT/OPP4870fAABsStVaW98ZVj0mycNba/+1P35Ckh9urT19p+GOSXJMf/gDSS5c10L3zMFJvrzRRexDLM+1Y1muLctzbVmea8eyXFuW59raW5bn97XWlnbuuWUDCvlCktvNPL5t73ctrbXjkhy3XkWthara3lrbttF17Cssz7VjWa4ty3NtWZ5rx7JcW5bn2trbl+dGnJ7x8SR3qqrbV9WNkvyXJO/YgDoAAGBV1r2lubV2ZVU9PcnfJrlhkuNba+evdx0AALBaG3F6Rlprf53krzdi3gu2V51OshewPNeOZbm2LM+1ZXmuHctybVmea2uvXp7r/kNAAADY27iNNgAADAjN66SqnlhVL1vlsEdW1buux7y+uafjDqZ7ZlVdXlXn7dT/hH4pwT2Z5jOr6oKqeu0a1XhQVf3SWkxrvVXV1qr62ZnHh1XVI9Zguru9jOfUsq2q/rh3X70uV9VTq+rnr2+N7L7d2adcz/ksZL1cC31fed+Zx7tcH6vq1lX15hWe+0BVrdmv+qvq2Kr6n4Nhnl9VD1mree6Oqjpq9m68K73+2W1/M+nr5XnjIRdew9z95N6kqi6pqnOr6qyq2j7T37F9JxtxG+2qKmF973RWkj9b42n+UpKHttZ+bo2md1Cf5qptonVya5KfnXl8WJLdCidVNe93Cru1jPs0rlVLa217a+2ZOw/bWntFa+01u1PjCvNj89qaxayXa+HIJFeH5tH62Fr7f621PQoBi9Bae25r7e82aPZHJbnLaKCVtv292Rquj1uziv3kXuJHW2uHreHl4PbNY3trbeF/mVasC5O8Jsn5Sb4vybMzXX7unCTPmxnuU0lOSPLpJK9N8pAkpyX5TJLD+3C3TPK2Pu5Hk9w90weAS5IcNDPfzyS5VZKlJG/p8/t4kiPm1PhXSe7eu89M8tze/fwkv9C7r1Nz7//4JB/LFCr/NMkNe/8n9dfxsSSvTPKy3v+xSc5LcnaSU+bUcmSSU3pNFyZ5RX99T07yhzPD/UKSl8wZ/5tJXtin/9Ekt+r9H5nk9P76/m6m/7FJjk/ygSQXJXnmzLR+o7+GDyV5XZ/ueTvN74QkjxmsA/+jv+bzkjyr93tFkm8nOTfJL+/O+5HkpklOTvKJPv6j+vOvT/Kv/b34/ZXet8xZJ6/nOn6TXvPZ/TX+TO9/7yQf7v0/luTAPu9Te+2fSHLfPuxHk/xTr/1Xk/xDkh398c/0eRzfp3PmzGt+YqbLNr4vyQd3qutay3i109ipll/OtE6+a2bY5XX52CT/M8mt+7DLf1dl2s7nbnt9vD/PtG2/bpXL8sG95nP7a/ievnzf2p9/VH/vb5Rk/yQXzXmfrrPt9WFf3ad7ZqaDx/LrfFuS92batzw903p8Zl8+t+zD3SHJu5Oc0d/XO8+Z7wNnls2ZSQ5cj33KLpblRq+XWzPt61+b5IIkb05y4/7cvTKtg2dkusrSIb3/M5N8si+r1/dpfDHTdf7PSnL/9PWxD3/HTPu5s/vruUMf57z+/AF9OhckOSnTvnFbf+5hST7Sx3tTkpuucj+w8/5yuZbD+nI8p8/rFjvvOzOtY8/LNfu0O/f+S5nWwfMzNVp8LsnBu7Hf39rfg3My7TMPzfRB46tJLu7L7g6Z9v+/19/HTye5/8zxaHnbPzYrHyv+V6Z96rVe+6L++uu6INN2cH6S9/T39BcybVNnZ9r3LK9XJ2TaH56e5MXZzbyQOdtwdr2f3DTLahXL8pIV1qmr189djLtPH9uv83rX6Q3ZmuQ7Se7THz8s0y8oK1MYfFeSB/Thrkxyt97/jL7SVaYD4tv6+C9N8lu9+0FJzurdf5TkSb37h5P8Xe/+yyT3692HJrlgTo3PSfK0JDfvb8Lf9v7vz3RHwpVq/sEk70yyXx/+/yb5+SSHZDq4LGU6iJ+Waw5w5ya5Te8+aE4tRyb5VpL/kOmyfO9N8pi+Mv39zLw+nORuc8ZvSR7Zu/9Pkt/s3bfINT/+/K9JXjSzcX84Uwg5OMlXkuyX6eB1bpIbJ7lZks9mD0LzzHRu0l/D+UnuOdhYR+/HliQ36/0O7rVVZg6Kq1jXrl4n12Ad/6kkr5x5fPP+vl+U5N6938163TdOsn/vd6ck22fe93fNTOOJy+tMf/zbSR6/vN5kOrDdpA/3+fQQN6e2q5fxaqcxp5arH2dOaN5pfk9L8sZdbXt9vDOSHLDKZbl/kn9M8v2932uSPKsvz4t6vz/o68oRmQ5wr5sz7etse0l+JdOlL5Pkzpm22/376/xspoPjUqaD41P7cC/JNQeIk5PcaWa/8745831nrjn43rTXvfB9ygrLcsPXy0zbX5tZJsdn+vC1X6Z90VLv/zMz783/S/I9O713x2Zm/cu1Q/PpSR7du/fvr29rrgnN/2Nm2nfPdOzZlml/ckqSm/TnfjX9wD7YB8zbXy7Xck6SB/bu56c3fuS6ofkZvfuXkvxZ735Zkl/r3Q/vy23ePnOl/f47kxzdu5+ca46jV8+7P/5ArjkmPCLXHD+vfv+z8rHi3pnCzP6ZtpfPZH1C85VJDuuP35jpw+b3zgzzgpllekKmbWz5A+hu5YXM34avXjabeVmtYllenCmknpHkmJn+11pHdrHO77PH9p3/1vNr0c+11j7aux/W/87sj2+aaSf9D0kubq2dmyRVdX6Sk1trrarO7QsjSe6X6WCQ1tr7qup7q+pmSd6Q5LmZWo3+S3+cTK3Vd6mq5VpuVlU3ba3Nnvt7aqaWjIszfRJ6aFXdOMntW2sXVtUvrFDz3TOtOB/v0z8gyWWZNsIPtNZ29NfyhiTf38c9LckJVfXGJG9dYXl9rLV2UR/3dZk24jdX1fuS/HhVXZDpoHrunHG/nWkFSqaN4KG9+7ZJ3lBVh2Q6cF48M85ftdauSHJFVV2W6RP3/ZOc1Fr7l17Hnt6E5n59Ov/cp/PWPu0zdzHO6P3YL8lvV9UDMm0gt+k172xX69rsOnl9nZvkRVX1e5l2mqdW1d2SXNpa+3iStNa+kSRVdZMkL6uqwzK1yH7/CtOc91p+YuY8yf0z7dST5L2tta+u0zRWVFVHZGotuF/vNXfb693vaK3965zJzFuW98i0b/h0H+bEJE9rrf1hVf19Vf1gksMztSA9INOHzVPnTHvetne/TB/E01r7VFV9Lte8J+9vrV2e5PKq+qdMB87lGu/eX8t9k7xp5jV+zwrzfXE/v++trbXPV9VK6+Za7lM283r5j62103r3X2Ta3t+d5K5J3ttf+w2TXNqHOSfJa6vqbZm+AVhRVR2Y6UPESf01fqv3nx3sAUn+uD9/TlWd0/vfJ9NpC6f14W+UqdV5ZO7+sqpuninkf7APd2Km1ut5lt+7M5L8ZO++X5JH9zrfXVVfW2Hclfb7PzIzrT/PFKhXMjv/rSsMM+9YcUSSt/fl/K2qeucK4661i1trZ/Xu5ZrvWlUvyPQB7qaZvq1Y9qbW2lW9e7fyQuZvw6P6NtOy2pX7tda+UFX/LtO296nW2imrGS/7/rH9WtYzNP/zTHcl+Z3W2p/ODlBVW5NcMdPrOzOPv5NxvR9JcseqWsp0vtYLev8bZPrU8a1djPvxTK0MF2Vq2T0408H/jEHNz0hyYmvt13bqf9RKM2qtPbWqfjjJf05yRlXdq7X2lZ0HW+HxnyX59Uxfbb56hVn8W+sfxTId/JaX20uTvLi19o6qOjLTJ+Fls8t9dpyNMno/fi5Ti9u9Wmv/VlWXZDpY72xX69o/zxl+j7TWPl1VP5SpheYFVXVypq9h5/nlJF9Kco9M6+au1stZleSnWmsXXqvntC6t9rWsxTTmT3j6MPaqJD8x84F07rbXDzZz57fCsnz7LmZ9SpIfS/Jvmb6OPyFT2Hr2nGlfZ9sbvKzR/ugGSb7eWjtsVxNprf1uVf1Vf02nVdV/yjrsUzb5ejlvH1dJzm+t/cic4f9zpqD7yCS/0cP/IlSmsP+4BU1/V5bXrz3ZB6+031/r+W+mY8XOtRyQafs/qrV2dlU9MVPr77LZ9XF388K8bXh369vo4+pcrbUv9P+XVdVJmRogVhOa98RedWzf2Ub9+Olvkzx5ucWpqm7TP+Gs1qmZFmx6+Ptya+0bfYdxUqbWpgtmguh7kjxjeeTeknItrbVvZ/r697GZNqZTM31duLzirFTzyUkes1x/Vd2yqr4v01eDD+yt4Pv16S7P/w6ttdNba8/NdG7g7ea8xsNrutX4DTJ9RfmhXufpffifzXQu1O64eabz/5Lk6FUMf0qSo6rqgN5y88jdnN+yU/t0btxbsx6d+a2AV1vF+3HzJJf1jepHM50/mySXZ/rKa9n1XddWpapuneRfWmt/keT3k/xQpvOqDqmqe/dhDqzpByg3z9TS950kT8gU8ObVPu+1PKN64qyqe+5Bqaudxs7z3qW+jr8pya/OtAYnq9j25kxrpWW5taru2Ad7QqbzXpNp3XhWko/0VtjvzfQ133V+Wb/Ctje7P/n+TK2kF+487jy9lfbiqnpsH796q/i8+Z7bWvu9TAeNO2cd9imbfL08tKqWw/HPZtrHXZhkabl/Ve1XVf+x7wdv11p7f6bTJW6eqWVp7nravx34/PIHjar6nt6aNeuUPt9U1V0ztfAn03mqRyyva1V1k75ejMzdX7bW/inJ16rq/n242XV3NU5L8tO9lodlOs1ud3w4U0tqMq3ny/ve3drGB05L8siq2r+vzz++RtPdEwcmubRvIyv+CG1388IK2/CeLMPNtKyW1+8Dl7sztd6u9qok+/yxfWcbdUfA99T0depH+n72m5nORbpqlyNe49gkx9f0ddq/5NoB8A2ZVugnzvR7ZpI/6cNvyfTmPHXOdE9N8uDW2r9W1amZTmc4dVc1t9Y+WVW/meQ9fcf+b5m+Nv5oVR2baaX4eqZzmJb9flXdKdMnpZMz/WBhZx/PdC7bHTOd6zPbOvTGTOdxrfQ13UqOzfQ18tcy/TDk9rsauLX2iZq+Aj4709fD+2XamRxQVZ/PdF75q/rgf1pVf9i7/3G2pahP54RMPzBJpnP1dvX1zbIV349MPyB6Z02n7WzP1PKe1tpXquq0mi5F9DettWdfz3Vtte6W6X39TqZ14L+11r5dVT+T5KVVdUCmHzE8JNM5qm+p6dJY7841n4rPSXJVVZ2dqbXkxCTPqaqzkvxOkv+d5A+TnNPXtYuz+zvc1U5j51pG79d9M7UePK+qntf7PSKr3/ZmzVuW36qqJ2Vaf7dk2j5e0Yc/PdPXd8s73XOS/PuZVrdZ87a9TyV5eV+XrkzyxNbaFTX+6nXZz/XxfzPTNvL6XHebflY/AHwn03l/f9Pnseh9ymZeLy9M8rSqOj7TD/xe3mt7TJI/rum0hi192p9O8he9XyX549ba12v6avvNVfWozASd7gmZ9kvP76/9sX35L3t5klfXdKrbBektXa21HTW1UL6uqpZPtfnNXsOK5uwvPz7z9NFJXtGD+0WZftC5Ws/rtTwh03v/xUwBYrWekel1PjvTB6rleb8+ySur6pmZfjOzx1prH6/pdJRzMn1bcW6m3wBshP+VaZ+wo//fVajdnbxwnW24d+/OfnKzLatk2nee1PdBW5L8ZWvt3TPPf7cf26/FHQH3QjVdw/klrbWTN7oWgN1V01eo72qt3XWja9nsenC/qrV2ZW+Bf/nodKCNUP13Qv2DwSmZflD2iY2uazOyrPZem/L8GuarqoMyfaI7W2AG+K5waJI39lb8b2c6/3MzOq6mm6Xsn+mcfCFwZZbVXkpLMwAADGyGu6ABAMCmJjQDAMCA0AwAAANCM8A+oqqOrKr7zjx+ar+EHADXk6tnAOw7jsx0vdIPJ0lr7RW7HBqAVXP1DIBNrqrelukuf/sn+aPW2nFV9fAkv53pzn1fTvKUTHezuyrTjR2ekeTBSb7ZWvuDfmezVyS5cZK/T/Lk1trXquoDmW4C8aNJDkrylNbaLu/qBfDdyOkZAJvfk1tr98p018VnVtWtkrwyyU+11u6R5LGttUsyheKXtNYOmxN8X5PpNud3z3QXst+aeW5La+3wTLcj/60AcB1CM8Dm98x+q96PZmpxPibJKa21i5OktfbVXY3cbz99UGvtg73XiUkeMDPIW/v/M5JsXcO6AfYZQjPAJlZVRyZ5SJIf6a3KZyY5a41nc0X/f1X81gVgLqEZYHO7eZKvtdb+parunOQ+mc5tfkBV3T5JquqWfdjLkxy48wRaa/+U5GtVdf/e6wlJPrjzcACsTIsCwOb27iRPraoLklyY6RSNHZlO0XhrVd0gyWVJHprknUneXFWPyvRDwFlHJ3lFVd04yUVJnrRO9QPsE1w9AwAABpyeAQAAA0IzAAAMCM0AADAgNAMAwIDQDAAAA0IzAAAMCM0AADAgNAMAwMD/B8ux33mhrnwYAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "fig, ax = plt.subplots(figsize=(12,6))\n", "sns.countplot(data = episode, x = \"action\", order=episode['action'].value_counts().index)" ] }, { "cell_type": "markdown", "id": "dadc4a5a-8ef3-4f1b-93e8-09772582dc1c", "metadata": {}, "source": [ "From this, we see that PPO learned that pesticide destroy the soil and should be avoided in small quantity. Herbicide is not useful if we can remove weeds by hand." ] }, { "cell_type": "code", "execution_count": null, "id": "8b426d5e-fcf7-4449-908f-2ca8b59b333f", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.8" } }, "nbformat": 4, "nbformat_minor": 5 }