EmergentCommunication.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6abc0467-5820-46cc-b19b-143aabef83b6",
   "metadata": {},
   "source": [
    "Authors: Ran Harari and Adi Amuzig  \n",
    "E-mails: [srharari@campus.technion.ac.il](srharari@campus.technion.ac.il), \n",
    "         [adi.amuzig@campus.technion.ac.il](mailto:adi.amuzig@campus.technion.ac.il)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b456d4c-707a-4d1b-9c82-d0a4b8fa1941",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Emergent Communication\n",
    "\n",
    "Because the real world is dynamic and continually changing, many real-world scenarios demand coordination among different actors, and having a centralized system to oversee all actions is not possible in a large-scale environment.\n",
    "\n",
    "Furthermore, communication is quite beneficial in resolving the coordination challenge. In theory, we could simply transfer sensory data among robots, but this would be a lot of information that would be irrelevant because each unit will perceive the world differently owing to its location and sensors.\n",
    "\n",
    "The interaction between two agents with communication abilities follows a typical cycle with three levels: sensori-motor, conseptual, and language. The cycle is as follows:\n",
    "\n",
    "1. **Grounding** - Both agents precive the world and creating a world model (sensori-motor level)\n",
    "2. **Conceptualization** - The speaker determines which information is most useful to the listener in order for the mission to succeed and builds a semantic framework (conceptual level)\n",
    "3. **Production** - conveying a message of semantic experience through verbal or implicit behaviors (language level)\n",
    "4. **Comprehension** - The communication is received and analyzed by the listener into a semantic form (language level)\n",
    "5. **Interpretation** - interpreting the semantic structure in reference to one's own world model and carrying out the appropriate action (conceptual level)\n",
    "\n",
    "![Communication Cycle](images/emergent_com_theory.png)\n",
    "[A Practical Guide to Studying Emergent Communication through Grounded Language Games - Nevens, Eecke and Beuls]\n",
    "\n",
    "When it comes to communication, there are two primary options:\n",
    "\n",
    "- A pre-defined communication protocol, which means that all communication and the actions that occur from that communication are programmed into the system. Before the system goes live, all aspects of the communication cycle are fully specified.\n",
    "- Emergent communication is the study of how a communication system emerges among previously uncommunicative individuals. To put it another way, at least some components of the communication cycle must be learned. It is important to note that emergent communication necessitates many interactions between the same agent in order to *learn* a functional communication system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f8bd21-c281-46b5-b172-745cdcd8a907",
   "metadata": {},
   "source": [
    "## The Importance of communication\n",
    "\n",
    "Communication in general is helpful in a multi-robot system for both robot cooperation and human robot collaboration. This session will concentrate on the former. \n",
    "\n",
    "To investigate the role of communication, we used the multitaxi domain to create a situation with no communication between the agents. To confirm that our communication strategy worked, we first solved the multitaxi domain without communication and demonstrated that it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "486d3e97-10cc-4544-9762-dc7080db613c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EnvironmentWrapper created\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m:\u001b[41m_\u001b[0m|\n",
      "| | :G| | |\u001b[43m\u001b[31;1mP\u001b[0m\u001b[0m:G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (6,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (5,11), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: (6, 5), Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 2]\n",
      "active:['(4,)', '(1,)', '(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(1,)', '(3,)', '(6,)']\n",
      "all plans:[['(1,)', '(1,)', '(1,)', '(1,)', '(2,)', '(1,)', '(4,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(0,)', '(5,)'], ['(4,)', '(1,)', '(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(1,)', '(3,)', '(6,)']]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m:\u001b[41m_\u001b[0m|\n",
      "| | :G| | |\u001b[43m\u001b[31;1mP\u001b[0m\u001b[0m:G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (6,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (5,11), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: (6, 5), Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 2]\n",
      "active:['(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(3,)', '(4,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(0,)', '(5,)']\n",
      "all plans:[['(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(3,)', '(4,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(0,)', '(5,)'], ['(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(4,)', '(1,)', '(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(1,)', '(3,)', '(6,)']]\n",
      "---------------------------ROUND 1---------------------------------\n",
      "taxi_0 obs:\n",
      " MultiDiscrete([ 7 12  7 12  7 12  7 12  7 12  5  5]) , action: 4\n",
      "taxi_1 obs:\n",
      " MultiDiscrete([ 7 12  7 12  7 12  7 12  7 12  5  5]) , action: 1\n",
      "joint Message: [(2, 12), (1, 18)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : :\u001b[41m_\u001b[0m|\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | |\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m:G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (6,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,11), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 2---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 6  5  1  6  6  5  5 10  0  0  2  3  2 12  1 18] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 4 11  1  6  6  5  5 10  0  0  2  3  2 12  1 18] , action: 1\n",
      "joint Message: [(2, 11), (1, 17)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : :\u001b[41m_\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,11), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 3---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  5  1  6  6  5  5 10  0  0  2  3  2 11  1 17] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 3 11  1  6  6  5  5 10  0  0  2  3  2 11  1 17] , action: 1\n",
      "joint Message: [(2, 10), (1, 16)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : :\u001b[41m_\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m| : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,11), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 4---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  5  1  6  6  5  5 10  0  0  2  3  2 10  1 16] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 2 11  1  6  6  5  5 10  0  0  2  3  2 10  1 16] , action: 3\n",
      "joint Message: [(2, 9), (1, 15)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : : : : : : :\u001b[41m_\u001b[0m: |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m| : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 5---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  5  1  6  6  5  5 10  0  0  2  3  2  9  1 15] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 2 10  1  6  6  5  5 10  0  0  2  3  2  9  1 15] , action: 3\n",
      "joint Message: [(2, 8), (1, 14)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[41m_\u001b[0m: : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,9), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 6---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  5  1  6  6  5  5 10  0  0  2  3  2  8  1 14] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 2  9  1  6  6  5  5 10  0  0  2  3  2  8  1 14] , action: 3\n",
      "joint Message: [(2, 7), (1, 13)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[41m_\u001b[0m: : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,4), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 7---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  4  1  6  6  5  5 10  0  0  2  3  2  7  1 13] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 2  8  1  6  6  5  5 10  0  0  2  3  2  7  1 13] , action: 3\n",
      "joint Message: [(2, 6), (1, 12)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m: | : | : |\n",
      "| : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[41m_\u001b[0m: : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,3), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,7), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 8---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  3  1  6  6  5  5 10  0  0  2  3  2  6  1 12] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 2  7  1  6  6  5  5 10  0  0  2  3  2  6  1 12] , action: 1\n",
      "joint Message: [(2, 5), (1, 11)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[33;1mP\u001b[0m:\u001b[41m_\u001b[0m| : | : |\n",
      "| : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,7), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 9---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  2  1  6  6  5  5 10  0  0  2  3  2  5  1 11] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 1  7  1  6  6  5  5 10  0  0  2  3  2  5  1 11] , action: 3\n",
      "joint Message: [(2, 4), (1, 10)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : |\u001b[41m\u001b[33;1mP\u001b[0m\u001b[0m: | : | : |\n",
      "| :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,6), Collided: False\n",
      "Passenger1: Location: (1, 6), Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [2, 3]\n",
      "---------------------------ROUND 10---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  1  1  6  6  5  5 10  0  0  2  3  2  4  1 10] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1  6  1  6  6  5  5 10  0  0  2  3  2  4  1 10] , action: 4\n",
      "joint Message: [(2, 3), (1, 9)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m| : : : |\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (1,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,6), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 3]\n",
      "---------------------------ROUND 11---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 1  1  1  6  6  5  5 10  0  0  4  3  2  3  1  9] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1  6  1  6  6  5  5 10  0  0  4  3  2  3  1  9] , action: 2\n",
      "joint Message: [(2, 2), (1, 8)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m:\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m|F: | : | : | : |F: |\n",
      "| : | : : : | :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m| : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,7), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 3]\n",
      "---------------------------ROUND 12---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 0  1  1  6  6  5  5 10  0  0  4  3  2  2  1  8] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 1  7  1  6  6  5  5 10  0  0  4  3  2  2  1  8] , action: 0\n",
      "joint Message: [(2, 1), (1, 7)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m\u001b[31;1mD\u001b[0m\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,7), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 3]\n",
      "---------------------------ROUND 13---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 0  0  1  6  6  5  5 10  0  0  4  3  2  1  1  7] , action: 6\n",
      "taxi_1 obs:\n",
      " [ 2  7  1  6  6  5  5 10  0  0  4  3  2  1  1  7] , action: 2\n",
      "joint Message: [(2, 0), (1, 6)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "---------------------------ROUND 14---------------------------------\n",
      "no plan found for passenger 0\n",
      "no available plan was found\n",
      "taxi_0 obs:\n",
      " [ 0  0  1  6  6  5  5 10  0  0  4  1  2  0  1  6] , action: 5\n",
      "taxi_1 obs:\n",
      " [ 2  8  1  6  6  5  5 10  0  0  4  1  2  0  1  6] , action: 2\n",
      "joint Message: [(2, 0), (1, 5)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,9), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "---------------------------ROUND 15---------------------------------\n",
      "no plan found for passenger 0\n",
      "no available plan was found\n",
      "taxi_0 obs:\n",
      " [ 0  0  1  6  6  5  5 10  0  0  4  1  2  0  1  5] , action: 5\n",
      "taxi_1 obs:\n",
      " [ 2  9  1  6  6  5  5 10  0  0  4  1  2  0  1  5] , action: 2\n",
      "joint Message: [(2, 0), (1, 4)]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "Thats all - part 1\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import math\n",
    "\n",
    "from src.Communication.COM_net import COM_net\n",
    "from src.agents.agent import DecisionMaker, Action_message_agent, Agent_Com, RandomDecisionMaker\n",
    "from src.control.Controller_COM import DecentralizedComController\n",
    "from src.decision_makers.planners.map_planner import AstarDM\n",
    "from src.environments.env_wrapper import EnvWrappper, TAXI_pickup_dropoff_REWARDS\n",
    "from multi_taxi import MultiTaxiEnv\n",
    "\n",
    "\n",
    "\n",
    "MAP2 = [\n",
    "    \"+-------+\",\n",
    "    \"| : |F: |\",\n",
    "    \"| : | : |\",\n",
    "    \"| : : : |\",\n",
    "    \"| | :G| |\",\n",
    "    \"+-------+\",\n",
    "]\n",
    "\n",
    "MAP = [\n",
    "    \"+-----------------------+\",\n",
    "    \"| : |F: | : | : | : |F: |\",\n",
    "    \"| : | : : : | : | : | : |\",\n",
    "    \"| : : : : : : : : : : : |\",\n",
    "    \"| : : : : : | : : : : : |\",\n",
    "    \"| : : : : : | : : : : : |\",\n",
    "    \"| : : : : : : : : : : : |\",\n",
    "    \"| | :G| | | :G| | | : | |\",\n",
    "    \"+-----------------------+\",\n",
    "]\n",
    "\n",
    "\"\"\"\n",
    "Builds Multi_taxi env\n",
    "\"\"\"\n",
    "env = MultiTaxiEnv(num_taxis=2, num_passengers=2, domain_map=MAP, observation_type='symbolic')\n",
    "\n",
    "env.seed(0)\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "def build_multi_env(env):\n",
    "    env.agents = env.taxis_names\n",
    "    env.action_spaces = {\n",
    "        agent_name: env.action_space for agent_name in env.agents\n",
    "    }\n",
    "    env.observation_spaces = {\n",
    "        agent_name: env.observation_space for agent_name in env.agents\n",
    "    }\n",
    "    env.possible_agents = [agent for agent in env.agents]\n",
    "    return EnvWrappper(env, env.agents)\n",
    "\n",
    "\n",
    "environment = build_multi_env(env)\n",
    "print('EnvironmentWrapper created')\n",
    "\n",
    "\"\"\"\n",
    "COM-agent - we will explain later\n",
    "\"\"\"\n",
    "\n",
    "class Heading_message_agent(Agent_Com):\n",
    "\n",
    "    def __init__(self, decision_maker : AstarDM , sensor_function =None, message_filter = None, AgentName = None, bandW = math.inf, union_recieve = True):\n",
    "        super().__init__(decision_maker , sensor_function, message_filter, AgentName, bandW, union_recieve)\n",
    "        self.last_action = None\n",
    "\n",
    "    def set_data_func(self, obs):\n",
    "        data = (self.decision_maker.taking_passenger,len(self.decision_maker.active_plan))\n",
    "        return data\n",
    "\n",
    "    # todo - implement your recive_func\n",
    "    def set_recive_func(self, obs, message):\n",
    "        pass\n",
    "\n",
    "    # saves last action of the agent - not necessary for com module\n",
    "    def set_last_action(self, action):\n",
    "        self.last_action = action\n",
    "\n",
    "\"\"\"\n",
    "part 1 - \n",
    "agents sends com but dont use it - A-star DM solves for each agent\n",
    "\"\"\"\n",
    "# after having our com-Astar-agents class we can set our env agents into a dicentralized_agents dict\n",
    "env_agents = environment.get_env_agents()\n",
    "decentralized_agents = {agent_name: Heading_message_agent(AstarDM(env ,single_plan=True, Taxi_index=int(agent_name[-1]), domain_map=MAP) ,AgentName=agent_name)             # Agent(LearningDecisionMaker(env.action_space))  # can use diffrent DM\n",
    "                        for agent_name in env_agents}\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "COM-module - we will explain later\n",
    "\"\"\"\n",
    "com = COM_net()\n",
    "\"\"\" \n",
    "    - MAC controller \n",
    "\"\"\"\n",
    "controller = DecentralizedComController(environment, decentralized_agents, com)\n",
    "\"\"\"\n",
    "activate \n",
    "\"\"\"\n",
    "controller.run(render=True, max_iteration=15)\n",
    "print(\"Thats all - part 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2d1b915-cb4f-4f48-8001-f8ec30e172e9",
   "metadata": {},
   "source": [
    "In the above example, we can see that both agents intended to go to the same location to pick up the same passenger. The first agent to arrive at the passenger's location got to pick him up, while the other wasted effort and received nothing in return."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039f92cf-3060-4f59-b4a8-3c28cd66bc75",
   "metadata": {},
   "source": [
    "## Creating the Communication Infrastructer\n",
    "\n",
    "In our solution we modeld the world as a POMDP usign the tuple $\\langle S, s_{0}, A, P, R, \\gamma, \\Omega, O \\rangle$ where \n",
    "\n",
    "- $S$ is the set of all possible environment configurations,\n",
    "- $s_{0}$ the initial state,\n",
    "- $A$ the action space, is the list of actions the taxi agents can execute within the environment **including communication**,\n",
    "- $P$ is a state transition probability function,\n",
    "- $R$ is the reward function,\n",
    "- $\\gamma \\in [0,1]$ is a discount factor,\n",
    "- $\\Omega$ is a set of observations where the **information form the observations also becomes an observation**, and\n",
    "- $O$ is an observation function\n",
    "\n",
    "\n",
    "## MAC-COM - simple to use: \n",
    "\n",
    "<img src='images/MAC-com.jpg' width=\"800\" height=\"800\">\n",
    "\n",
    "\n",
    "##### All you need for using com module:\n",
    "    - Implement Agent_com class (inherit from Agent_com)\n",
    "        for this agent: \n",
    "        1 - make sure to implement set_data_func -  decides what data to transmit whenever it called\n",
    "        2 - recieve_func - implement yours to decide what to do with a recieved data\n",
    "            (defualt is the union_func that add the message data to the observation)\n",
    "     - implement COM module (network) - and use COM-controller\n",
    "        3 - defualt is all recieve the last messages as Joint_Message\n",
    "<img src='images/MAC_COM_pic.jpg' width=\"500\" height=\"500\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a88cc71a-7d90-4602-9b77-08d40862ca8f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EnvironmentWrapper created\n",
      "---------------------------ROUND 1---------------------------------\n",
      "taxi_0 obs:\n",
      " MultiDiscrete([ 7 12  7 12  7 12  7 12  7 12  5  5]) , action: 3\n",
      "taxi_1 obs:\n",
      " MultiDiscrete([ 7 12  7 12  7 12  7 12  7 12  5  5]) , action: 2\n",
      "joint Message: [3, 2]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,11), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "---------------------------ROUND 2---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 0  0  1  6  6  5  5 10  0  0  4  1  3  2] , action: 5\n",
      "taxi_1 obs:\n",
      " [ 2 11  1  6  6  5  5 10  0  0  4  1  3  2] , action: 0\n",
      "joint Message: [5, 0]\n",
      "+-----------------------+\n",
      "|\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,11), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "---------------------------ROUND 3---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 0  0  1  6  6  5  5 10  0  0  4  1  5  0] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 3 11  1  6  6  5  5 10  0  0  4  1  5  0] , action: 6\n",
      "joint Message: [2, 6]\n",
      "+-----------------------+\n",
      "|\u001b[32m \u001b[0m:\u001b[43m_\u001b[0m|F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,11), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n",
      "+-----------------------+\n",
      "|\u001b[32m \u001b[0m:\u001b[43m_\u001b[0m|F: | : | : | : |F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : : : : : : : : : : : |\n",
      "| : : : : : | : : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,11), Collided: False\n",
      "Passenger1: Location: Taxi2, Destination: (5, 10)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 0)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, '__all__': False}\n",
      "Passengers Status's: [4, 1]\n"
     ]
    }
   ],
   "source": [
    "environment = build_multi_env(env)\n",
    "#\n",
    "print('EnvironmentWrapper created')\n",
    "\n",
    "\"\"\"\n",
    "in order to use com module: \n",
    "    - implement Agent_com class (inherit from Agent_com\n",
    "    - make sure to implement set_data_func - that decides what is the data that the agent will transmit whenever it called\n",
    "    - u can implement your recieve_func - that decides what to do with a recieved data\n",
    "        defualt is the union_func that add the message data to the observation \n",
    "\"\"\"\n",
    "\n",
    "class Heading_message_agent(Agent_Com):\n",
    "\n",
    "    def __init__(self, decision_maker , sensor_function =None, message_filter = None, AgentName = None, bandW = math.inf, union_recieve = True):\n",
    "        super().__init__(decision_maker , sensor_function, message_filter, AgentName, bandW, union_recieve)\n",
    "        self.last_action = None\n",
    "\n",
    "    def set_data_func(self, obs):\n",
    "        data = (self.last_action)\n",
    "        return data\n",
    "\n",
    "    # todo - implement your recive_func (defualt is union with obs)\n",
    "    # def set_recive_func(self, obs, message):\n",
    "    #     pass\n",
    "\n",
    "    # saves last action of the agent - not necessary for com module\n",
    "    def set_last_action(self, action):\n",
    "        self.last_action = action\n",
    "\n",
    "# after having our com-agent class we can set our env agents into a dicentralized_agents dict with a Random DM\n",
    "env_agents = environment.get_env_agents()\n",
    "decentralized_agents = {agent_name: Heading_message_agent(RandomDecisionMaker(env.action_space) ,AgentName=agent_name)             # Agent(LearningDecisionMaker(env.action_space))  # can use diffrent DM\n",
    "                        for agent_name in env_agents}\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "Simple use of communication network - \n",
    "    - build one using COM_net() - defualt architecture is - all masseges sent to all other agents\n",
    "            *U can use more options - see at COM_net() class doc.\n",
    "\"\"\"\n",
    "com = COM_net()\n",
    "\"\"\" \n",
    "    - initailze our new controller (DecentralizedComController) - using our env, our agents and our com module\n",
    "        - this controller will perform all joint action and message delieveries at any time-step\n",
    "\"\"\"\n",
    "controller = DecentralizedComController(environment, decentralized_agents, com)\n",
    "\"\"\"\n",
    "activate \n",
    "\"\"\"\n",
    "controller.run(render=True, max_iteration=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb23d9c8-6dca-4230-91ef-a0d00ff9d4ff",
   "metadata": {},
   "source": [
    "## Types of Communication\n",
    "\n",
    "When it comes to communication, we may split it into two categories:\n",
    "\n",
    "- **Explicit communication** – A direct, deliberate form of communication, where there is a clear associated intent for the transmitted information to be received by another agent or system over an established channel.\n",
    "\n",
    "<img src='https://zanzidigital.co.uk/wp-content/uploads/2017/11/chatbot-image.jpg' width=700/>\n",
    "\n",
    "\n",
    "- **Implicit communication** – Information is inferred, meaning This form of communication involves an action (or practical behaviour) representing as a message in itself, rather than a message being conveyed through language or codified gestures (such as a thumbs-up or a head nod).\n",
    "<img src='https://www.robocup.org/system/leagues/images/000/000/001/list/soccer.png?1461148054' width=800/>\n",
    "\n",
    "In the multiTaxi scenario, for example, implicit communication might take the form of the initial route each taxi takes (assuming that all agents are aware of each other's movements), but explicit communication can take the form of each taxi stating which passenger they want to pick up. There are other more explicit and implicit communication methods that might be used.\n",
    "\n",
    "The agents in the last example employ **explicit communication** to relay a message to the other agents. They specify which passenger they are going to in the message. We have completed the first three steps of the communication cycle, which means that the agents still do not know how to use communication to their benefit, nor do they know when to communicate.\n",
    "\n",
    "In the following example - we use a simple rule-based (grounded communication rule) in order to 'wisely-use' other taxis messages: if someones has a shortest path to handle your target passenger, switch to other passenger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3f6a637c-faeb-45f3-a7f9-6d35e44da87b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EnvironmentWrapper created\n",
      "send_recieve - Joint message: [(None, 0), (None, 0), (None, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| :\u001b[33;1mP\u001b[0m| : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m:\u001b[41m_\u001b[0m:\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : :\u001b[47m_\u001b[0m: : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "|\u001b[43m_\u001b[0m: : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,3), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (3,2), Collided: False\n",
      "Passenger1: Location: (1, 1), Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 1---------------------------------\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 10 -> active:['(1,)', '(1,)', '(1,)', '(2,)', '(4,)', '(0,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 10), ('Red', 23), ('White', 17), ('Green', 14), ('Cyan', 24)]\u001b[30m\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 9 -> active:['(3,)', '(3,)', '(1,)', '(4,)', '(0,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 9), ('Red', 18), ('White', 14), ('Green', 13), ('Cyan', 19)]\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 9 -> active:['(1,)', '(3,)', '(1,)', '(4,)', '(0,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 9), ('Red', 20), ('White', 16), ('Green', 13), ('Cyan', 21)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 4  0  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 2  3  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_2 obs:\n",
      " [ 3  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 1\n",
      "joint Message: [(1, 9), (1, 8), (1, 8)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| :\u001b[33;1mP\u001b[0m| : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[47m\u001b[41m\u001b[36;1mD\u001b[0m\u001b[0m\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "|\u001b[43m_\u001b[0m: : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,2), Collided: False\n",
      "Passenger1: Location: (1, 1), Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 2---------------------------------\n",
      "\u001b[33mchanging plan acording to message\u001b[30m\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[32mGreen passenger\u001b[30m, len. of plan: 15 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(4,)', '(1,)', '(8,)']\n",
      "  # all plans:[('Red', 22), ('White', 18), ('Green', 15), ('Cyan', 23)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 3  0  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_2 obs:\n",
      " [ 2  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(4, 14), (1, 7), (1, 7)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| :\u001b[33;1mP\u001b[0m| : : : | : | : | : |\n",
      "| :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m:\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| :\u001b[43m_\u001b[0m: : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,1), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,1), Collided: False\n",
      "Passenger1: Location: (1, 1), Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 3---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 1\n",
      "taxi_2 obs:\n",
      " [ 2  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 1\n",
      "joint Message: [(4, 13), (1, 6), (1, 6)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| :\u001b[47m\u001b[41m\u001b[33;1mP\u001b[0m\u001b[0m\u001b[0m| : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : :\u001b[43m_\u001b[0m: : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,1), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,1), Collided: False\n",
      "Passenger1: Location: (1, 1), Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 4---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 1  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  2  2\n",
      "  2  2  2] , action: 4\n",
      "joint Message: [(4, 12), (1, 5), (1, 5)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| :\u001b[47m\u001b[41m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m| : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : :\u001b[43m_\u001b[0m: : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,3), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,1), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,1), Collided: False\n",
      "Passenger1: Location: Taxi3, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [5, 2, 2, 2, 2]\n",
      "---------------------------ROUND 5---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  3  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 0\n",
      "joint Message: [(4, 11), (1, 4), (1, 4)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| :\u001b[47m\u001b[41m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m:\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : :\u001b[43m_\u001b[0m: | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,4), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,1), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,1), Collided: False\n",
      "Passenger1: Location: Taxi3, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [5, 2, 2, 2, 2]\n",
      "---------------------------ROUND 6---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  4  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  1  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 10), (1, 3), (1, 3)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[47m\u001b[41m\u001b[36;1mD\u001b[0m\u001b[0m\u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : :\u001b[43m_\u001b[0m| : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,2), Collided: False\n",
      "Passenger1: Location: Taxi3, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [5, 2, 2, 2, 2]\n",
      "---------------------------ROUND 7---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  5  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 0\n",
      "taxi_1 obs:\n",
      " [ 2  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  2  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 9), (1, 2), (1, 2)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m:\u001b[47m\u001b[41m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m:\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : :\u001b[43m_\u001b[0m| : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,3), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,3), Collided: False\n",
      "Passenger1: Location: Taxi3, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [5, 2, 2, 2, 2]\n",
      "---------------------------ROUND 8---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  5  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 0\n",
      "taxi_1 obs:\n",
      " [ 2  3  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  3  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 8), (1, 1), (1, 1)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[47m\u001b[41m\u001b[33;1mD\u001b[0m\u001b[0m\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : :\u001b[43m_\u001b[0m: : : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,4), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,4), Collided: False\n",
      "Passenger1: Location: Taxi3, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [5, 2, 2, 2, 2]\n",
      "---------------------------ROUND 9---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  5  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  4  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  4  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  5  2\n",
      "  2  2  2] , action: 5\n",
      "joint Message: [(4, 7), (1, 0), (1, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m\u001b[47m\u001b[41m_\u001b[0m\u001b[0m\u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : :\u001b[43m_\u001b[0m: : : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,4), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,4), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 2, 2]\n",
      "---------------------------ROUND 10---------------------------------\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[32mGreen passenger\u001b[30m, len. of plan: 12 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(0,)', '(4,)', '(1,)', '(8,)']\n",
      "  # all plans:[('Red', 17), ('White', 13), ('Green', 12), ('Cyan', 18)]\u001b[30m\n",
      "\u001b[31mchanging plan acording to message\u001b[30m\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[37mWhite passenger\u001b[30m, len. of plan: 13 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(1,)', '(2,)', '(2,)', '(2,)', '(7,)']\n",
      "  # all plans:[('Red', 17), ('White', 13), ('Cyan', 18)]\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[32mGreen passenger\u001b[30m, len. of plan: 12 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(0,)', '(4,)', '(1,)', '(8,)']\n",
      "  # all plans:[('Red', 17), ('White', 13), ('Green', 12), ('Cyan', 18)]\u001b[30m\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[37mWhite passenger\u001b[30m, len. of plan: 13 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(1,)', '(2,)', '(2,)', '(2,)', '(7,)']\n",
      "  # all plans:[('Red', 17), ('White', 13), ('Cyan', 18)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 5  6  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  4  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  4  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 6), (3, 12), (3, 12)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m:\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : :\u001b[43m_\u001b[0m: : :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,7), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,5), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,5), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 2, 2]\n",
      "---------------------------ROUND 11---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  7  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  5  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  5  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 5), (3, 11), (3, 11)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : :\u001b[43m_\u001b[0m: :\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,6), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,6), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 2, 2]\n",
      "---------------------------ROUND 12---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  6  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  6  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 4), (3, 10), (3, 10)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : :\u001b[43m_\u001b[0m:\u001b[32;1mP\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,9), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,7), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,7), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 2, 2]\n",
      "---------------------------ROUND 13---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  9  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  7  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  7  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 2\n",
      "joint Message: [(4, 3), (3, 9), (3, 9)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[43m\u001b[32;1mP\u001b[0m\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: (5, 10), Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 2, 2]\n",
      "---------------------------ROUND 14---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5 10  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 4\n",
      "taxi_1 obs:\n",
      " [ 2  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 2  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  2  2] , action: 0\n",
      "joint Message: [(4, 2), (3, 8), (3, 8)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m:\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[37;1mP\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (3,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: Taxi1, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 3, 2]\n",
      "---------------------------ROUND 15---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5 10  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 3  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 3  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 0\n",
      "joint Message: [(4, 1), (3, 7), (3, 7)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[47m\u001b[41m\u001b[37;1mP\u001b[0m\u001b[0m\u001b[0m: :\u001b[32m\u001b[43m\u001b[32;1mD\u001b[0m\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (4,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: (4, 8), Destination: (2, 11)\n",
      "Passenger4: Location: Taxi1, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 2, 3, 2]\n",
      "---------------------------ROUND 16---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4 10  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 8\n",
      "taxi_1 obs:\n",
      " [ 4  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 4  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  2  3  2] , action: 4\n",
      "joint Message: [(4, 0), (3, 6), (3, 6)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[31;1mP\u001b[0m: : |\n",
      "| : : : : : | : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m: :\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (4,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 4, 1, 2]\n",
      "---------------------------ROUND 17---------------------------------\n",
      "agent_0: no available plan found for passenger 2\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[31mRed passenger\u001b[30m, len. of plan: 13 -> active:['(1,)', '(3,)', '(4,)', '(3,)', '(3,)', '(3,)', '(0,)', '(0,)', '(3,)', '(3,)', '(3,)', '(0,)', '(6,)']\n",
      "  # all plans:[('Red', 13), ('Cyan', 16)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 4 10  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 4  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 1\n",
      "taxi_2 obs:\n",
      " [ 4  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 1\n",
      "joint Message: [(2, 12), (3, 5), (3, 5)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m:\u001b[31;1mP\u001b[0m:\u001b[43m_\u001b[0m: |\n",
      "| : : : : : | : : : :\u001b[32m \u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (3,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 4, 1, 2]\n",
      "---------------------------ROUND 18---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3 10  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 3  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 1\n",
      "taxi_2 obs:\n",
      " [ 3  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 1\n",
      "joint Message: [(2, 11), (3, 4), (3, 4)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m: : :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[43m\u001b[31;1mP\u001b[0m\u001b[0m: : |\n",
      "| : : : : : | : : : :\u001b[32m \u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,9), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: (3, 9), Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 2, 4, 1, 2]\n",
      "---------------------------ROUND 19---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  9  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 4\n",
      "taxi_1 obs:\n",
      " [ 2  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  8  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  2\n",
      "  4  1  2] , action: 2\n",
      "joint Message: [(2, 10), (3, 3), (3, 3)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m: :\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : |\n",
      "| : : : : : | : : : :\u001b[32m \u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,9), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,9), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,9), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: Taxi1, Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 3, 4, 1, 2]\n",
      "---------------------------ROUND 20---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  9  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  3\n",
      "  4  1  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 2  9  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  3\n",
      "  4  1  2] , action: 2\n",
      "taxi_2 obs:\n",
      " [ 2  9  1  1  3  9  4  8  5 10  0  9  2  4  6  3  2 11  4 10  2  2  1  3\n",
      "  4  1  2] , action: 2\n",
      "joint Message: [(2, 9), (3, 2), (3, 2)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m:\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : |\n",
      "| : : : : : | : : : :\u001b[32m \u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: Taxi1, Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 3, 4, 1, 2]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | :\u001b[36;1mP\u001b[0m|F: |\n",
      "| : | : : : | : | : | : |\n",
      "| : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : : : :\u001b[47m\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m\u001b[0m:\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : |\n",
      "| : : : : : | : : : :\u001b[32m \u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m| | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (2, 4)\n",
      "Passenger2: Location: Taxi1, Destination: (6, 3)\n",
      "Passenger3: Location: Taxi2, Destination: (2, 11)\n",
      "Passenger4: Location: Arrived!, Destination: (4, 10)\n",
      "Passenger5: Location: (0, 9), Destination: (2, 2)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [1, 3, 4, 1, 2]\n",
      "Thats all - part 2\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "part 2 - \n",
    "com_emerge_use of message for getting better + introducing com-module\n",
    "\"\"\"\n",
    "from src.decision_makers.planners.MA_com_planner import Astar_message_DM\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "Builds Multi_taxi env\n",
    "\"\"\"\n",
    "m = MAP\n",
    "env = MultiTaxiEnv(num_taxis=3, num_passengers=5, domain_map=m, observation_type='symbolic',rewards_table=TAXI_pickup_dropoff_REWARDS ,option_to_stand_by=True)\n",
    "obs = env.reset()\n",
    "\n",
    "environment = build_multi_env(env)\n",
    "#\n",
    "print('EnvironmentWrapper created')\n",
    "\n",
    "# making agent class that communicates and heads towards 1 passenger (pickup->dropoff)\n",
    "\"\"\"\n",
    "in order to use com module: \n",
    "    - implement Agent_com class (inherit from Agent_com\n",
    "    - make sure to implement set_data_func - that decides what is the data that the agent will transmit whenever it called\n",
    "    - u can implement your recieve_func - that decides what to do with a recieved data\n",
    "        defualt is the union_func that add the message data to the observation \n",
    "\"\"\"\n",
    "\n",
    "class Heading_message_agent(Agent_Com):\n",
    "\n",
    "    def __init__(self, decision_maker : Astar_message_DM , sensor_function =None, message_filter = None, AgentName = None, bandW = math.inf, union_recieve = False):\n",
    "        super().__init__(decision_maker , sensor_function, message_filter, AgentName, bandW, union_recieve)\n",
    "        self.last_action = None\n",
    "        self.last_message = None\n",
    "\n",
    "    def set_data_func(self, obs):\n",
    "        data = (self.decision_maker.taking_passenger,len(self.decision_maker.active_plan))\n",
    "        return data\n",
    "\n",
    "    # implement our recive_func\n",
    "    def set_recive_func(self, obs, message):\n",
    "        self.last_message = message\n",
    "        self.decision_maker.save_last_message(message)\n",
    "        # self.decision_maker.updateplan_message(message)\n",
    "\n",
    "    # saves last action of the agent - not necessary for com module\n",
    "    def set_last_action(self, action):\n",
    "        self.last_action = action\n",
    "#\n",
    "#\n",
    "\n",
    "\n",
    "# after having our com-Astar-agents class we can set our env agents into a dicentralized_agents dict\n",
    "env_agents = environment.get_env_agents()\n",
    "decentralized_agents = {agent_name: Heading_message_agent(Astar_message_DM(env ,single_plan=True, Taxi_index=int(agent_name[-1]), domain_map=m) ,AgentName=agent_name)             # Agent(LearningDecisionMaker(env.action_space))  # can use diffrent DM\n",
    "                        for agent_name in env_agents}\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "Simple use of communication network - \n",
    "    - build one using COM_net() - defualt architecture is - all masseges sent to all other agents\n",
    "            *U can use more options - see at COM_net() class doc.\n",
    "\"\"\"\n",
    "com = COM_net()\n",
    "\"\"\" \n",
    "    - initailze our new controller (DecentralizedComController) - using our env, our agents and our com module\n",
    "        - this controller will perform all joint action and message delieveries at any time-step\n",
    "\"\"\"\n",
    "controller = DecentralizedComController(environment, decentralized_agents, com)\n",
    "\"\"\"\n",
    "activate \n",
    "\"\"\"\n",
    "#communicate first\n",
    "controller.send_recieve()\n",
    "\n",
    "#run (communication inside after each time_click)\n",
    "controller.run(render=True, max_iteration=20,reset=True)\n",
    "print(\"Thats all - part 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d72a76c-44b1-41a4-8611-8410c0c7cf11",
   "metadata": {},
   "source": [
    "## Using Communication to Our Advantage\n",
    "\n",
    "Having established that the agents can send and receive messages, they must now learn how to utilize them and what to do with them. They are pointless if they do not behave in line with the massages. In emergent communication we can learn how to do it autonomously"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7bdcd3f0-a301-4e19-abbb-fd2e85ec0fdc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EnvironmentWrapper Multi_Taxi_Task_Wrapper created\n",
      "send_recieve - Joint message: [(None, 0), (None, 0), (None, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[37;1mP\u001b[0m: : : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : :\u001b[47m_\u001b[0m| : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| :\u001b[43m_\u001b[0m: : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| |\u001b[41m_\u001b[0m| :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (6,4), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (3,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: (2, 2), Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 1---------------------------------\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[37mWhite passenger\u001b[30m, len. of plan: 7 -> active:['(1,)', '(1,)', '(1,)', '(2,)', '(4,)', '(1,)', '(7,)']\n",
      "  # all plans:[('Yellow', 18), ('Red', 19), ('White', 7), ('Green', 24), ('Cyan', 16)]\u001b[30m\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[37mWhite passenger\u001b[30m, len. of plan: 9 -> active:['(1,)', '(1,)', '(1,)', '(1,)', '(3,)', '(3,)', '(4,)', '(1,)', '(7,)']\n",
      "  # all plans:[('Yellow', 22), ('Red', 17), ('White', 9), ('Green', 22), ('Cyan', 14)]\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[37mWhite passenger\u001b[30m, len. of plan: 7 -> active:['(1,)', '(3,)', '(3,)', '(3,)', '(4,)', '(1,)', '(7,)']\n",
      "  # all plans:[('Yellow', 24), ('Red', 17), ('White', 7), ('Green', 18), ('Cyan', 14)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 5  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 6  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_2 obs:\n",
      " [ 3  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(3, 6), (3, 8), (3, 6)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[37;1mP\u001b[0m: : :\u001b[47m_\u001b[0m: : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| :\u001b[43m_\u001b[0m: : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : :\u001b[41m_\u001b[0m: : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (5,4), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: (2, 2), Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 2---------------------------------\n",
      "\u001b[31mchanging plan acording to message\u001b[30m\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[36mCyan passenger\u001b[30m, len. of plan: 13 -> active:['(2,)', '(2,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(4,)', '(1,)', '(1,)', '(3,)', '(3,)', '(9,)']\n",
      "  # all plans:[('Yellow', 21), ('Red', 16), ('Green', 21), ('Cyan', 13)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 4  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 5  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(3, 5), (5, 12), (3, 5)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[37;1mP\u001b[0m: :\u001b[47m_\u001b[0m: : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| :\u001b[43m_\u001b[0m:\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : :\u001b[41m_\u001b[0m: : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (5,5), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,4), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: (2, 2), Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 3---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 5  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(3, 4), (5, 11), (3, 4)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| :\u001b[43m_\u001b[0m:\u001b[37;1mP\u001b[0m:\u001b[47m_\u001b[0m: : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : : :\u001b[41m_\u001b[0m: : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (5,6), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,3), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: (2, 2), Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 4---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 5  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  3  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(3, 3), (5, 10), (3, 3)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[47m\u001b[43m\u001b[37;1mP\u001b[0m\u001b[0m\u001b[0m: : : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : |\u001b[41m_\u001b[0m: :\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,6), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,2), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: (2, 2), Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 2, 2, 2]\n",
      "---------------------------ROUND 5---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 4  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  2  2  2] , action: 3\n",
      "joint Message: [(3, 2), (5, 9), (3, 2)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[37;1mD\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[47m\u001b[43m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m: : : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | :\u001b[41m_\u001b[0m:\u001b[31;1mP\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,7), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,2), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Taxi3, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 5, 2, 2]\n",
      "---------------------------ROUND 6---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 4  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 3\n",
      "joint Message: [(3, 1), (5, 8), (3, 1)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[47m\u001b[37;1mD\u001b[0m\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[43m_\u001b[0m: : : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[41m\u001b[31;1mP\u001b[0m\u001b[0m: :\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,2), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Taxi3, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 5, 2, 2]\n",
      "---------------------------ROUND 7---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 3\n",
      "taxi_1 obs:\n",
      " [ 4  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 1  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  5  2  2] , action: 3\n",
      "joint Message: [(3, 0), (5, 7), (3, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m\u001b[47m_\u001b[0m\u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : :\u001b[43m_\u001b[0m: : : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m:\u001b[41m_\u001b[0m:\u001b[36;1mP\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,9), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,2), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 2]\n",
      "---------------------------ROUND 8---------------------------------\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[36mCyan passenger\u001b[30m, len. of plan: 16 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(1,)', '(3,)', '(3,)', '(9,)']\n",
      "  # all plans:[('Yellow', 22), ('Red', 19), ('Green', 20), ('Cyan', 16)]\u001b[30m\n",
      "\u001b[33mchanging plan acording to message\u001b[30m\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[31mRed passenger\u001b[30m, len. of plan: 19 -> active:['(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(3,)', '(3,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(6,)']\n",
      "  # all plans:[('Yellow', 22), ('Red', 19), ('Green', 20)]\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[36mCyan passenger\u001b[30m, len. of plan: 17 -> active:['(2,)', '(2,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(1,)', '(3,)', '(3,)', '(9,)']\n",
      "  # all plans:[('Yellow', 23), ('Red', 20), ('Green', 21), ('Cyan', 17)]\u001b[30m\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[31mRed passenger\u001b[30m, len. of plan: 20 -> active:['(2,)', '(2,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(3,)', '(3,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(6,)']\n",
      "  # all plans:[('Yellow', 23), ('Red', 20), ('Green', 21)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 4  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 1  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 2\n",
      "joint Message: [(2, 18), (5, 6), (2, 19)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m:\u001b[47m_\u001b[0m: :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : :\u001b[43m_\u001b[0m: : : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[41m\u001b[36;1mP\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,3), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,3), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: (4, 10), Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 2]\n",
      "---------------------------ROUND 9---------------------------------\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[36mCyan passenger\u001b[30m, len. of plan: 16 -> active:['(2,)', '(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(1,)', '(3,)', '(3,)', '(9,)']\n",
      "  # all plans:[('Yellow', 24), ('Green', 20), ('Cyan', 16)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 2  3  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 4 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 1  3  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  2] , action: 5\n",
      "joint Message: [(2, 17), (5, 5), (5, 15)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: :\u001b[47m_\u001b[0m:\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : :\u001b[43m_\u001b[0m: : : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,4), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (4,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,4), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Taxi2, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 4]\n",
      "---------------------------ROUND 10---------------------------------\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[31mRed passenger\u001b[30m, len. of plan: 18 -> active:['(2,)', '(0,)', '(2,)', '(2,)', '(2,)', '(0,)', '(0,)', '(4,)', '(1,)', '(3,)', '(3,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(6,)']\n",
      "  # all plans:[('Yellow', 25), ('Red', 18), ('Green', 19)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 2  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 4 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 1  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "joint Message: [(2, 16), (5, 4), (2, 17)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m\u001b[32;1mD\u001b[0m\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : :\u001b[43m_\u001b[0m: : :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m:\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (3,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Taxi2, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 4]\n",
      "---------------------------ROUND 11---------------------------------\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "agent_2: no available plan found for passenger 4\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[32mGreen passenger\u001b[30m, len. of plan: 18 -> active:['(0,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(1,)', '(1,)', '(4,)', '(0,)', '(0,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(8,)']\n",
      "  # all plans:[('Yellow', 26), ('Green', 18)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 2  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 3 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 4\n",
      "joint Message: [(2, 15), (5, 3), (4, 17)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : :\u001b[47m_\u001b[0m:\u001b[43m_\u001b[0m: :\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m: :\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Taxi2, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 4]\n",
      "---------------------------ROUND 12---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 4\n",
      "joint Message: [(2, 14), (5, 2), (4, 16)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : :\u001b[47m_\u001b[0m:\u001b[43m_\u001b[0m:\u001b[32m\u001b[36;1mD\u001b[0m\u001b[0m:\u001b[41m\u001b[41;1m \u001b[0m\u001b[0m: : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,7), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,9), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,6), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Taxi2, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 4]\n",
      "---------------------------ROUND 13---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 4\n",
      "joint Message: [(2, 13), (5, 1), (4, 15)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : :\u001b[47m_\u001b[0m:\u001b[32m\u001b[41m\u001b[43m\u001b[36;1mD\u001b[0m\u001b[0m\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,7), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Taxi2, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 4]\n",
      "---------------------------ROUND 14---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 5\n",
      "taxi_2 obs:\n",
      " [ 2  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  4] , action: 4\n",
      "joint Message: [(2, 12), (5, 0), (4, 14)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : : :\u001b[32m\u001b[47m\u001b[41m_\u001b[0m\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : :\u001b[43m_\u001b[0m: : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[31;1mP\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,8), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 1]\n",
      "---------------------------ROUND 15---------------------------------\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[31mRed passenger\u001b[30m, len. of plan: 13 -> active:['(0,)', '(0,)', '(4,)', '(1,)', '(3,)', '(3,)', '(1,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(6,)']\n",
      "  # all plans:[('Yellow', 28), ('Red', 13), ('Green', 14)]\u001b[30m\n",
      "\u001b[31mchanging plan acording to message\u001b[30m\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[32mGreen passenger\u001b[30m, len. of plan: 14 -> active:['(2,)', '(2,)', '(1,)', '(1,)', '(4,)', '(0,)', '(0,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(1,)', '(8,)']\n",
      "  # all plans:[('Yellow', 28), ('Green', 14)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 3  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 4\n",
      "joint Message: [(2, 11), (4, 13), (4, 13)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m:\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[43m\u001b[31;1mP\u001b[0m\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,9), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,9), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: (4, 8), Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 2, 1, 2, 1]\n",
      "---------------------------ROUND 16---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  2\n",
      "  1  2  1] , action: 4\n",
      "joint Message: [(2, 10), (4, 12), (4, 12)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: :\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (2,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 2, 1]\n",
      "---------------------------ROUND 17---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 2 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "joint Message: [(2, 9), (4, 11), (4, 11)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[32;1mP\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[47m\u001b[41m_\u001b[0m\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 2, 1]\n",
      "---------------------------ROUND 18---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "joint Message: [(2, 8), (4, 10), (4, 10)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[47m\u001b[41m\u001b[32;1mP\u001b[0m\u001b[0m\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,7), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (0,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (0,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: (0, 10), Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 2, 1]\n",
      "---------------------------ROUND 19---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 0 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 0 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  2  1] , action: 4\n",
      "joint Message: [(2, 7), (4, 9), (4, 9)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |\u001b[47m\u001b[41m\u001b[47;1mF\u001b[0m\u001b[0m\u001b[0m: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : | : |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : |\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (0,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (0,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 20---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 0 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 0 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 6), (4, 8), (4, 8)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[47m\u001b[41m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m: |\n",
      "| : : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 21---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 5), (4, 7), (4, 7)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : :\u001b[32m \u001b[0m: :\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m: |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,10), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 22---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 4), (4, 6), (4, 6)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[32m \u001b[0m:\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m: : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,4), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,9), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 23---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 3), (4, 5), (4, 5)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : :\u001b[32m\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m\u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,3), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,8), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 24---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  3  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 2), (4, 4), (4, 4)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : :\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m:\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,7), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 25---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 2  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 1), (4, 3), (4, 3)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : :\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m: :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[43m\u001b[31;1mD\u001b[0m\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,6), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Taxi1, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 3, 1, 5, 1]\n",
      "---------------------------ROUND 26---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 2\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  3\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(2, 0), (4, 2), (4, 2)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[32;1mD\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : :\u001b[47m\u001b[47;1m \u001b[0m\u001b[0m: : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m\u001b[43m_\u001b[0m\u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (2,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 5, 1]\n",
      "---------------------------ROUND 27---------------------------------\n",
      "agent_0: no available plan found for passenger 3\n",
      "\u001b[33mYellow Taxi_0 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 21 -> active:['(3,)', '(3,)', '(0,)', '(0,)', '(0,)', '(4,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(1,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 21)]\u001b[30m\n",
      "taxi_0 obs:\n",
      " [ 3  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 2  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(1, 20), (4, 1), (4, 1)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m\u001b[32;1mD\u001b[0m\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| :\u001b[43m_\u001b[0m:\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Taxi3, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 5, 1]\n",
      "---------------------------ROUND 28---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 4\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  5  1] , action: 4\n",
      "joint Message: [(1, 19), (4, 0), (4, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "|\u001b[43m_\u001b[0m: :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 1, 1]\n",
      "---------------------------ROUND 29---------------------------------\n",
      "\u001b[31mRed Taxi_1 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 31 -> active:['(0,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(3,)', '(0,)', '(0,)', '(0,)', '(0,)', '(4,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(1,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 31)]\u001b[30m\n",
      "\u001b[31mchanging plan acording to message\u001b[30m\n",
      "agent_1: no available plan was found (for all)\n",
      "agent_1: no action available - using defualt\n",
      "\u001b[37mWhite Taxi_2 (re)planning results:\n",
      "  # taking \u001b[33mYellow passenger\u001b[30m, len. of plan: 26 -> active:['(3,)', '(3,)', '(3,)', '(0,)', '(3,)', '(3,)', '(0,)', '(0,)', '(0,)', '(0,)', '(4,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(1,)', '(1,)', '(2,)', '(2,)', '(2,)', '(2,)', '(2,)', '(5,)']\n",
      "  # all plans:[('Yellow', 26)]\u001b[30m\n",
      "\u001b[37mchanging plan acording to message\u001b[30m\n",
      "agent_2: no available plan was found (for all)\n",
      "agent_2: no action available - using defualt\n",
      "taxi_0 obs:\n",
      " [ 3  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 18), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "|\u001b[43m_\u001b[0m: : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 1, 1]\n",
      "---------------------------ROUND 30---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 17), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "|\u001b[43m_\u001b[0m: : : : : : : : : : : |\n",
      "|\u001b[33;1mP\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 1, 1]\n",
      "---------------------------ROUND 31---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 16), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[43m\u001b[33;1mP\u001b[0m\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (6,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: (6, 0), Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [2, 1, 1, 1, 1]\n",
      "---------------------------ROUND 32---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 6  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  2  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 15), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "|\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m| :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (6,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 33---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 6  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 14), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "|\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,0), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 34---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  0  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 13), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,1), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 35---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  1  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 12), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,2), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 36---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  2  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 11), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,3), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 37---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  3  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 10), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,4), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 38---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  4  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 9), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,5), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 39---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 8), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (5,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 40---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 5  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 7), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : |\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (4,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 41---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 4  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 6), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : |\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,6), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 42---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  6  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 5), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,7), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 43---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  7  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 4), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,8), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 44---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  8  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 3), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,9), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 45---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3  9  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 2), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m:\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,10), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 46---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 1), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[43m\u001b[33;1mD\u001b[0m\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,11), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: False, {'taxi_0': False, 'taxi_1': False, 'taxi_2': False, '__all__': False}\n",
      "Passengers Status's: [3, 1, 1, 1, 1]\n",
      "---------------------------ROUND 47---------------------------------\n",
      "taxi_0 obs:\n",
      " [ 3 11  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 1\n",
      "taxi_1 obs:\n",
      " [ 1 10  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "taxi_2 obs:\n",
      " [ 1  5  6  0  4  8  2  2  0 10  4 10  3 11  3  2  1  2  1  5  2  8  3  1\n",
      "  1  1  1] , action: 0\n",
      "joint Message: [(1, 0), (0, 0), (0, 0)]\n",
      "+-----------------------+\n",
      "| : |F: | : | : | : |F: |\n",
      "| : |\u001b[32m \u001b[0m: : :\u001b[32m\u001b[47m_\u001b[0m\u001b[0m| : | : |\u001b[41m_\u001b[0m: |\n",
      "| : : : : : : : :\u001b[32m \u001b[0m: : : |\n",
      "| : :\u001b[32m \u001b[0m: : : | : : : : :\u001b[32m\u001b[43m_\u001b[0m\u001b[0m|\n",
      "| : : : : : | : : : : : |\n",
      "| : : : : : : : : : : : |\n",
      "| | :G| | | :G| | | : | |\n",
      "+-----------------------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,11), Collided: False\n",
      "Taxi1-RED: Fuel: inf, Location: (1,10), Collided: False\n",
      "Taxi2-WHITE: Fuel: inf, Location: (1,5), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (3, 11)\n",
      "Passenger2: Location: Arrived!, Destination: (3, 2)\n",
      "Passenger3: Location: Arrived!, Destination: (1, 2)\n",
      "Passenger4: Location: Arrived!, Destination: (1, 5)\n",
      "Passenger5: Location: Arrived!, Destination: (2, 8)\n",
      "Done: True, {'taxi_0': True, 'taxi_1': True, 'taxi_2': True, '__all__': True}\n",
      "Passengers Status's: [1, 1, 1, 1, 1]\n",
      "----------------------------------\n",
      " total reward of all agents: 657, {'taxi_0': 257, 'taxi_1': 124, 'taxi_2': 276} \n",
      "----------------------------------\n",
      "Thats all - part 2.2\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "part 2.2 - \n",
    "agents learn now - hierarchical_tasks\n",
    "\"\"\"\n",
    "\n",
    "# import hierarchical_tasks multi taxi env\n",
    "from src.environments.hirarchical_Wrapper import Multi_Taxi_Task_Wrapper\n",
    "from src.decision_makers.planners.Com_High_level_Planner import Astar_message_highlevel_DM\n",
    "\"\"\"\n",
    "Builds Multi_taxi env\n",
    "\"\"\"\n",
    "m = MAP\n",
    "env = MultiTaxiEnv(num_taxis=3, num_passengers=5, domain_map=m, observation_type='symbolic',rewards_table=TAXI_pickup_dropoff_REWARDS ,option_to_stand_by=True)\n",
    "\n",
    "obs = env.reset()\n",
    "\n",
    "environment = build_multi_env(env)\n",
    "\n",
    "environment =  Multi_Taxi_Task_Wrapper(environment)\n",
    "#\n",
    "print('EnvironmentWrapper Multi_Taxi_Task_Wrapper created')\n",
    "\n",
    "# making agent class that communicates and heads towards 1 passenger (pickup->dropoff)\n",
    "\n",
    "class Heading_message_agent(Agent_Com):\n",
    "\n",
    "    def __init__(self, decision_maker : Astar_message_highlevel_DM , sensor_function =None, message_filter = None, AgentName = None, bandW = math.inf, union_recieve = False):\n",
    "        super().__init__(decision_maker , sensor_function, message_filter, AgentName, bandW, union_recieve)\n",
    "        self.last_action = None\n",
    "        self.last_message = None\n",
    "\n",
    "    def set_data_func(self, obs):\n",
    "        data = (self.decision_maker.taking_passenger,len(self.decision_maker.active_plan))\n",
    "        return data\n",
    "\n",
    "    # implement our recive_func\n",
    "    def set_recive_func(self, obs, message):\n",
    "        self.last_message = message\n",
    "        self.decision_maker.save_last_message(message)\n",
    "        # self.decision_maker.updateplan_message(message)\n",
    "\n",
    "    # saves last action of the agent - not necessary for com module\n",
    "    def set_last_action(self, action):\n",
    "        self.last_action = action\n",
    "\n",
    "\n",
    "# run again - high-level\n",
    "env_agents = environment.get_env_agents()\n",
    "decentralized_agents = {agent_name: Heading_message_agent(Astar_message_highlevel_DM(env ,single_plan=True, Taxi_index=int(agent_name[-1]), domain_map=m) ,AgentName=agent_name)             # Agent(LearningDecisionMaker(env.action_space))  # can use diffrent DM\n",
    "                        for agent_name in env_agents}\n",
    "\n",
    "controller = DecentralizedComController(environment, decentralized_agents, com)\n",
    "\"\"\"\n",
    "activate \n",
    "\"\"\"\n",
    "#communicate first\n",
    "controller.send_recieve()\n",
    "\n",
    "#run (communication inside after each time_click)\n",
    "controller.run(render=True, max_iteration=250,reset=True)\n",
    "\n",
    "# SHOW REWARDS\n",
    "reward = controller.total_rewards\n",
    "totals = {}\n",
    "for r in reward[0]:\n",
    "    totals[r] = 0\n",
    "total=0\n",
    "for r in reward:\n",
    "    for key, value in r.items():\n",
    "        totals[key]+=value\n",
    "        total+=value\n",
    "print(f\"----------------------------------\\n total reward of all agents: {total}, {totals} \\n----------------------------------\")\n",
    "print(\"Thats all - part 2.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cfc77e9-1691-4b25-98f6-28d307f45dad",
   "metadata": {},
   "source": [
    "We can see a major performence improvement - this method can easily solve a wide, Multi-Agent domain, using low-bandwidth com.\n",
    "\n",
    "After we thought about the solution of the 5th and final step ('Interpretation') by ourself, We can design a RL Agent that can learn how to interpret and act by it own. \n",
    "#### PPO RL agent learn how to act in the high-level action space (X - handling passenger x, 0 - stby)\n",
    "\n",
    "we show how high-level single agent can learn how to act.\n",
    "further research - learning what to communicate (cooperative/adverserial), and learning how to interpret-act acorrding to other agents 'signals'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e0f8f3f4-e90b-4300-8434-7b4bc22ae32f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+\n",
      "| :\u001b[43m\u001b[33;1mP\u001b[0m\u001b[0m|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[31;1mP\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,1), Collided: False\n",
      "Passenger1: Location: (0, 1), Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [2, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 1\n",
      "+-------+\n",
      "| :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m|\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m\u001b[33;1mD\u001b[0m\u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[31;1mP\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,1), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [3, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 1\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m\u001b[43m\u001b[33;1mD\u001b[0m\u001b[0m\u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[31;1mP\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (1,1), Collided: False\n",
      "Passenger1: Location: Taxi1, Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [3, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 1\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m\u001b[43m_\u001b[0m\u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[31;1mP\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (1,1), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| :\u001b[43m_\u001b[0m: : |\n",
      "| |\u001b[31;1mP\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,1), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[43m\u001b[31;1mP\u001b[0m\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,1), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: (3, 1), Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 2]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| : : : |\n",
      "| |\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m:G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (3,1), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 3]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: : |\n",
      "| | :G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,1), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 3]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| : :\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: |\n",
      "| | :G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (2,2), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 3]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[31;1mD\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m|\u001b[43m\u001b[43;1m \u001b[0m\u001b[0m: |\n",
      "| : : : |\n",
      "| | :G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (1,2), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 3]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[43m\u001b[31;1mD\u001b[0m\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| : : : |\n",
      "| | :G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,2), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Taxi1, Destination: (0, 2)\n",
      "Done: False, {'taxi_0': False, '__all__': False}\n",
      "Passengers Status's: [1, 3]\n",
      "obs:<class 'numpy.ndarray'>\n",
      "next action: handling pass 2\n",
      "+-------+\n",
      "| : |\u001b[32m\u001b[43mF\u001b[0m\u001b[0m: |\n",
      "| :\u001b[32m \u001b[0m| : |\n",
      "| : : : |\n",
      "| | :G| |\n",
      "+-------+\n",
      "Taxi0-YELLOW: Fuel: inf, Location: (0,2), Collided: False\n",
      "Passenger1: Location: Arrived!, Destination: (1, 1)\n",
      "Passenger2: Location: Arrived!, Destination: (0, 2)\n",
      "Done: True, {'taxi_0': True, '__all__': True}\n",
      "Passengers Status's: [1, 1]\n",
      "that all - part 3\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "part 3 - \n",
    "agents learn now - high level PPO, low-level a-star\n",
    "\"\"\"\n",
    "\n",
    "from src.decision_makers.High_level_learner import LearningHighLevelDecisionMaker\n",
    "\n",
    "env = MultiTaxiEnv(num_taxis=1, num_passengers=2, domain_map=MAP2, observation_type='symbolic', option_to_stand_by=True)\n",
    "env.agents = env.taxis_names\n",
    "env = EnvWrappper(env, env.agents)\n",
    "env = Multi_Taxi_Task_Wrapper(env)\n",
    "obs = env.reset()\n",
    "\n",
    "D_M = LearningHighLevelDecisionMaker(env.action_space)\n",
    "obs = env.reset()\n",
    "env.env.env.render()\n",
    "for i in range(50):\n",
    "    print(f\"obs:{type(obs)}\")\n",
    "    a = D_M.get_action(obs)\n",
    "    print(f\"next action: {env.index_action_dictionary[a]}\")\n",
    "    obs, r, done, info = env.step(a)\n",
    "    env.env.env.render()\n",
    "    if done: break\n",
    "print(\"that all - part 3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad3e7964-aebe-45d4-a77d-09ba8b724a78",
   "metadata": {},
   "source": [
    "## Compilling Communication for RL as an Action + Com-Reward \n",
    "\n",
    "To explain implicit communication we will look at the Speaker Lister environment. In this environment there are 2 agents in which one agent (speaker) has information about the goal and has a limmited mode of communication with the second agent (listener) must use the speaker's communications and its limitted observations to navigate a 2D space toward the goal. The speaker agent cannot navigate, and the listener object cannot communicate.\n",
    "\n",
    "We used this setting to implement our emergent communication in the form of deciding when to 'communicate' - SPEAKER will allways 'point' on the right target, Listener should learn when to use the action to observe the Speaker's pointer (or in other wards, Listener learns when to ask for help)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f603b4fd-c4e9-48a1-87f4-51067f92d2d6",
   "metadata": {},
   "source": [
    "\n",
    "#### speaker-listener com as action - env-wrapper\n",
    "#### speaker - 'point-to-target' if asked, listener can ask for 'help' from speaker (action 0 - no move)\n",
    "<img src='images/seaker-point.jpg' width=\"500\" height=\"500\">\n",
    "\n",
    "##### Listener new-wrapper :\n",
    "The speaker observation is of type `Box(-inf, inf, (9,), float32)` \n",
    "1. listener agent velocity X\n",
    "2. listener agent velocity Y\n",
    "3. red landmark X pos - listener agent X pos\n",
    "4. red landmark Y pos - listener agent Y pos\n",
    "5. blue landmark X pos - listener agent X pos\n",
    "6. blue landmark Y pos - listener agent Y pos\n",
    "7. green landmark X pos - listener agent X pos\n",
    "8. green landmark Y pos - listener agent Y pos\n",
    "9. communication channel - \n",
    "    if com action was choosen last - speaker 'points' at destination (1-8 - directions shown above). else (no com from SPEAKER) - 0\n",
    "##### Discrete ACTIONS:\n",
    "* 0 - do nothing and ask for communicate (com penalty)\n",
    "* 1 - push left (add velocity in negative x-axis direction)\n",
    "* 2 - push right (add velocity in positive x-axis direction)\n",
    "* 3 - push down (add velocity in negative y-axis direction)\n",
    "* 4 - push up (add velocity in positive t-axis direction)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "943a2941-e77b-42e9-8f8d-4e64ea6ab720",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "custom wrapped environment: <ListenerOnlyCOMWrappersimple_speaker_listener_v3>\n",
      "step 0\n",
      "observation: [ 0.          0.          1.0223006  -0.9109972  -0.5290924  -0.26840162\n",
      " -0.69068325  0.3932359   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.3015689154408136\n",
      "\n",
      "step 1\n",
      "observation: [-0.5         0.          1.0723007  -0.9109972  -0.47909236 -0.26840162\n",
      " -0.64068323  0.3932359   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.36704323920623083\n",
      "\n",
      "step 2\n",
      "observation: [-0.375       0.          1.1098007  -0.9109972  -0.44159234 -0.26840162\n",
      " -0.6031832   0.3932359   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.34299468515529385\n",
      "\n",
      "step 3\n",
      "observation: [-0.28125     0.          1.1379256  -0.9109972  -0.41346735 -0.26840162\n",
      " -0.5750582   0.3932359   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.26773383760434716\n",
      "\n",
      "step 4\n",
      "observation: [ 0.2890625   0.          1.1090194  -0.9109972  -0.4423736  -0.26840162\n",
      " -0.60396445  0.3932359   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.24347956052858805\n",
      "\n",
      "step 5\n",
      "observation: [-0.28320312  0.          1.1373397  -0.9109972  -0.4140533  -0.26840162\n",
      " -0.57564414  0.3932359   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.20200136719990605\n",
      "\n",
      "step 6\n",
      "observation: [-0.21240234 -0.5         1.15858    -0.8609972  -0.39281306 -0.21840163\n",
      " -0.5544039   0.44323587  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.19535626652428562\n",
      "\n",
      "step 7\n",
      "observation: [-0.15930176  0.125       1.1745101  -0.8734972  -0.37688288 -0.23090163\n",
      " -0.5384737   0.4307359   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.16938292977113267\n",
      "\n",
      "step 8\n",
      "observation: [-0.11947632 -0.40625     1.1864578  -0.8328722  -0.36493525 -0.19027664\n",
      " -0.5265261   0.4713609   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.17073720705708567\n",
      "\n",
      "step 9\n",
      "observation: [-0.08960724  0.1953125   1.1954185  -0.85240346 -0.35597453 -0.20980789\n",
      " -0.51756537  0.45182964  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.15241334876069654\n",
      "\n",
      "step 10\n",
      "observation: [-0.06720543 -0.35351562  1.202139   -0.8170519  -0.34925398 -0.17445631\n",
      " -0.5108448   0.4871812   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1280757448246315\n",
      "\n",
      "step 11\n",
      "observation: [-0.05040407 -0.7651367   1.2071794  -0.74053824 -0.34421358 -0.09794265\n",
      " -0.5058044   0.5636949   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.15408303730865383\n",
      "\n",
      "step 12\n",
      "observation: [ 0.46219695 -0.57385254  1.1609597  -0.683153   -0.39043325 -0.04055739\n",
      " -0.5520241   0.6210801   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.22572430275404795\n",
      "\n",
      "step 13\n",
      "observation: [ 0.84664774 -0.4303894   1.076295   -0.640114   -0.47509804  0.00248155\n",
      " -0.6366889   0.66411906  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2399349622334742\n",
      "\n",
      "step 14\n",
      "observation: [ 0.13498577 -0.32279205  1.0627964  -0.6078348  -0.48859662  0.03476075\n",
      " -0.6501875   0.69639826  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.24880264950095143\n",
      "\n",
      "step 15\n",
      "observation: [ 0.10123934  0.25790596  1.0526725  -0.6336254  -0.49872056  0.00897016\n",
      " -0.6603114   0.6706077   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20832960419570815\n",
      "\n",
      "step 16\n",
      "observation: [-0.4240705   0.19342947  1.0950795  -0.65296835 -0.4563135  -0.01037279\n",
      " -0.61790437  0.6512647   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.18083823619097913\n",
      "\n",
      "step 17\n",
      "observation: [-0.3180529  -0.3549279   1.1268848  -0.61747557 -0.4245082   0.02512\n",
      " -0.5860991   0.6867575   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1708747695242788\n",
      "\n",
      "step 18\n",
      "observation: [-0.23853965 -0.7661959   1.1507387  -0.540856   -0.40065426  0.10173959\n",
      " -0.56224513  0.76337713  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.271854109867627\n",
      "\n",
      "step 19\n",
      "observation: [-0.17890474 -0.57464695  1.1686292  -0.48339128 -0.38276377  0.15920429\n",
      " -0.5443546   0.8208418   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.27734283156828576\n",
      "\n",
      "step 20\n",
      "observation: [-0.13417855 -0.4309852   1.1820471  -0.44029278 -0.3693459   0.20230281\n",
      " -0.5309368   0.8639403   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2841336195930504\n",
      "\n",
      "step 21\n",
      "observation: [-0.10063392 -0.3232389   1.1921104  -0.40796888 -0.35928252  0.2346267\n",
      " -0.52087337  0.8962642   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.22840447251754284\n",
      "\n",
      "step 22\n",
      "observation: [ 0.42452458 -0.24242918  1.149658   -0.38372597 -0.40173498  0.25886962\n",
      " -0.5633258   0.92050713  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.23953921515306395\n",
      "\n",
      "step 23\n",
      "observation: [ 0.3183934   0.31817812  1.1178187  -0.41554376 -0.43357432  0.22705181\n",
      " -0.5951652   0.88868934  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.23273070603351326\n",
      "\n",
      "step 24\n",
      "observation: [ 0.23879507  0.7386336   1.0939392  -0.48940712 -0.45745382  0.15318844\n",
      " -0.6190447   0.81482595  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.22825439154626426\n",
      "\n",
      "step 25\n",
      "observation: [ 0.1790963   1.0539752   1.0760295  -0.59480464 -0.47536346  0.04779093\n",
      " -0.6369543   0.7094284   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.23927251112501075\n",
      "\n",
      "step 26\n",
      "observation: [ 0.13432223  0.2904814   1.0625973  -0.6238528  -0.48879567  0.01874279\n",
      " -0.6503866   0.6803803   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20149340331924048\n",
      "\n",
      "step 27\n",
      "observation: [-0.39925835  0.21786104  1.1025231  -0.6456389  -0.44886985 -0.00304332\n",
      " -0.6104607   0.6585942   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.22026679718073774\n",
      "\n",
      "step 28\n",
      "observation: [ 0.20055625  0.16339578  1.0824676  -0.6619785  -0.46892548 -0.0193829\n",
      " -0.63051635  0.6422546   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.24088893862462088\n",
      "\n",
      "step 29\n",
      "observation: [ 0.1504172   0.62254685  1.0674258  -0.72423315 -0.48396719 -0.08163758\n",
      " -0.64555806  0.5799999   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2147144388501882\n",
      "\n",
      "step 30\n",
      "observation: [-0.38718712  0.46691012  1.1061445  -0.77092415 -0.44524848 -0.12832859\n",
      " -0.60683936  0.5333089   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.21874718517735012\n",
      "\n",
      "step 31\n",
      "observation: [-0.29039034  0.8501826   1.1351836  -0.8559424  -0.41620946 -0.21334685\n",
      " -0.57780033  0.44829065  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.33236541793360075\n",
      "\n",
      "step 32\n",
      "observation: [-0.21779275  0.63763696  1.1569629  -0.9197061  -0.39443016 -0.27711055\n",
      " -0.55602103  0.38452697  8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.28353136541334356\n",
      "\n",
      "step 33\n",
      "observation: [-0.16334456  0.97822773  1.1732973  -1.0175289  -0.37809572 -0.37493333\n",
      " -0.53968656  0.28670418  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.3739002066835405\n",
      "\n",
      "step 34\n",
      "observation: [ 0.3774916   0.7336708   1.1355481  -1.090896   -0.4158449  -0.4483004\n",
      " -0.57743573  0.21333712  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.4027794089079376\n",
      "\n",
      "step 35\n",
      "observation: [ 0.2831187   0.05025309  1.1072363  -1.0959213  -0.44415674 -0.45332572\n",
      " -0.60574764  0.20831181  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.3823145373841803\n",
      "\n",
      "step 36\n",
      "observation: [ 0.21233901 -0.4623102   1.0860023  -1.0496902  -0.46539065 -0.4070947\n",
      " -0.6269815   0.25454283  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.33562073507663526\n",
      "\n",
      "step 37\n",
      "observation: [ 0.15925425 -0.8467326   1.070077   -0.965017   -0.48131606 -0.32242143\n",
      " -0.64290696  0.33921608  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.26351729401925966\n",
      "\n",
      "step 38\n",
      "observation: [-0.3805593  -0.63504946  1.1081328  -0.901512   -0.44326013 -0.25891647\n",
      " -0.604851    0.40272105  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.17766187996471874\n",
      "\n",
      "step 39\n",
      "observation: [-0.78541946 -0.4762871   1.1866748  -0.8538833  -0.3647182  -0.21128777\n",
      " -0.5263091   0.45034975  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.09626314400191413\n",
      "\n",
      "step 40\n",
      "observation: [-1.0890646  -0.35721534  1.2955812  -0.8181618  -0.25581172 -0.17556623\n",
      " -0.4174026   0.4860713   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15245593880454955\n",
      "\n",
      "step 41\n",
      "observation: [-0.81679845 -0.2679115   1.3772612  -0.79137063 -0.17413189 -0.14877509\n",
      " -0.33572274  0.51286244  8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04466724559606315\n",
      "\n",
      "step 42\n",
      "observation: [-0.61259884  0.29906636  1.438521   -0.82127726 -0.112872   -0.17868172\n",
      " -0.27446288  0.48295578  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.054117858891986434\n",
      "\n",
      "step 43\n",
      "observation: [ 0.04055087  0.22429979  1.4344659  -0.84370726 -0.11692709 -0.2011117\n",
      " -0.27851796  0.4605258   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.0763845656039267\n",
      "\n",
      "step 44\n",
      "observation: [ 0.53041315  0.16822484  1.3814245  -0.8605298  -0.16996841 -0.21793418\n",
      " -0.33155927  0.44370332  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07659348619949254\n",
      "\n",
      "step 45\n",
      "observation: [ 0.39780986 -0.37383136  1.3416436  -0.82314664 -0.20974939 -0.18055104\n",
      " -0.37134025  0.48108646  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09841282958088889\n",
      "\n",
      "step 46\n",
      "observation: [ 0.2983574   0.21962647  1.3118079  -0.8451093  -0.23958513 -0.2025137\n",
      " -0.401176    0.45912382  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2165787817332148\n",
      "\n",
      "step 47\n",
      "observation: [ 0.22376806  0.16471985  1.2894311  -0.86158127 -0.26196194 -0.21898568\n",
      " -0.4235528   0.44265184  8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.1568505249081218\n",
      "\n",
      "step 48\n",
      "observation: [ 0.16782604  0.62353987  1.2726485  -0.92393523 -0.27874455 -0.28133968\n",
      " -0.44033542  0.38029784  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1622165160401472\n",
      "\n",
      "step 49\n",
      "observation: [ 0.12586953 -0.03234508  1.2600615  -0.9207007  -0.2913315  -0.27810517\n",
      " -0.45292237  0.38353235  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.14139475449024214\n",
      "\n",
      "step 50\n",
      "observation: [ 0.09440214 -0.5242588   1.2506213  -0.86827487 -0.3007717  -0.22567928\n",
      " -0.4623626   0.43595824  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.22950277108317188\n",
      "\n",
      "step 51\n",
      "observation: [ 0.07080161 -0.3931941   1.2435411  -0.8289554  -0.30785188 -0.18635987\n",
      " -0.46944273  0.47527766  8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.15649492229299078\n",
      "\n",
      "step 52\n",
      "observation: [ 0.5531012  -0.2948956   1.188231   -0.7994659  -0.36316198 -0.1568703\n",
      " -0.52475286  0.5047672   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.22486010060967027\n",
      "\n",
      "step 53\n",
      "observation: [ 0.9148259  -0.22117169  1.0967485  -0.7773487  -0.4546446  -0.13475314\n",
      " -0.61623544  0.5268844   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2784438897380146\n",
      "\n",
      "step 54\n",
      "observation: [ 0.68611944 -0.6658788   1.0281365  -0.71076083 -0.52325654 -0.06816526\n",
      " -0.6848474   0.59347224  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.33495244344338737\n",
      "\n",
      "step 55\n",
      "observation: [ 5.1458955e-01  5.9092644e-04  9.7667754e-01 -7.1081990e-01\n",
      " -5.7471550e-01 -6.8224356e-02 -7.3630637e-01  5.9341317e-01\n",
      "  0.0000000e+00]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.32197842554365436\n",
      "\n",
      "step 56\n",
      "observation: [-1.14057824e-01  4.43194818e-04  9.88083303e-01 -7.10864246e-01\n",
      " -5.63309669e-01 -6.82686716e-02 -7.24900544e-01  5.93368828e-01\n",
      "  0.00000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.3080884690013221\n",
      "\n",
      "step 57\n",
      "observation: [-0.08554336 -0.4996676   0.99663764 -0.6608975  -0.55475533 -0.01830191\n",
      " -0.7163462   0.6433356   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.24870997401708475\n",
      "\n",
      "step 58\n",
      "observation: [-0.56415755 -0.3747507   1.0530534  -0.6234224  -0.4983396   0.01917316\n",
      " -0.65993047  0.6808107   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.20796874914587352\n",
      "\n",
      "step 59\n",
      "observation: [-0.42311814  0.21893698  1.0953652  -0.6453161  -0.4560278  -0.00272054\n",
      " -0.6176187   0.65891695  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.28039171148466757\n",
      "\n",
      "step 60\n",
      "observation: [-0.31733862  0.16420273  1.127099   -0.66173637 -0.42429394 -0.01914081\n",
      " -0.5858848   0.6424967   8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.16703015553384684\n",
      "\n",
      "step 61\n",
      "observation: [-0.23800395  0.623152    1.1508994  -0.7240516  -0.40049353 -0.08145602\n",
      " -0.5620844   0.5801815   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.15252990366325125\n",
      "\n",
      "step 62\n",
      "observation: [-0.17850296 -0.03263596  1.1687498  -0.720788   -0.38264325 -0.07819242\n",
      " -0.5442341   0.5834451   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.10766134848042384\n",
      "\n",
      "step 63\n",
      "observation: [-0.6338772  -0.02447697  1.2321374  -0.7183403  -0.31925553 -0.07574473\n",
      " -0.48084638  0.5858928   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.08918232004403329\n",
      "\n",
      "step 64\n",
      "observation: [-0.47540793  0.48164228  1.2796782  -0.7665045  -0.27171472 -0.12390895\n",
      " -0.4333056   0.53772855  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.0998374053429412\n",
      "\n",
      "step 65\n",
      "observation: [-0.35655594  0.8612317   1.3153338  -0.8526277  -0.23605913 -0.21003212\n",
      " -0.39765     0.45160538  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.14266414782775094\n",
      "\n",
      "step 66\n",
      "observation: [ 0.23258305  0.6459238   1.2920755  -0.91722006 -0.25931743 -0.2746245\n",
      " -0.4209083   0.38701302  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.2111462963747115\n",
      "\n",
      "step 67\n",
      "observation: [ 0.6744373   0.48444283  1.2246318  -0.9656643  -0.32676116 -0.32306877\n",
      " -0.48835203  0.33856872  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.23811805878946532\n",
      "\n",
      "step 68\n",
      "observation: [ 0.50582796 -0.13666788  1.174049   -0.9519976  -0.37734395 -0.309402\n",
      " -0.5389348   0.35223553  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2229221127727815\n",
      "\n",
      "step 69\n",
      "observation: [-0.12062903 -0.10250091  1.1861119  -0.9417475  -0.36528105 -0.2991519\n",
      " -0.5268719   0.36248562  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.18520760330829708\n",
      "\n",
      "step 70\n",
      "observation: [-0.09047177 -0.5768757   1.1951591  -0.8840599  -0.3562339  -0.24146433\n",
      " -0.51782477  0.42017317  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.14407709558231716\n",
      "\n",
      "step 71\n",
      "observation: [-0.06785383 -0.93265676  1.2019445  -0.79079425 -0.3494485  -0.14819866\n",
      " -0.5110394   0.5134389   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.13503134769876574\n",
      "\n",
      "step 72\n",
      "observation: [-0.05089037 -0.19949257  1.2070335  -0.770845   -0.34435946 -0.1282494\n",
      " -0.50595033  0.53338814  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.22880336767500614\n",
      "\n",
      "step 73\n",
      "observation: [-0.03816778 -0.14961943  1.2108504  -0.75588304 -0.34054267 -0.11328746\n",
      " -0.50213355  0.54835004  8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.16071332877259947\n",
      "\n",
      "step 74\n",
      "observation: [ 0.47137415 -0.11221457  1.1637129  -0.74466157 -0.3876801  -0.102066\n",
      " -0.549271    0.5595715   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14792404586115204\n",
      "\n",
      "step 75\n",
      "observation: [-0.14646937 -0.08416093  1.1783599  -0.73624545 -0.37303317 -0.09364991\n",
      " -0.53462404  0.5679876   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.17741141905909696\n",
      "\n",
      "step 76\n",
      "observation: [ 0.39014795 -0.06312069  1.139345   -0.72993344 -0.41204795 -0.08733784\n",
      " -0.57363886  0.5742997   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.2482079751772568\n",
      "\n",
      "step 77\n",
      "observation: [ 0.792611   -0.04734052  1.060084   -0.72519934 -0.49130905 -0.08260379\n",
      " -0.6528999   0.57903373  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.3199856790285818\n",
      "\n",
      "step 78\n",
      "observation: [ 0.5944582   0.46449462  1.0006381  -0.7716488  -0.5507549  -0.12905325\n",
      " -0.7123458   0.53258425  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.36739983024116335\n",
      "\n",
      "step 79\n",
      "observation: [ 0.44584367 -0.15162905  0.95605373 -0.75648594 -0.59533924 -0.11389034\n",
      " -0.7569301   0.5477472   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.34549339566824805\n",
      "\n",
      "step 80\n",
      "observation: [-0.16561724 -0.11372178  0.9726155  -0.74511373 -0.57877755 -0.10251816\n",
      " -0.74036837  0.55911934  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.27545769427744005\n",
      "\n",
      "step 81\n",
      "observation: [-0.6242129  -0.08529133  1.0350368  -0.7365846  -0.5163562  -0.09398903\n",
      " -0.6779471   0.5676485   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2393996641972474\n",
      "\n",
      "step 82\n",
      "observation: [-0.4681597   0.4360315   1.0818528  -0.7801877  -0.46954027 -0.13759218\n",
      " -0.6311311   0.52404535  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.20319870991862857\n",
      "\n",
      "step 83\n",
      "observation: [-0.3511198  -0.17297637  1.1169647  -0.7628901  -0.43442827 -0.12029454\n",
      " -0.59601915  0.541343    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1698266903165877\n",
      "\n",
      "step 84\n",
      "observation: [-0.26333982 -0.6297323   1.1432987  -0.6999169  -0.4080943  -0.05732132\n",
      " -0.56968516  0.6043162   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11457837125148555\n",
      "\n",
      "step 85\n",
      "observation: [-0.6975049  -0.47229922  1.2130492  -0.65268695 -0.3383438  -0.0100914\n",
      " -0.49993467  0.6515461   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.11355845875754522\n",
      "\n",
      "step 86\n",
      "observation: [-0.02312865 -0.3542244   1.2153621  -0.6172645  -0.33603096  0.02533105\n",
      " -0.4976218   0.68696856  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.1117576162447459\n",
      "\n",
      "step 87\n",
      "observation: [-0.01734649  0.2343317   1.2170967  -0.6406977  -0.3342963   0.00189788\n",
      " -0.49588716  0.6635354   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.21113164505205578\n",
      "\n",
      "step 88\n",
      "observation: [-0.01300987  0.17574877  1.2183977  -0.65827256 -0.3329953  -0.015677\n",
      " -0.49458617  0.6459605   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.14677174531675044\n",
      "\n",
      "step 89\n",
      "observation: [ 0.4902426   0.13181157  1.1693734  -0.6714537  -0.38201958 -0.02885816\n",
      " -0.54361045  0.63277936  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18325869445588677\n",
      "\n",
      "step 90\n",
      "observation: [ 0.36768195  0.59885865  1.1326052  -0.7313396  -0.41878778 -0.08874403\n",
      " -0.58037865  0.5728935   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2062394738357664\n",
      "\n",
      "step 91\n",
      "observation: [ 0.27576146 -0.05085599  1.1050291  -0.726254   -0.44636393 -0.08365843\n",
      " -0.6079548   0.5779791   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.32450708650853893\n",
      "\n",
      "step 92\n",
      "observation: [ 0.2068211  -0.03814199  1.084347   -0.7224398  -0.46704602 -0.07984423\n",
      " -0.6286369   0.5817933   8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.19303255152807358\n",
      "\n",
      "step 93\n",
      "observation: [-0.3448842  -0.02860649  1.1188354  -0.71957916 -0.4325576  -0.07698358\n",
      " -0.59414846  0.5846539   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1328294162996674\n",
      "\n",
      "step 94\n",
      "observation: [-0.7586631  -0.02145487  1.1947017  -0.7174337  -0.3566913  -0.07483809\n",
      " -0.5182822   0.58679944  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.19523745943306703\n",
      "\n",
      "step 95\n",
      "observation: [-0.5689973  -0.01609115  1.2516015  -0.71582454 -0.29979154 -0.07322898\n",
      " -0.46138242  0.5884085   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1712962137201378\n",
      "\n",
      "step 96\n",
      "observation: [-0.426748   -0.01206836  1.2942762  -0.7146177  -0.25711676 -0.07202214\n",
      " -0.4187076   0.58961535  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.03572137026437999\n",
      "\n",
      "step 97\n",
      "observation: [-0.820061   -0.00905127  1.3762823  -0.7137126  -0.17511065 -0.07111701\n",
      " -0.3367015   0.5905205   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.03172848391130595\n",
      "\n",
      "step 98\n",
      "observation: [-0.11504576 -0.00678845  1.3877869  -0.71303374 -0.16360608 -0.07043817\n",
      " -0.32519695  0.59119934  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02441523688044176\n",
      "\n",
      "step 99\n",
      "observation: [-0.08628432 -0.50509137  1.3964154  -0.6625246  -0.15497765 -0.01992903\n",
      " -0.31656852  0.6417085   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.023081149421315686\n",
      "\n",
      "step 100\n",
      "observation: [-0.06471324  0.1211815   1.4028866  -0.67464274 -0.14850633 -0.03204718\n",
      " -0.3100972   0.62959033  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02071470923661379\n",
      "\n",
      "step 101\n",
      "observation: [-0.04853493 -0.40911388  1.4077401  -0.63373137 -0.14365283  0.0088642\n",
      " -0.3052437   0.6705017   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02762235735705485\n",
      "\n",
      "step 102\n",
      "observation: [-0.0364012  -0.8068354   1.4113803  -0.55304784 -0.14001271  0.08954775\n",
      " -0.3016036   0.75118524  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05759290343056021\n",
      "\n",
      "step 103\n",
      "observation: [ 0.4726991  -0.60512656  1.3641104  -0.49253517 -0.18728262  0.1500604\n",
      " -0.3488735   0.8116979   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.11258311467408033\n",
      "\n",
      "step 104\n",
      "observation: [ 0.8545243  -0.4538449   1.2786579  -0.44715068 -0.27273506  0.1954449\n",
      " -0.43432593  0.8570824   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.20229566556195872\n",
      "\n",
      "step 105\n",
      "observation: [ 1.1408932  -0.34038368  1.1645687  -0.4131123  -0.38682437  0.22948326\n",
      " -0.54841524  0.8911208   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2434456089516193\n",
      "\n",
      "step 106\n",
      "observation: [ 0.35566995 -0.25528777  1.1290016  -0.38758352 -0.42239136  0.25501204\n",
      " -0.5839822   0.9166496   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2519079117987542\n",
      "\n",
      "step 107\n",
      "observation: [ 0.26675245  0.30853418  1.1023264  -0.41843694 -0.4490666   0.22415861\n",
      " -0.6106575   0.8857961   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.30984529024739693\n",
      "\n",
      "step 108\n",
      "observation: [ 0.70006436  0.23140064  1.0323199  -0.44157702 -0.51907307  0.20101856\n",
      " -0.68066394  0.86265606  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.3057757635277348\n",
      "\n",
      "step 109\n",
      "observation: [ 0.02504825  0.17355047  1.0298151  -0.45893207 -0.5215779   0.18366352\n",
      " -0.68316877  0.84530103  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2885624527476315\n",
      "\n",
      "step 110\n",
      "observation: [ 0.01878619  0.63016284  1.0279365  -0.52194834 -0.5234565   0.12064722\n",
      " -0.6850474   0.78228474  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2760306067270891\n",
      "\n",
      "step 111\n",
      "observation: [ 0.01408964  0.97262216  1.0265275  -0.61921054 -0.52486545  0.02338501\n",
      " -0.6864563   0.68502253  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.28650666175048295\n",
      "\n",
      "step 112\n",
      "observation: [ 0.01056723  1.2294666   1.0254709  -0.7421572  -0.5259222  -0.09956165\n",
      " -0.68751305  0.56207585  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.33588192743348927\n",
      "\n",
      "step 113\n",
      "observation: [ 0.00792542  1.4221      1.0246782  -0.8843672  -0.52671474 -0.24177164\n",
      " -0.6883056   0.41986588  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.36711487061555137\n",
      "\n",
      "step 114\n",
      "observation: [ 0.00594407  0.566575    1.0240839  -0.9410247  -0.5273091  -0.29842913\n",
      " -0.6889      0.36320838  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.36316103470819133\n",
      "\n",
      "step 115\n",
      "observation: [ 0.00445805 -0.07506878  1.023638   -0.9335178  -0.52775496 -0.29092225\n",
      " -0.6893458   0.37071526  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.3342406716121147\n",
      "\n",
      "step 116\n",
      "observation: [ 0.00334354 -0.5563016   1.0233037  -0.87788767 -0.5280893  -0.2352921\n",
      " -0.68968016  0.4263454   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2997554100549949\n",
      "\n",
      "step 117\n",
      "observation: [ 0.00250765 -0.9172262   1.0230529  -0.78616506 -0.52834004 -0.14356948\n",
      " -0.6899309   0.518068    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2345808470318776\n",
      "\n",
      "step 118\n",
      "observation: [-0.49811926 -0.6879196   1.0728649  -0.7173731  -0.47852814 -0.07477752\n",
      " -0.640119    0.58686     0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1535508069280965\n",
      "\n",
      "step 119\n",
      "observation: [-0.87358946 -0.5159397   1.1602238  -0.6657791  -0.3911692  -0.02318355\n",
      " -0.55276006  0.63845396  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.20628852608767004\n",
      "\n",
      "step 120\n",
      "observation: [-0.6551921  -0.38695478  1.225743   -0.62708366 -0.32564998  0.01551193\n",
      " -0.48724085  0.6771494   8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.07648797674652966\n",
      "\n",
      "step 121\n",
      "observation: [-0.49139407  0.2097839   1.2748824  -0.64806205 -0.27651057 -0.00546646\n",
      " -0.43810144  0.656171    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.08435005714263653\n",
      "\n",
      "step 122\n",
      "observation: [ 0.13145445  0.15733793  1.261737   -0.6637958  -0.289656   -0.02120025\n",
      " -0.4512469   0.64043725  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12324984311658972\n",
      "\n",
      "step 123\n",
      "observation: [ 0.59859085  0.11800344  1.201878   -0.6755962  -0.3495151  -0.0330006\n",
      " -0.51110595  0.6286369   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.19925121864174947\n",
      "\n",
      "step 124\n",
      "observation: [ 0.94894314  0.08850259  1.1069835  -0.68444645 -0.4444094  -0.04185086\n",
      " -0.6060003   0.6197867   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2191160114696015\n",
      "\n",
      "step 125\n",
      "observation: [ 0.21170735  0.06637694  1.0858128  -0.69108415 -0.46558014 -0.04848855\n",
      " -0.62717104  0.613149    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.28530651567619464\n",
      "\n",
      "step 126\n",
      "observation: [ 0.6587805   0.0497827   1.0199348  -0.6960624  -0.5314582  -0.05346682\n",
      " -0.6930491   0.6081707   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.3374580112836239\n",
      "\n",
      "step 127\n",
      "observation: [ 0.49408537 -0.46266297  0.9705263  -0.64979607 -0.58086675 -0.00720052\n",
      " -0.7424576   0.654437    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.3232928973909028\n",
      "\n",
      "step 128\n",
      "observation: [-0.12943596 -0.34699723  0.98346984 -0.6150964  -0.5679231   0.0274992\n",
      " -0.729514    0.6891367   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2611477526151034\n",
      "\n",
      "step 129\n",
      "observation: [-0.59707695 -0.26024792  1.0431776  -0.5890716  -0.5082154   0.05352399\n",
      " -0.6698063   0.7151615   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3201069113601594\n",
      "\n",
      "step 130\n",
      "observation: [-0.44780773 -0.19518594  1.0879583  -0.56955296 -0.46343467  0.07304259\n",
      " -0.6250255   0.7346801   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.23794319904945288\n",
      "\n",
      "step 131\n",
      "observation: [ 0.1641442  -0.14638945  1.0715439  -0.55491406 -0.4798491   0.08768153\n",
      " -0.64144     0.749319    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20523932255752084\n",
      "\n",
      "step 132\n",
      "observation: [-0.37689185 -0.10979209  1.1092331  -0.5439348  -0.4421599   0.09866074\n",
      " -0.60375077  0.76029825  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14384469934655897\n",
      "\n",
      "step 133\n",
      "observation: [-0.7826689  -0.08234407  1.1875     -0.53570044 -0.363893    0.10689515\n",
      " -0.5254839   0.76853263  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09712062023214428\n",
      "\n",
      "step 134\n",
      "observation: [-0.5870017   0.43824196  1.2462002  -0.57952464 -0.30519286  0.06307095\n",
      " -0.46678373  0.72470844  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.045504017307675136\n",
      "\n",
      "step 135\n",
      "observation: [-0.94025123  0.32868147  1.3402252  -0.6123928  -0.21116772  0.03020281\n",
      " -0.3727586   0.6918403   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.021757759245621528\n",
      "\n",
      "step 136\n",
      "observation: [-0.70518845  0.7465111   1.4107441  -0.68704385 -0.14064889 -0.0444483\n",
      " -0.30223975  0.6171892   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03033295449639447\n",
      "\n",
      "step 137\n",
      "observation: [-0.5288913   1.0598834   1.4636333  -0.7930322  -0.08775975 -0.15043664\n",
      " -0.24935062  0.5112009   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15517975876600984\n",
      "\n",
      "step 138\n",
      "observation: [-0.3966685   0.7949125   1.5033001  -0.8725234  -0.0480929  -0.22992788\n",
      " -0.20968376  0.43170962  8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11562816096978194\n",
      "\n",
      "step 139\n",
      "observation: [-0.2975014   1.0961844   1.5330502  -0.9821419  -0.01834276 -0.33954632\n",
      " -0.17993364  0.3220912   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.13822136773365842\n",
      "\n",
      "step 140\n",
      "observation: [-0.22312602  0.32213828  1.5553628  -1.0143558   0.00396984 -0.37176016\n",
      " -0.15762103  0.28987736  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.12008967347881479\n",
      "\n",
      "step 141\n",
      "observation: [-0.16734453 -0.2583963   1.5720973  -0.9885161   0.02070429 -0.34592053\n",
      " -0.14088657  0.31571698  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.10690928293255697\n",
      "\n",
      "step 142\n",
      "observation: [ 0.3744916  -0.19379722  1.5346482  -0.96913636 -0.01674487 -0.3265408\n",
      " -0.17833574  0.33509672  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.19935763244530846\n",
      "\n",
      "step 143\n",
      "observation: [ 0.2808687  -0.14534791  1.5065613  -0.9546016  -0.04483174 -0.312006\n",
      " -0.20642261  0.34963152  8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.19500656844249734\n",
      "\n",
      "step 144\n",
      "observation: [ 0.21065153 -0.10901093  1.4854962  -0.9437005  -0.06589689 -0.3011049\n",
      " -0.22748776  0.3605326   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1031512250424652\n",
      "\n",
      "step 145\n",
      "observation: [ 0.65798867 -0.0817582   1.4196973  -0.93552464 -0.13169576 -0.29292908\n",
      " -0.29328662  0.36870843  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.21502990670526606\n",
      "\n",
      "step 146\n",
      "observation: [ 0.49349147 -0.06131865  1.3703481  -0.9293928  -0.1810449  -0.28679723\n",
      " -0.34263578  0.3748403   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.22718464938458072\n",
      "\n",
      "step 147\n",
      "observation: [ 0.37011862 -0.04598899  1.3333362  -0.9247939  -0.21805677 -0.28219834\n",
      " -0.37964764  0.37943918  8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2381264303246871\n",
      "\n",
      "step 148\n",
      "observation: [ 0.27758896 -0.03449174  1.3055773  -0.9213447  -0.24581566 -0.27874917\n",
      " -0.40740654  0.38288835  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.12319625237626841\n",
      "\n",
      "step 149\n",
      "observation: [-0.29180828 -0.02586881  1.3347582  -0.91875786 -0.21663484 -0.27616227\n",
      " -0.3782257   0.38547525  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08820281175543193\n",
      "\n",
      "step 150\n",
      "observation: [-0.21885622 -0.5194016   1.3566438  -0.8668177  -0.1947492  -0.22422211\n",
      " -0.35634008  0.4374154   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05010053041731897\n",
      "\n",
      "step 151\n",
      "observation: [-0.16414216 -0.8895512   1.373058   -0.77786255 -0.178335   -0.13526699\n",
      " -0.33992586  0.5263705   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05136570531640269\n",
      "\n",
      "step 152\n",
      "observation: [ 0.37689337 -0.66716343  1.3353686  -0.71114624 -0.21602434 -0.06855065\n",
      " -0.3776152   0.59308684  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.06066966438649247\n",
      "\n",
      "step 153\n",
      "observation: [ 0.28267005 -1.0003725   1.3071016  -0.61110896 -0.24429134  0.0314866\n",
      " -0.4058822   0.6931241   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09498258776535698\n",
      "\n",
      "step 154\n",
      "observation: [ 0.21200253 -1.2502794   1.2859014  -0.48608103 -0.2654916   0.15651454\n",
      " -0.42708245  0.81815207  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11618498804709343\n",
      "\n",
      "step 155\n",
      "observation: [-0.3409981  -0.93770957  1.3200012  -0.39231008 -0.23139179  0.2502855\n",
      " -0.39298266  0.911923    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.16823545400450973\n",
      "\n",
      "step 156\n",
      "observation: [ 0.24425142 -0.7032822   1.2955761  -0.32198185 -0.25581694  0.3206137\n",
      " -0.41740778  0.9822512   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.17971203791509094\n",
      "\n",
      "step 157\n",
      "observation: [ 0.18318857 -0.02746163  1.2772572  -0.31923568 -0.27413577  0.32335988\n",
      " -0.43572664  0.9849974   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.2200573156041543\n",
      "\n",
      "step 158\n",
      "observation: [ 0.63739145 -0.02059622  1.213518   -0.31717607 -0.33787492  0.32541952\n",
      " -0.4994658   0.98705703  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.35565410679861054\n",
      "\n",
      "step 159\n",
      "observation: [ 0.47804356 -0.01544717  1.1657137  -0.31563136 -0.38567927  0.32696423\n",
      " -0.5472701   0.98860174  8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3853542312162608\n",
      "\n",
      "step 160\n",
      "observation: [ 0.35853267 -0.01158537  1.1298604  -0.31447282 -0.42153254  0.32812276\n",
      " -0.58312345  0.9897603   8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2789190823022471\n",
      "\n",
      "step 161\n",
      "observation: [ 0.2688995   0.49131098  1.1029705  -0.36360392 -0.4484225   0.27899167\n",
      " -0.61001337  0.9406292   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2338509506050986\n",
      "\n",
      "step 162\n",
      "observation: [-0.29832536  0.36848322  1.1328031  -0.40045223 -0.41858995  0.24214333\n",
      " -0.5801808   0.9037809   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3030000619625547\n",
      "\n",
      "step 163\n",
      "observation: [-0.22374402  0.27636242  1.1551775  -0.4280885  -0.39621556  0.21450709\n",
      " -0.55780643  0.8761446   8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14607791308867282\n",
      "\n",
      "step 164\n",
      "observation: [-0.667808    0.20727181  1.2219583  -0.44881564 -0.32943475  0.19377992\n",
      " -0.49102563  0.85541743  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09448004393598708\n",
      "\n",
      "step 165\n",
      "observation: [-0.50085604  0.65545386  1.2720438  -0.514361   -0.27934915  0.12823452\n",
      " -0.44094002  0.78987205  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07512044502546847\n",
      "\n",
      "step 166\n",
      "observation: [-0.375642   -0.0084096   1.3096081  -0.51352006 -0.24178496  0.12907548\n",
      " -0.40337583  0.790713    0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.051983081638777434\n",
      "\n",
      "step 167\n",
      "observation: [-0.28173152  0.4936928   1.3377812  -0.56288934 -0.2136118   0.07970621\n",
      " -0.3752027   0.74134374  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.13887081512143545\n",
      "\n",
      "step 168\n",
      "observation: [-0.21129863  0.3702696   1.358911   -0.59991634 -0.19248194  0.04267925\n",
      " -0.3540728   0.70431674  8.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.035412943292888265\n",
      "\n",
      "step 169\n",
      "observation: [-0.15847397 -0.2222978   1.3747585  -0.57768655 -0.17663454  0.06490903\n",
      " -0.33822542  0.7265465   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05277264916371095\n",
      "\n",
      "step 170\n",
      "observation: [ 0.38114452 -0.16672336  1.336644   -0.56101424 -0.214749    0.08158136\n",
      " -0.37633985  0.7432189   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.046230460431670284\n",
      "\n",
      "step 171\n",
      "observation: [-0.2141416  -0.12504251  1.3580582  -0.54850996 -0.19333483  0.09408561\n",
      " -0.3549257   0.7557231   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.054977285081135474\n",
      "\n",
      "step 172\n",
      "observation: [-0.1606062  -0.5937819   1.3741188  -0.48913178 -0.17727421  0.1534638\n",
      " -0.33886507  0.8151013   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.049203782198469354\n",
      "\n",
      "step 173\n",
      "observation: [-0.12045465  0.05466359  1.3861643  -0.49459812 -0.16522874  0.14799744\n",
      " -0.32681963  0.809635    0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03321354129195102\n",
      "\n",
      "step 174\n",
      "observation: [-0.09034099  0.5409977   1.3951983  -0.5486979  -0.15619464  0.09389768\n",
      " -0.31778553  0.7555352   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04261129284080818\n",
      "\n",
      "step 175\n",
      "observation: [ 0.43224427  0.40574828  1.3519739  -0.58927274 -0.19941907  0.05332285\n",
      " -0.36100996  0.71496034  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05448343478069831\n",
      "\n",
      "step 176\n",
      "observation: [ 0.3241832   0.8043112   1.3195556  -0.66970384 -0.23183739 -0.02710827\n",
      " -0.39342827  0.63452923  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17325768888115967\n",
      "\n",
      "step 177\n",
      "observation: [ 0.24313739  0.6032334   1.2952418  -0.7300272  -0.25615114 -0.08743161\n",
      " -0.417742    0.5742059   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.19289033750712398\n",
      "\n",
      "step 178\n",
      "observation: [ 0.18235305  0.45242506  1.2770065  -0.7752697  -0.27438644 -0.13267411\n",
      " -0.4359773   0.5289634   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.21073780038007006\n",
      "\n",
      "step 179\n",
      "observation: [ 0.13676478  0.33931878  1.2633301  -0.80920154 -0.2880629  -0.166606\n",
      " -0.44965377  0.4950315   8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.22588007176173308\n",
      "\n",
      "step 180\n",
      "observation: [ 0.10257359  0.2544891   1.2530727  -0.83465046 -0.29832026 -0.1920549\n",
      " -0.45991114  0.46958262  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1101235753041975\n",
      "\n",
      "step 181\n",
      "observation: [-0.4230698   0.19086681  1.2953798  -0.8537372  -0.2560133  -0.21114159\n",
      " -0.41760415  0.45049593  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12606187258717458\n",
      "\n",
      "step 182\n",
      "observation: [ 0.18269764  0.14315012  1.27711    -0.8680522  -0.27428305 -0.2254566\n",
      " -0.43587393  0.43618092  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1124241063332859\n",
      "\n",
      "step 183\n",
      "observation: [-0.36297676  0.10736258  1.3134077  -0.8787884  -0.23798539 -0.23619285\n",
      " -0.39957625  0.42544466  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1276525289082604\n",
      "\n",
      "step 184\n",
      "observation: [ 0.22776742  0.08052194  1.2906308  -0.88684064 -0.26076213 -0.24424505\n",
      " -0.422353    0.41739246  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.11731142470284835\n",
      "\n",
      "step 185\n",
      "observation: [ 0.17082557 -0.43960854  1.2735484  -0.8428798  -0.27784467 -0.2002842\n",
      " -0.43943554  0.46135333  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08590942311528754\n",
      "\n",
      "step 186\n",
      "observation: [-0.37188083 -0.3297064   1.3107364  -0.8099091  -0.2406566  -0.16731356\n",
      " -0.40224746  0.49432397  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.04682326524320332\n",
      "\n",
      "step 187\n",
      "observation: [-0.77891064 -0.24727981  1.3886274  -0.78518116 -0.16276553 -0.14258558\n",
      " -0.3243564   0.51905197  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.039208889548024047\n",
      "\n",
      "step 188\n",
      "observation: [-0.08418296 -0.18545985  1.3970457  -0.7666352  -0.15434724 -0.12403959\n",
      " -0.31593812  0.53759795  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05134591237852213\n",
      "\n",
      "step 189\n",
      "observation: [ 0.43686277 -0.13909489  1.3533595  -0.75272566 -0.19803351 -0.1101301\n",
      " -0.3596244   0.5515074   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.08878733072443279\n",
      "\n",
      "step 190\n",
      "observation: [ 0.8276471  -0.10432117  1.2705947  -0.74229354 -0.28079823 -0.09969798\n",
      " -0.4423891   0.56193954  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.11931446416719303\n",
      "\n",
      "step 191\n",
      "observation: [ 0.6207353  -0.5782409   1.2085212  -0.68446946 -0.34287176 -0.04187389\n",
      " -0.5044626   0.6197636   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.15400612938646746\n",
      "\n",
      "step 192\n",
      "observation: [ 0.4655515   0.06631934  1.1619661  -0.6911014  -0.38942692 -0.04850583\n",
      " -0.55101776  0.6131317   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2829272940718768\n",
      "\n",
      "step 193\n",
      "observation: [ 0.34916362  0.04973951  1.1270497  -0.6960754  -0.42434326 -0.05347978\n",
      " -0.58593416  0.60815775  8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3062507764570951\n",
      "\n",
      "step 194\n",
      "observation: [ 0.2618727   0.03730463  1.1008625  -0.6998058  -0.45053053 -0.05721024\n",
      " -0.6121214   0.6044273   8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.23316253970866382\n",
      "\n",
      "step 195\n",
      "observation: [ 0.19640453  0.5279785   1.081222   -0.75260365 -0.470171   -0.11000809\n",
      " -0.63176185  0.5516294   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.30850153025211385\n",
      "\n",
      "step 196\n",
      "observation: [ 0.6473034   0.39598384  1.0164917  -0.79220206 -0.5349013  -0.14960648\n",
      " -0.6964922   0.512031    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.31671830402708284\n",
      "\n",
      "step 197\n",
      "observation: [-0.01452245  0.2969879   1.0179439  -0.82190084 -0.5334491  -0.17930527\n",
      " -0.6950399   0.48233226  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.306383365554357\n",
      "\n",
      "step 198\n",
      "observation: [-0.01089184 -0.27725908  1.0190331  -0.7941749  -0.5323599  -0.15157935\n",
      " -0.6939508   0.51005816  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.24898837026942686\n",
      "\n",
      "step 199\n",
      "observation: [-0.5081689  -0.20794432  1.06985    -0.7733805  -0.481543   -0.13078493\n",
      " -0.6431339   0.5308526   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.223917912313753\n",
      "\n",
      "step 0\n",
      "observation: [ 0.          0.          0.5784319   0.6491301   0.6006349   1.2409122\n",
      " -0.84865373  0.4064499   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.8472696823667175\n",
      "\n",
      "step 1\n",
      "observation: [ 0.          0.5         0.5784319   0.5991301   0.6006349   1.1909122\n",
      " -0.84865373  0.35644993  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.7925471948371194\n",
      "\n",
      "step 2\n",
      "observation: [ 0.          0.875       0.5784319   0.5116301   0.6006349   1.1034122\n",
      " -0.84865373  0.26894993  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.7437216644408651\n",
      "\n",
      "step 3\n",
      "observation: [ 0.          1.15625     0.5784319   0.3960051   0.6006349   0.9877872\n",
      " -0.84865373  0.15332493  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.8246495137139866\n",
      "\n",
      "step 4\n",
      "observation: [ 0.          0.8671875   0.5784319   0.30928636  0.6006349   0.90106845\n",
      " -0.84865373  0.06660618  1.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.8075809581790273\n",
      "\n",
      "step 5\n",
      "observation: [ 0.5         0.6503906   0.5284319   0.24424727  0.55063486  0.8360294\n",
      " -0.89865375  0.00156712  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.8858339891330302\n",
      "\n",
      "step 6\n",
      "observation: [ 0.375       0.98779297  0.4909319   0.14546798  0.5131349   0.7372501\n",
      " -0.9361537  -0.09721218  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.058103858338775\n",
      "\n",
      "step 7\n",
      "observation: [ 0.78125     0.7408447   0.4128069   0.07138351  0.43500987  0.6631656\n",
      " -1.0142788  -0.17129666  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.3123080393380355\n",
      "\n",
      "step 8\n",
      "observation: [ 1.0859375   0.55563354  0.30421314  0.01582015  0.32641613  0.60760224\n",
      " -1.1228725  -0.22686     0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.4981377769054804\n",
      "\n",
      "step 9\n",
      "observation: [ 0.8144531  -0.08327484  0.22276784  0.02414764  0.24497081  0.6159297\n",
      " -1.2043178  -0.21853252  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.7463073553512902\n",
      "\n",
      "step 10\n",
      "observation: [ 0.61083984 -0.06245613  0.16168386  0.03039325  0.18388683  0.62217534\n",
      " -1.2654017  -0.2122869   1.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6337615313513767\n",
      "\n",
      "step 11\n",
      "observation: [-0.04187012 -0.0468421   0.16587086  0.03507746  0.18807384  0.62685955\n",
      " -1.2612147  -0.2076027   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.5010965222231298\n",
      "\n",
      "step 12\n",
      "observation: [-0.5314026  -0.03513157  0.21901113  0.03859062  0.2412141   0.6303727\n",
      " -1.2080745  -0.20408954  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.524642246322385\n",
      "\n",
      "step 13\n",
      "observation: [ 0.10144806 -0.02634868  0.20886631  0.04122549  0.2310693   0.6330076\n",
      " -1.2182193  -0.20145467  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.5648935825751318\n",
      "\n",
      "step 14\n",
      "observation: [ 0.07608604  0.4802385   0.20125772 -0.00679836  0.22346069  0.58498377\n",
      " -1.2258279  -0.24947852  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.698185072238347\n",
      "\n",
      "step 15\n",
      "observation: [ 0.05706453  0.36017886  0.19555126 -0.04281625  0.21775423  0.5489659\n",
      " -1.2315344  -0.2854964   8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.503817700279296\n",
      "\n",
      "step 16\n",
      "observation: [-0.4572016   0.27013415  0.24127142 -0.06982967  0.2634744   0.52195245\n",
      " -1.1858141  -0.31250983  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.5543959046011562\n",
      "\n",
      "step 17\n",
      "observation: [ 0.1570988   0.20260061  0.22556154 -0.09008972  0.24776451  0.5016924\n",
      " -1.201524   -0.3327699   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6304887185954136\n",
      "\n",
      "step 18\n",
      "observation: [ 0.1178241   0.6519505   0.21377914 -0.15528478  0.2359821   0.43649733\n",
      " -1.2133064  -0.39796492  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.8180335087053043\n",
      "\n",
      "step 19\n",
      "observation: [ 0.58836806  0.48896286  0.15494232 -0.20418106  0.1771453   0.38760105\n",
      " -1.2721432  -0.4468612   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.1005007504834117\n",
      "\n",
      "step 20\n",
      "observation: [ 0.9412761   0.36672214  0.06081472 -0.24085326  0.08301769  0.35092884\n",
      " -1.3662709  -0.48353344  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.42574505711865\n",
      "\n",
      "step 21\n",
      "observation: [ 0.70595706  0.2750416  -0.00978099 -0.26835743  0.01242199  0.3234247\n",
      " -1.4368666  -0.5110376   6.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.3557310037429313\n",
      "\n",
      "step 22\n",
      "observation: [ 0.02946778  0.2062812  -0.01272776 -0.28898555  0.00947521  0.30279657\n",
      " -1.4398134  -0.5316657   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.237088008362915\n",
      "\n",
      "step 23\n",
      "observation: [-0.47789916  0.1547109   0.03506215 -0.30445665  0.05726513  0.28732547\n",
      " -1.3920234  -0.5471368   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.0182993993224323\n",
      "\n",
      "step 24\n",
      "observation: [-0.85842437  0.11603317  0.12090459 -0.31605995  0.14310756  0.27572215\n",
      " -1.3061811  -0.55874014  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.9233006270623068\n",
      "\n",
      "step 25\n",
      "observation: [-0.64381826  0.58702487  0.18528642 -0.37476245  0.20748939  0.21701966\n",
      " -1.2417992  -0.6174426   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.9838660207903624\n",
      "\n",
      "step 26\n",
      "observation: [ 0.01713629  0.44026867  0.18357278 -0.4187893   0.20577577  0.1729928\n",
      " -1.2435129  -0.66146946  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.9648890620024462\n",
      "\n",
      "step 27\n",
      "observation: [ 0.01285222 -0.16979851  0.18228757 -0.40180945  0.20449054  0.18997265\n",
      " -1.2447981  -0.64448965  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.078113121510284\n",
      "\n",
      "step 28\n",
      "observation: [ 0.50963914 -0.12734888  0.13132365 -0.38907456  0.15352662  0.20270754\n",
      " -1.295762   -0.63175476  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.0357545169594617\n",
      "\n",
      "step 29\n",
      "observation: [-0.11777063 -0.09551166  0.14310071 -0.3795234   0.16530369  0.21225871\n",
      " -1.2839849  -0.6222036   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.945283319319919\n",
      "\n",
      "step 30\n",
      "observation: [-0.08832797 -0.57163376  0.1519335  -0.32236004  0.17413649  0.26942208\n",
      " -1.2751521  -0.5650402   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.757468361543012\n",
      "\n",
      "step 31\n",
      "observation: [-0.566246   -0.4287253   0.20855811 -0.2794875   0.23076108  0.3122946\n",
      " -1.2185276  -0.5221677   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6747292434181449\n",
      "\n",
      "step 32\n",
      "observation: [-0.4246845   0.17845602  0.25102657 -0.2973331   0.27322954  0.294449\n",
      " -1.176059   -0.54001325  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6732998219515434\n",
      "\n",
      "step 33\n",
      "observation: [-0.31851336  0.633842    0.2828779  -0.3607173   0.30508086  0.23106481\n",
      " -1.1442077  -0.6033975   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.616238778849674\n",
      "\n",
      "step 34\n",
      "observation: [-0.23888502 -0.02461849  0.3067664  -0.35825545  0.32896936  0.23352666\n",
      " -1.1203192  -0.60093564  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.6741999431472916\n",
      "\n",
      "step 35\n",
      "observation: [-0.17916377 -0.01846387  0.32468277 -0.35640907  0.34688574  0.23537304\n",
      " -1.1024028  -0.5990892   7.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.6430965988465163\n",
      "\n",
      "step 36\n",
      "observation: [-0.13437283 -0.0138479   0.33812004 -0.35502428  0.36032304  0.23675783\n",
      " -1.0889655  -0.59770447  7.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.4146198330463033\n",
      "\n",
      "step 37\n",
      "observation: [-0.6007796  -0.01038592  0.398198   -0.3539857   0.42040098  0.23779643\n",
      " -1.0288876  -0.59666586  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3850896518697886\n",
      "\n",
      "step 38\n",
      "observation: [-0.4505847   0.49221057  0.4432565  -0.40320674  0.46545947  0.18857537\n",
      " -0.98382914 -0.6458869   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.4662900775454708\n",
      "\n",
      "step 39\n",
      "observation: [ 0.16206147  0.3691579   0.42705035 -0.44012254  0.44925332  0.15165958\n",
      " -1.0000353  -0.6828027   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.4607747998149005\n",
      "\n",
      "step 40\n",
      "observation: [ 0.1215461  -0.22313157  0.41489574 -0.41780937  0.4370987   0.17397274\n",
      " -1.0121899  -0.66048956  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.5574857065854637\n",
      "\n",
      "step 41\n",
      "observation: [ 0.09115957 -0.16734867  0.40577978 -0.4010745   0.42798275  0.1907076\n",
      " -1.0213058  -0.64375466  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.355181248266522\n",
      "\n",
      "step 42\n",
      "observation: [-0.4316303  -0.1255115   0.4489428  -0.38852337  0.47114578  0.20325875\n",
      " -0.9781428  -0.63120353  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.3781819679269842\n",
      "\n",
      "step 43\n",
      "observation: [ 0.17627726 -0.09413363  0.43131506 -0.37911     0.45351806  0.21267211\n",
      " -0.9957705  -0.6217902   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.297557518782252\n",
      "\n",
      "step 44\n",
      "observation: [-0.36779204 -0.07060022  0.4680943  -0.37205     0.49029726  0.21973214\n",
      " -0.95899135 -0.6147301   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3023735459699795\n",
      "\n",
      "step 45\n",
      "observation: [-0.27584404  0.44704983  0.4956787  -0.41675496  0.5178817   0.17502715\n",
      " -0.9314069  -0.6594351   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3814037105121404\n",
      "\n",
      "step 46\n",
      "observation: [-0.20688303  0.8352874   0.516367   -0.5002837   0.53857     0.09149841\n",
      " -0.9107186  -0.74296385  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.54241570962057\n",
      "\n",
      "step 47\n",
      "observation: [ 0.34483773  0.62646556  0.48188323 -0.5629302   0.5040862   0.02885186\n",
      " -0.9452024  -0.8056104   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.587126935373024\n",
      "\n",
      "step 48\n",
      "observation: [ 0.2586283  -0.03015085  0.45602039 -0.5599152   0.47822335  0.03186695\n",
      " -0.9710652  -0.8025953   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.7230964341085007\n",
      "\n",
      "step 49\n",
      "observation: [ 0.6939712  -0.02261314  0.38662326 -0.55765384  0.40882623  0.03412826\n",
      " -1.0404624  -0.80033404  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.7540374151791351\n",
      "\n",
      "step 50\n",
      "observation: [ 0.5204784  -0.51695985  0.3345754  -0.50595784  0.3567784   0.08582424\n",
      " -1.0925102  -0.74863803  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.8999609149739811\n",
      "\n",
      "step 51\n",
      "observation: [ 0.8903588  -0.3877199   0.24553955 -0.46718588  0.2677425   0.12459624\n",
      " -1.1815461  -0.70986605  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.899448976480736\n",
      "\n",
      "step 52\n",
      "observation: [ 0.1677691  -0.29078993  0.22876263 -0.4381069   0.2509656   0.15367523\n",
      " -1.198323   -0.680787    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.7819537123143492\n",
      "\n",
      "step 53\n",
      "observation: [-0.37417316 -0.21809244  0.26617995 -0.41629764  0.28838292  0.17548446\n",
      " -1.1609057  -0.6589778   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.6345320317648755\n",
      "\n",
      "step 54\n",
      "observation: [-0.28062987 -0.66356933  0.29424295 -0.34994072  0.31644592  0.2418414\n",
      " -1.1328427  -0.59262085  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.4789933449617494\n",
      "\n",
      "step 55\n",
      "observation: [-0.2104724  -0.99767697  0.31529018 -0.250173    0.33749315  0.3416091\n",
      " -1.1117954  -0.49285316  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.488085842237861\n",
      "\n",
      "step 56\n",
      "observation: [ 0.34214568 -0.74825776  0.2810756  -0.17534724  0.3032786   0.41643488\n",
      " -1.14601    -0.4180274   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6234573212181347\n",
      "\n",
      "step 57\n",
      "observation: [ 0.75660926 -0.5611933   0.20541468 -0.1192279   0.22761765  0.4725542\n",
      " -1.221671   -0.36190808  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6112913336023433\n",
      "\n",
      "step 58\n",
      "observation: [ 0.06745695 -0.42089498  0.19866899 -0.0771384   0.22087196  0.5146437\n",
      " -1.2284167  -0.31981856  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6358768393639145\n",
      "\n",
      "step 59\n",
      "observation: [ 0.05059272  0.18432876  0.19360971 -0.09557128  0.21581268  0.49621084\n",
      " -1.2334759  -0.33825144  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.533568477505171\n",
      "\n",
      "step 60\n",
      "observation: [-0.46205547  0.13824657  0.23981526 -0.10939594  0.26201823  0.48238617\n",
      " -1.1872704  -0.3520761   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.559890177794187\n",
      "\n",
      "step 61\n",
      "observation: [-0.34654158  0.10368493  0.27446944 -0.11976443  0.2966724   0.47201768\n",
      " -1.1526161  -0.36244458  8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.5215113071924913\n",
      "\n",
      "step 62\n",
      "observation: [ 0.2400938   0.0777637   0.25046003 -0.1275408   0.27266303  0.46424133\n",
      " -1.1766256  -0.37022096  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6086684386566579\n",
      "\n",
      "step 63\n",
      "observation: [ 0.18007036  0.5583228   0.232453   -0.18337308  0.254656    0.40840903\n",
      " -1.1946326  -0.42605323  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.8018670401643335\n",
      "\n",
      "step 64\n",
      "observation: [ 0.63505274  0.4187421   0.16894773 -0.22524728  0.19115071  0.36653483\n",
      " -1.258138   -0.46792746  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.9069272908130135\n",
      "\n",
      "step 65\n",
      "observation: [ 0.47628957 -0.18594344  0.12131877 -0.20665294  0.14352174  0.38512918\n",
      " -1.3057668  -0.4493331   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.857504848223266\n",
      "\n",
      "step 66\n",
      "observation: [-0.14278282 -0.13945758  0.13559705 -0.19270718  0.15780003  0.39907494\n",
      " -1.2914885  -0.43538734  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.7809680525742122\n",
      "\n",
      "step 67\n",
      "observation: [-0.10708711 -0.60459316  0.14630577 -0.13224787  0.16850874  0.45953426\n",
      " -1.2807798  -0.37492803  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.7639719421112094\n",
      "\n",
      "step 68\n",
      "observation: [-0.08031534  0.04655511  0.1543373  -0.13690338  0.17654027  0.45487875\n",
      " -1.2727484  -0.37958354  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.7921454867436482\n",
      "\n",
      "step 69\n",
      "observation: [-0.0602365   0.53491634  0.16036095 -0.19039501  0.18256393  0.4013871\n",
      " -1.2667247  -0.43307516  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.7722594070397102\n",
      "\n",
      "step 70\n",
      "observation: [-0.04517738 -0.09881275  0.16487868 -0.18051374  0.18708166  0.41126838\n",
      " -1.2622069  -0.4231939   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.885881649932104\n",
      "\n",
      "step 71\n",
      "observation: [ 0.46611696 -0.07410956  0.11826699 -0.17310278  0.14046997  0.41867933\n",
      " -1.3088186  -0.41578293  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.0175444933496998\n",
      "\n",
      "step 72\n",
      "observation: [ 0.34958774  0.44441783  0.08330822 -0.21754456  0.10551119  0.37423754\n",
      " -1.3437774  -0.46022472  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.259987582783751\n",
      "\n",
      "step 73\n",
      "observation: [ 0.7621908   0.33331338  0.00708914 -0.2508759   0.02929211  0.3409062\n",
      " -1.4199965  -0.49355605  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.305686749076257\n",
      "\n",
      "step 74\n",
      "observation: [ 7.1643099e-02  2.4998502e-01 -7.5170246e-05 -2.7587441e-01\n",
      "  2.2127803e-02  3.1590772e-01 -1.4271607e+00 -5.1855457e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -2.289618361180604\n",
      "\n",
      "step 75\n",
      "observation: [ 0.05373232 -0.31251124 -0.0054484  -0.24462327  0.01675457  0.34715882\n",
      " -1.432534   -0.48730344  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.137730400239514\n",
      "\n",
      "step 76\n",
      "observation: [-0.45970076 -0.23438342  0.04052167 -0.22118494  0.06272465  0.37059718\n",
      " -1.3865639  -0.4638651   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.0744377310783757\n",
      "\n",
      "step 77\n",
      "observation: [-0.34477556  0.32421243  0.07499923 -0.25360617  0.0972022   0.33817592\n",
      " -1.3520864  -0.49628633  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.8997851958711636\n",
      "\n",
      "step 78\n",
      "observation: [-0.7585817   0.24315932  0.1508574  -0.27792212  0.17306037  0.31386\n",
      " -1.2762282  -0.5206023   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.901558019460778\n",
      "\n",
      "step 79\n",
      "observation: [-0.06893626  0.1823695   0.15775102 -0.29615906  0.17995399  0.29562306\n",
      " -1.2693346  -0.5388392   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.0033865294274658\n",
      "\n",
      "step 80\n",
      "observation: [-0.05170219  0.13677712  0.16292125 -0.30983678  0.18512422  0.28194535\n",
      " -1.2641643  -0.55251694  7.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.8512610168925827\n",
      "\n",
      "step 81\n",
      "observation: [-0.03877664 -0.39741716  0.1667989  -0.27009505  0.18900189  0.32168704\n",
      " -1.2602867  -0.51277524  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.691021739176851\n",
      "\n",
      "step 82\n",
      "observation: [-0.5290825  -0.29806286  0.21970716 -0.24028876  0.24191013  0.35149333\n",
      " -1.2073785  -0.48296893  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.4614126563026775\n",
      "\n",
      "step 83\n",
      "observation: [-0.89681184 -0.22354715  0.30938834 -0.21793406  0.3315913   0.37384805\n",
      " -1.1176972  -0.4606142   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.4079614463269403\n",
      "\n",
      "step 84\n",
      "observation: [-0.1726089  -0.16766037  0.32664922 -0.20116802  0.34885222  0.3906141\n",
      " -1.1004363  -0.4438482   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.4798820984437928\n",
      "\n",
      "step 85\n",
      "observation: [ 0.37054333 -0.12574527  0.2895949  -0.18859349  0.3117979   0.40318862\n",
      " -1.1374907  -0.43127364  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.6358322118140411\n",
      "\n",
      "step 86\n",
      "observation: [ 0.2779075  -0.09430896  0.26180416 -0.17916259  0.28400713  0.4126195\n",
      " -1.1652814  -0.42184275  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.462812757489089\n",
      "\n",
      "step 87\n",
      "observation: [-0.29156938 -0.07073171  0.2909611  -0.17208943  0.31316406  0.4196927\n",
      " -1.1361245  -0.4147696   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.3707831517813278\n",
      "\n",
      "step 88\n",
      "observation: [-0.21867703 -0.5530488   0.31282878 -0.11678454  0.33503178  0.47499758\n",
      " -1.1142569  -0.3594647   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3407016365409863\n",
      "\n",
      "step 89\n",
      "observation: [-0.16400777  0.08521341  0.32922956 -0.12530589  0.35143253  0.46647623\n",
      " -1.097856   -0.36798605  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3585265538860851\n",
      "\n",
      "step 90\n",
      "observation: [-0.12300583  0.56391007  0.34153014 -0.18169689  0.3637331   0.41008523\n",
      " -1.0855554  -0.42437705  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.4864005102203017\n",
      "\n",
      "step 91\n",
      "observation: [ 0.40774563  0.42293254  0.3007556  -0.22399014  0.32295856  0.36779195\n",
      " -1.12633    -0.4666703   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.7050268832020634\n",
      "\n",
      "step 92\n",
      "observation: [ 0.8058092   0.3171994   0.22017467 -0.2557101   0.24237764  0.33607203\n",
      " -1.206911   -0.49839026  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.8291216569834086\n",
      "\n",
      "step 93\n",
      "observation: [ 0.60435694 -0.26210046  0.15973897 -0.22950004  0.18194194  0.36228207\n",
      " -1.2673466  -0.47218022  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.027888251316034\n",
      "\n",
      "step 94\n",
      "observation: [ 0.4532677  -0.19657533  0.1144122  -0.2098425   0.13661519  0.3819396\n",
      " -1.3126734  -0.45252267  7.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.051444875817141\n",
      "\n",
      "step 95\n",
      "observation: [ 0.33995077  0.3525685   0.08041713 -0.24509937  0.1026201   0.34668276\n",
      " -1.3466685  -0.48777953  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.0125441725248945\n",
      "\n",
      "step 96\n",
      "observation: [-0.24503693  0.26442638  0.10492082 -0.27154198  0.1271238   0.3202401\n",
      " -1.3221648  -0.51422215  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.934169014157006\n",
      "\n",
      "step 97\n",
      "observation: [-0.18377769 -0.3016802   0.12329859 -0.24137397  0.14550157  0.35040814\n",
      " -1.303787   -0.48405415  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.9256682631217859\n",
      "\n",
      "step 98\n",
      "observation: [-0.13783327  0.27373984  0.13708192 -0.26874796  0.15928489  0.32303417\n",
      " -1.2900037  -0.5114281   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.8698296900750275\n",
      "\n",
      "step 99\n",
      "observation: [-0.10337495 -0.2946951   0.14741941 -0.23927845  0.16962239  0.35250366\n",
      " -1.2796662  -0.4819586   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.7045395543852366\n",
      "\n",
      "step 100\n",
      "observation: [-0.5775312  -0.22102134  0.20517254 -0.21717632  0.2273755   0.3746058\n",
      " -1.2219131  -0.45985648  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6324188446333445\n",
      "\n",
      "step 101\n",
      "observation: [-0.4331484   0.334234    0.24848738 -0.2505997   0.27069035  0.3411824\n",
      " -1.1785983  -0.49327987  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6993682282799145\n",
      "\n",
      "step 102\n",
      "observation: [ 0.1751387   0.2506755   0.23097351 -0.27566725  0.25317648  0.31611484\n",
      " -1.1961122  -0.51834744  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.8508076079310372\n",
      "\n",
      "step 103\n",
      "observation: [ 0.13135402  0.18800662  0.21783811 -0.29446793  0.24004108  0.2973142\n",
      " -1.2092475  -0.53714806  7.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6706676162587886\n",
      "\n",
      "step 104\n",
      "observation: [-0.4014845   0.14100496  0.25798655 -0.30856842  0.2801895   0.2832137\n",
      " -1.1690991  -0.5512486   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6716215904296001\n",
      "\n",
      "step 105\n",
      "observation: [-0.30111337  0.6057537   0.2880979  -0.36914378  0.31030086  0.22263832\n",
      " -1.1389878  -0.611824    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.792483583099745\n",
      "\n",
      "step 106\n",
      "observation: [ 0.27416497  0.4543153   0.2606814  -0.4145753   0.28288436  0.17720678\n",
      " -1.1664042  -0.6572555   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.9868256683668701\n",
      "\n",
      "step 107\n",
      "observation: [ 0.20562373  0.34073648  0.24011903 -0.44864896  0.262322    0.14313315\n",
      " -1.1869665  -0.6913291   7.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.0338494386733394\n",
      "\n",
      "step 108\n",
      "observation: [ 0.1542178   0.75555235  0.22469725 -0.5242042   0.24690022  0.06757791\n",
      " -1.2023884  -0.7668844   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.251921949045028\n",
      "\n",
      "step 109\n",
      "observation: [ 0.11566335  0.5666643   0.2131309  -0.5808706   0.23533387  0.01091148\n",
      " -1.2139547  -0.8235508   7.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.125103519515682\n",
      "\n",
      "step 110\n",
      "observation: [-0.4132525   0.4249982   0.25445616 -0.62337047  0.27665913 -0.03158834\n",
      " -1.1726295  -0.8660506   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -2.0223092889743897\n",
      "\n",
      "step 111\n",
      "observation: [-0.30993935 -0.18125135  0.2854501  -0.6052453   0.30765307 -0.0134632\n",
      " -1.1416355  -0.8479255   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.8659724558637176\n",
      "\n",
      "step 112\n",
      "observation: [-0.23245452 -0.6359385   0.30869555 -0.5416515   0.33089852  0.05013065\n",
      " -1.1183901  -0.7843316   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.8547370253279611\n",
      "\n",
      "step 113\n",
      "observation: [-0.17434089 -0.4769539   0.32612965 -0.4939561   0.3483326   0.09782603\n",
      " -1.100956   -0.7366362   7.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.5684070824605256\n",
      "\n",
      "step 114\n",
      "observation: [-0.63075566 -0.35771543  0.3892052  -0.45818454  0.4114082   0.13359758\n",
      " -1.0378804  -0.7008647   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.3706573174246417\n",
      "\n",
      "step 115\n",
      "observation: [-0.47306675 -0.7682866   0.43651187 -0.38135588  0.45871484  0.21042624\n",
      " -0.99057376 -0.624036    0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.292170938515926\n",
      "\n",
      "step 116\n",
      "observation: [-0.35480008 -0.07621492  0.4719919  -0.37373438  0.49419487  0.21804772\n",
      " -0.95509374 -0.61641455  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.3303829823125408\n",
      "\n",
      "step 117\n",
      "observation: [ 0.23389995 -0.05716119  0.44860187 -0.36801827  0.47080487  0.22376384\n",
      " -0.97848374 -0.6106984   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.3016618239860862\n",
      "\n",
      "step 118\n",
      "observation: [ 0.17542496 -0.5428709   0.4310594  -0.31373116  0.45326236  0.27805093\n",
      " -0.9960262  -0.5564113   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.2353233137832358\n",
      "\n",
      "step 119\n",
      "observation: [ 0.13156873 -0.9071532   0.41790253 -0.22301586  0.4401055   0.36876625\n",
      " -1.009183   -0.46569604  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.159331590964556\n",
      "\n",
      "step 120\n",
      "observation: [ 0.09867654 -1.1803648   0.40803486 -0.10497937  0.43023783  0.48680273\n",
      " -1.0190507  -0.34765953  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.220752151578778\n",
      "\n",
      "step 121\n",
      "observation: [ 0.07400741 -0.88527364  0.4006341  -0.01645201  0.4228371   0.5753301\n",
      " -1.0264515  -0.25913218  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.0014754404394255\n",
      "\n",
      "step 122\n",
      "observation: [-0.44449446 -0.6639553   0.44508356  0.04994352  0.46728653  0.6417256\n",
      " -0.9820021  -0.19273664  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.93719106307109\n",
      "\n",
      "step 123\n",
      "observation: [-0.33337083  0.00203357  0.47842064  0.04974016  0.50062364  0.6415223\n",
      " -0.94866496 -0.19294     0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.9122457476606551\n",
      "\n",
      "step 124\n",
      "observation: [-2.5002813e-01  5.0152516e-01  5.0342345e-01 -4.1235605e-04\n",
      "  5.2562642e-01  5.9136975e-01 -9.2366213e-01 -2.4309252e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.872087865971391\n",
      "\n",
      "step 125\n",
      "observation: [-0.1875211  -0.12385612  0.52217555  0.01197326  0.5443785   0.60375535\n",
      " -0.90491    -0.2307069   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.867274096038082\n",
      "\n",
      "step 126\n",
      "observation: [-0.14064083  0.40710792  0.5362397  -0.02873754  0.55844265  0.56304455\n",
      " -0.89084595 -0.2714177   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.9566284652876549\n",
      "\n",
      "step 127\n",
      "observation: [ 0.3945194   0.30533093  0.49678773 -0.05927063  0.5189907   0.5325115\n",
      " -0.9302979  -0.30195078  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.0618957552224693\n",
      "\n",
      "step 128\n",
      "observation: [ 0.29588953  0.7289982   0.46719876 -0.13217045  0.48940173  0.45961165\n",
      " -0.95988685 -0.3748506   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.2496783102718172\n",
      "\n",
      "step 129\n",
      "observation: [ 0.72191715  0.54674864  0.39500704 -0.18684532  0.41721     0.4049368\n",
      " -1.0320786  -0.42952546  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.501279018890623\n",
      "\n",
      "step 130\n",
      "observation: [ 0.54143786  0.41006148  0.34086326 -0.22785147  0.36306623  0.36393064\n",
      " -1.0862224  -0.4705316   8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.4108511638544292\n",
      "\n",
      "step 131\n",
      "observation: [-0.0939216   0.3075461   0.35025543 -0.25860608  0.3724584   0.33317605\n",
      " -1.0768301  -0.5012862   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.3694522552338495\n",
      "\n",
      "step 132\n",
      "observation: [-0.0704412  -0.26934043  0.35729954 -0.23167203  0.3795025   0.36011007\n",
      " -1.0697861  -0.47435218  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.4394203429186416\n",
      "\n",
      "step 133\n",
      "observation: [-0.0528309  -0.20200531  0.36258262 -0.2114715   0.3847856   0.3803106\n",
      " -1.064503   -0.45415166  8.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.2139145556589466\n",
      "\n",
      "step 134\n",
      "observation: [-0.5396232  -0.15150398  0.41654494 -0.1963211   0.4387479   0.395461\n",
      " -1.0105407  -0.43900126  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.2234152828131828\n",
      "\n",
      "step 135\n",
      "observation: [ 0.09528262 -0.11362799  0.4070167  -0.18495831  0.42921966  0.4068238\n",
      " -1.0200689  -0.42763847  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.191417811369125\n",
      "\n",
      "step 136\n",
      "observation: [ 0.07146196 -0.585221    0.39987049 -0.1264362   0.42207345  0.46534592\n",
      " -1.0272151  -0.36911637  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.207004311446748\n",
      "\n",
      "step 137\n",
      "observation: [ 0.05359647  0.06108426  0.39451084 -0.13254464  0.4167138   0.4592375\n",
      " -1.0325748  -0.3752248   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.3249403132962445\n",
      "\n",
      "step 138\n",
      "observation: [ 0.5401974   0.04581319  0.34049112 -0.13712595  0.36269408  0.45465615\n",
      " -1.0865945  -0.3798061   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.5324608412638292\n",
      "\n",
      "step 139\n",
      "observation: [ 0.905148    0.03435989  0.2499763  -0.14056194  0.27217928  0.45122018\n",
      " -1.1771094  -0.3832421   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.7988699523631044\n",
      "\n",
      "step 140\n",
      "observation: [ 0.678861    0.02576992  0.18209021 -0.14313893  0.20429318  0.44864318\n",
      " -1.2449954  -0.3858191   8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.8710093718459633\n",
      "\n",
      "step 141\n",
      "observation: [ 0.50914574  0.51932746  0.13117562 -0.19507168  0.1533786   0.39671043\n",
      " -1.29591    -0.43775183  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.142965739969164\n",
      "\n",
      "step 142\n",
      "observation: [ 0.8818593   0.38949558  0.0429897  -0.23402123  0.06519267  0.35776088\n",
      " -1.3840959  -0.4767014   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.5066547005313256\n",
      "\n",
      "step 143\n",
      "observation: [ 1.1613945   0.29212168 -0.07314976 -0.2632334  -0.05094678  0.3285487\n",
      " -1.5002353  -0.50591356  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.6420109359466792\n",
      "\n",
      "step 144\n",
      "observation: [ 0.37104586  0.21909127 -0.11025434 -0.28514254 -0.08805136  0.30663958\n",
      " -1.5373399  -0.5278227   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.5919482705886567\n",
      "\n",
      "step 145\n",
      "observation: [-0.2217156   0.16431844 -0.08808278 -0.30157438 -0.06587981  0.29020774\n",
      " -1.5151684  -0.54425454  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.4080468773876547\n",
      "\n",
      "step 146\n",
      "observation: [-6.6628671e-01  1.2323884e-01 -2.1454109e-02 -3.1389827e-01\n",
      "  7.4886438e-04  2.7788386e-01 -1.4485397e+00 -5.5657840e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -2.2220647249953127\n",
      "\n",
      "step 147\n",
      "observation: [-0.49971503 -0.40757087  0.02851739 -0.27314118  0.05072037  0.31864095\n",
      " -1.3985683  -0.51582134  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.139061219701711\n",
      "\n",
      "step 148\n",
      "observation: [-0.37478626  0.19432184  0.06599602 -0.29257336  0.088199    0.29920876\n",
      " -1.3610896  -0.5352535   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.1366305690015808\n",
      "\n",
      "step 149\n",
      "observation: [-0.2810897   0.6457414   0.09410499 -0.35714749  0.11630797  0.23463462\n",
      " -1.3329806  -0.59982765  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -2.2086433197506983\n",
      "\n",
      "step 150\n",
      "observation: [-0.21081728  0.98430604  0.11518672 -0.4555781   0.13738969  0.13620402\n",
      " -1.311899   -0.6982583   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -2.408061235874168\n",
      "\n",
      "step 151\n",
      "observation: [ 0.34188706  0.7382295   0.08099801 -0.52940106  0.10320099  0.06238106\n",
      " -1.3460876  -0.7720812   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -2.486066868558804\n",
      "\n",
      "step 152\n",
      "observation: [ 0.25641528  0.05367215  0.05535648 -0.5347683   0.07755946  0.05701385\n",
      " -1.3717291  -0.7774484   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -2.408875827401622\n",
      "\n",
      "step 153\n",
      "observation: [-0.30768853  0.04025411  0.08612534 -0.5387937   0.10832831  0.05298844\n",
      " -1.3409603  -0.7814738   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -2.2762970976461765\n",
      "\n",
      "step 154\n",
      "observation: [-0.2307664  -0.4698094   0.10920198 -0.49181274  0.13140495  0.09996938\n",
      " -1.3178836  -0.7344929   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.2804589593688895\n",
      "\n",
      "step 155\n",
      "observation: [-0.1730748  -0.35235706  0.12650946 -0.45657703  0.14871243  0.13520509\n",
      " -1.3005762  -0.6992572   7.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.9843434997738338\n",
      "\n",
      "step 156\n",
      "observation: [-0.6298061  -0.2642678   0.18949006 -0.43015024  0.21169305  0.16163187\n",
      " -1.2375956  -0.6728304   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.964915637218117\n",
      "\n",
      "step 157\n",
      "observation: [ 0.02764542 -0.19820085  0.18672553 -0.41033018  0.2089285   0.18145195\n",
      " -1.2403601  -0.6530103   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.829127018643641\n",
      "\n",
      "step 158\n",
      "observation: [-0.47926593 -0.14865063  0.23465212 -0.3954651   0.2568551   0.19631702\n",
      " -1.1924335  -0.63814527  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.6703906893739764\n",
      "\n",
      "step 159\n",
      "observation: [-0.35944945 -0.611488    0.27059707 -0.3343163   0.29280004  0.2574658\n",
      " -1.1564885  -0.57699645  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.4474889272202618\n",
      "\n",
      "step 160\n",
      "observation: [-0.7695871  -0.458616    0.3475558  -0.2884547   0.36975875  0.3033274\n",
      " -1.0795299  -0.53113484  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.1911655563561971\n",
      "\n",
      "step 161\n",
      "observation: [-1.0771903  -0.34396198  0.45527482 -0.2540585   0.4774778   0.3377236\n",
      " -0.9718108  -0.49673867  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.0652994458563096\n",
      "\n",
      "step 162\n",
      "observation: [-0.80789274  0.2420285   0.5360641  -0.27826136  0.55826706  0.31352076\n",
      " -0.89102155 -0.5209415   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.080235261690658\n",
      "\n",
      "step 163\n",
      "observation: [-0.60591954  0.18152139  0.596656   -0.29641348  0.618859    0.2953686\n",
      " -0.83042955 -0.5390937   8.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.0026868469258956\n",
      "\n",
      "step 164\n",
      "observation: [ 0.04556034  0.13614103  0.5921     -0.3100276   0.614303    0.28175452\n",
      " -0.8349856  -0.55270773  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.1061363078606272\n",
      "\n",
      "step 165\n",
      "observation: [ 0.53417027  0.10210577  0.538683   -0.32023817  0.56088597  0.27154395\n",
      " -0.88840264 -0.56291837  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.2829518398054054\n",
      "\n",
      "step 166\n",
      "observation: [ 0.9006277   0.07657933  0.4486202  -0.32789612  0.47082317  0.263886\n",
      " -0.97846544 -0.57057625  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.3241852946004782\n",
      "\n",
      "step 167\n",
      "observation: [ 0.17547077  0.0574345   0.43107313 -0.33363956  0.4532761   0.25814256\n",
      " -0.9960125  -0.5763197   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.2571405206295196\n",
      "\n",
      "step 168\n",
      "observation: [-0.36839694  0.04307587  0.46791282 -0.33794716  0.4901158   0.25383496\n",
      " -0.9591728  -0.5806273   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.3086625703591301\n",
      "\n",
      "step 169\n",
      "observation: [-0.2762977   0.03230691  0.4955426  -0.34117785  0.51774555  0.25060427\n",
      " -0.93154305 -0.583858    8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.2733197746482419\n",
      "\n",
      "step 170\n",
      "observation: [-0.20722327  0.02423018  0.5162649  -0.34360087  0.5384679   0.24818125\n",
      " -0.9108207  -0.586281    8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.2086938186323575\n",
      "\n",
      "step 171\n",
      "observation: [-0.15541746  0.5181726   0.53180665 -0.3954181   0.5540096   0.19636399\n",
      " -0.89527893 -0.6380983   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.3390654808353049\n",
      "\n",
      "step 172\n",
      "observation: [-0.11656309  0.38862947  0.543463   -0.43428105  0.56566596  0.15750104\n",
      " -0.88362265 -0.67696124  8.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.3640048626708743\n",
      "\n",
      "step 173\n",
      "observation: [-0.08742232  0.2914721   0.5522052  -0.4634283   0.5744082   0.12835383\n",
      " -0.87488043 -0.70610845  8.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.3592216410559566\n",
      "\n",
      "step 174\n",
      "observation: [-0.06556674  0.7186041   0.5587619  -0.5352887   0.58096486  0.05649343\n",
      " -0.86832374 -0.7779688   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.5263089151346676\n",
      "\n",
      "step 175\n",
      "observation: [ 0.45082495  0.53895307  0.5136794  -0.589184    0.53588235  0.00259812\n",
      " -0.91340625 -0.8318642   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.5733757975505291\n",
      "\n",
      "step 176\n",
      "observation: [ 0.3381187  -0.09578521  0.47986752 -0.57960546  0.5020705   0.01217664\n",
      " -0.9472181  -0.82228565  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.53128671038814\n",
      "\n",
      "step 177\n",
      "observation: [ 0.25358903 -0.5718389   0.4545086  -0.5224216   0.47671157  0.06936053\n",
      " -0.97257704 -0.76510173  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6065154195135276\n",
      "\n",
      "step 178\n",
      "observation: [ 0.69019175 -0.42887917  0.38548943 -0.47953367  0.4076924   0.11224845\n",
      " -1.0415962  -0.7222138   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.783439472295254\n",
      "\n",
      "step 179\n",
      "observation: [ 1.0176438  -0.3216594   0.28372505 -0.44736773  0.30592802  0.14441438\n",
      " -1.1433606  -0.69004786  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.866990380023959\n",
      "\n",
      "step 180\n",
      "observation: [ 0.7632329  -0.74124455  0.20740177 -0.37324327  0.22960474  0.21853884\n",
      " -1.2196839  -0.6159234   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.8193180072689628\n",
      "\n",
      "step 181\n",
      "observation: [ 0.07242466 -0.5559334   0.2001593  -0.31764993  0.22236226  0.2741322\n",
      " -1.2269263  -0.5603301   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6669531938185327\n",
      "\n",
      "step 182\n",
      "observation: [-0.4456815  -0.41695005  0.24472745 -0.27595493  0.26693043  0.3158272\n",
      " -1.1823581  -0.5186351   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6749615808539051\n",
      "\n",
      "step 183\n",
      "observation: [ 0.16573887 -0.31271255  0.22815356 -0.24468367  0.25035653  0.34709844\n",
      " -1.198932   -0.48736385  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.5639756418834816\n",
      "\n",
      "step 184\n",
      "observation: [-0.37569585 -0.2345344   0.26572314 -0.22123022  0.2879261   0.37055188\n",
      " -1.1613625  -0.4639104   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.59912923397978\n",
      "\n",
      "step 185\n",
      "observation: [ 0.21822812 -0.1759008   0.24390033 -0.20364015  0.2661033   0.38814196\n",
      " -1.1831853  -0.44632033  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.6723383636553795\n",
      "\n",
      "step 186\n",
      "observation: [ 0.16367109  0.3680744   0.22753322 -0.2404476   0.2497362   0.3513345\n",
      " -1.1995524  -0.48312774  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6106921928618463\n",
      "\n",
      "step 187\n",
      "observation: [-0.37724668  0.2760558   0.2652579  -0.26805317  0.28746086  0.32372895\n",
      " -1.1618277  -0.5107333   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6831790863544709\n",
      "\n",
      "step 188\n",
      "observation: [ 0.21706499  0.20704184  0.2435514  -0.28875735  0.26575437  0.30302477\n",
      " -1.1835343  -0.5314375   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.6865287248374816\n",
      "\n",
      "step 189\n",
      "observation: [ 0.16279873 -0.3447186   0.22727153 -0.25428548  0.2494745   0.3374966\n",
      " -1.1998141  -0.49696565  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.8146508670994324\n",
      "\n",
      "step 190\n",
      "observation: [ 0.62209904 -0.25853896  0.16506162 -0.2284316   0.18726459  0.3633505\n",
      " -1.262024   -0.47111174  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.7883311072161205\n",
      "\n",
      "step 191\n",
      "observation: [-0.03342571 -0.19390422  0.16840419 -0.20904118  0.19060716  0.38274094\n",
      " -1.2586814  -0.45172134  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.6459819863218494\n",
      "\n",
      "step 192\n",
      "observation: [-0.5250693  -0.14542817  0.22091112 -0.19449836  0.24311408  0.39728376\n",
      " -1.2061745  -0.43717852  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.5882434047064504\n",
      "\n",
      "step 193\n",
      "observation: [-0.39380196  0.39092886  0.2602913  -0.23359124  0.28249428  0.35819086\n",
      " -1.1667943  -0.47627142  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -1.6489807264905878\n",
      "\n",
      "step 194\n",
      "observation: [-0.29535148  0.29319665  0.28982645 -0.2629109   0.31202942  0.3288712\n",
      " -1.1372591  -0.5055911   8.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.4715488431108708\n",
      "\n",
      "step 195\n",
      "observation: [-0.2215136  -0.28010252  0.31197783 -0.23490067  0.3341808   0.35688144\n",
      " -1.1151078  -0.47758082  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.4633060188068845\n",
      "\n",
      "step 196\n",
      "observation: [-0.1661352   0.28992313  0.32859135 -0.26389298  0.35079432  0.32788914\n",
      " -1.0984943  -0.50657314  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -1.5139211267236108\n",
      "\n",
      "step 197\n",
      "observation: [-0.1246014   0.71744233  0.3410515  -0.3356372   0.36325446  0.2561449\n",
      " -1.0860342  -0.57831734  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.6690108792514367\n",
      "\n",
      "step 198\n",
      "observation: [ 0.40654895  0.53808177  0.3003966  -0.3894454   0.32259956  0.20233673\n",
      " -1.1266891  -0.63212556  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -1.909515648006728\n",
      "\n",
      "step 199\n",
      "observation: [ 0.80491173  0.40356132  0.21990542 -0.42980152  0.24210839  0.1619806\n",
      " -1.2071803  -0.67248166  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -2.2005352474360134\n",
      "\n",
      "step 0\n",
      "observation: [ 0.          0.         -1.2117513   0.80807555 -0.82522124  0.27134967\n",
      "  0.3380344   0.16979893  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.6745986094126206\n",
      "\n",
      "step 1\n",
      "observation: [-0.5         0.         -1.1617514   0.80807555 -0.7752212   0.27134967\n",
      "  0.3880344   0.16979893  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.5465911436359415\n",
      "\n",
      "step 2\n",
      "observation: [-0.875       0.         -1.0742513   0.80807555 -0.68772125  0.27134967\n",
      "  0.4755344   0.16979893  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.4009247491274724\n",
      "\n",
      "step 3\n",
      "observation: [-1.15625     0.         -0.95862633  0.80807555 -0.5720962   0.27134967\n",
      "  0.5911594   0.16979893  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.28458698223027123\n",
      "\n",
      "step 4\n",
      "observation: [-0.8671875   0.5        -0.8719076   0.75807554 -0.4853775   0.22134969\n",
      "  0.67787814  0.11979893  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.1946001248232591\n",
      "\n",
      "step 5\n",
      "observation: [-0.6503906   0.875      -0.80686855  0.67057556 -0.42033842  0.13384968\n",
      "  0.74291724  0.03229893  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.15203325752094055\n",
      "\n",
      "step 6\n",
      "observation: [-0.48779297  0.15625    -0.75808924  0.65495056 -0.3715591   0.11822468\n",
      "  0.7916965   0.01667392  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11540093702060268\n",
      "\n",
      "step 7\n",
      "observation: [-0.36584473  0.6171875  -0.72150475  0.5932318  -0.33497465  0.05650593\n",
      "  0.828281   -0.04504482  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09820464489170283\n",
      "\n",
      "step 8\n",
      "observation: [-0.27438354 -0.03710938 -0.6940664   0.5969427  -0.3075363   0.06021687\n",
      "  0.8557193  -0.04133389  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09511364053379773\n",
      "\n",
      "step 9\n",
      "observation: [-0.20578766 -0.52783203 -0.67348766  0.6497259  -0.28695753  0.11300007\n",
      "  0.8762981   0.01144932  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1147666718279977\n",
      "\n",
      "step 10\n",
      "observation: [-0.15434074 -0.895874   -0.6580536   0.7393133  -0.27152345  0.20258747\n",
      "  0.89173216  0.10103672  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11587529146096714\n",
      "\n",
      "step 11\n",
      "observation: [-0.11575556 -0.17190552 -0.646478    0.7565039  -0.2599479   0.21977803\n",
      "  0.90330774  0.11822727  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.14303758011377538\n",
      "\n",
      "step 12\n",
      "observation: [-0.08681667 -0.62892914 -0.63779634  0.8193968  -0.25126624  0.28267094\n",
      "  0.9119894   0.18112019  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.19567533800241332\n",
      "\n",
      "step 13\n",
      "observation: [ 0.4348875  -0.47169685 -0.6812851   0.8665665  -0.29475498  0.32984063\n",
      "  0.86850065  0.22828987  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.20653444642825874\n",
      "\n",
      "step 14\n",
      "observation: [ 0.32616562  0.14622736 -0.71390164  0.85194373 -0.32737154  0.31521788\n",
      "  0.8358841   0.21366714  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.31635571246952965\n",
      "\n",
      "step 15\n",
      "observation: [ 0.24462421  0.10967052 -0.7383641   0.8409767  -0.35183397  0.30425084\n",
      "  0.8114217   0.20270008  3.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1901468593849583\n",
      "\n",
      "step 16\n",
      "observation: [-0.31653184  0.08225289 -0.7067109   0.8327514  -0.32018077  0.29602554\n",
      "  0.84307486  0.19447479  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2718940425527133\n",
      "\n",
      "step 17\n",
      "observation: [-0.23739888  0.06168967 -0.682971    0.82658243 -0.2964409   0.28985658\n",
      "  0.86681473  0.18830582  3.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1336304823131959\n",
      "\n",
      "step 18\n",
      "observation: [-0.67804915  0.04626725 -0.61516607  0.82195574 -0.22863598  0.28522986\n",
      "  0.93461967  0.1836791   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.08531915447441295\n",
      "\n",
      "step 19\n",
      "observation: [-0.5085369   0.53470045 -0.5643124   0.76848567 -0.1777823   0.23175982\n",
      "  0.98547333  0.13020906  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07789813678720019\n",
      "\n",
      "step 20\n",
      "observation: [-0.38140264 -0.09897467 -0.52617216  0.77838314 -0.13964203  0.24165727\n",
      "  1.0236136   0.14010653  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17437021310317585\n",
      "\n",
      "step 21\n",
      "observation: [-0.286052   -0.07423101 -0.49756694  0.78580624 -0.11103683  0.24908037\n",
      "  1.0522188   0.14752962  3.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.06641226206136601\n",
      "\n",
      "step 22\n",
      "observation: [-0.714539   -0.05567325 -0.42611304  0.79137355 -0.03958293  0.2546477\n",
      "  1.1236727   0.15309694  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04380333832316009\n",
      "\n",
      "step 23\n",
      "observation: [-0.5359042   0.45824507 -0.37252262  0.7455491   0.01400749  0.2088232\n",
      "  1.1772631   0.10727245  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.030452126655049535\n",
      "\n",
      "step 24\n",
      "observation: [ 0.09807182  0.3436838  -0.3823298   0.7111807   0.00420031  0.17445482\n",
      "  1.1674559   0.07290407  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12211526120584208\n",
      "\n",
      "step 25\n",
      "observation: [ 0.07355386  0.25776285 -0.38968518  0.6854044  -0.00315507  0.14867854\n",
      "  1.1601006   0.04712778  3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.020172829109744998\n",
      "\n",
      "step 26\n",
      "observation: [ 0.5551654   0.19332214 -0.44520172  0.6660722  -0.05867161  0.12934633\n",
      "  1.104584    0.02779557  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12325176965566824\n",
      "\n",
      "step 27\n",
      "observation: [ 0.41637406  0.1449916  -0.48683915  0.651573   -0.10030902  0.11484716\n",
      "  1.0629466   0.01329641  3.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.01745863544034287\n",
      "\n",
      "step 28\n",
      "observation: [-0.18771946  0.1087437  -0.4680672   0.6406987  -0.08153708  0.10397279\n",
      "  1.0817186   0.00242204  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.022977308260170526\n",
      "\n",
      "step 29\n",
      "observation: [ 0.3592104   0.08155777 -0.5039882   0.63254285 -0.11745811  0.09581701\n",
      "  1.0457975  -0.00573374  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.016957273278657203\n",
      "\n",
      "step 30\n",
      "observation: [-0.23059219  0.06116833 -0.48092902  0.62642604 -0.09439889  0.08970018\n",
      "  1.0688567  -0.01185057  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.024200503132381514\n",
      "\n",
      "step 31\n",
      "observation: [-0.17294414 -0.45412377 -0.46363458  0.6718384  -0.07710448  0.13511255\n",
      "  1.0861511   0.0335618   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.018315053876472337\n",
      "\n",
      "step 32\n",
      "observation: [-0.12970811  0.15940718 -0.45066378  0.6558977  -0.06413367  0.11917184\n",
      "  1.0991219   0.01762108  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02767692906476981\n",
      "\n",
      "step 33\n",
      "observation: [-0.09728108 -0.38044462 -0.44093567  0.6939421  -0.05440556  0.1572163\n",
      "  1.1088501   0.05566554  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.03451128522182872\n",
      "\n",
      "step 34\n",
      "observation: [-0.5729608  -0.28533345 -0.3836396   0.7224755   0.00289052  0.18574965\n",
      "  1.1661462   0.08419888  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1450143553241492\n",
      "\n",
      "step 35\n",
      "observation: [-0.4297206  -0.21400009 -0.34066752  0.7438755   0.04586259  0.20714965\n",
      "  1.2091182   0.1055989   3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03609642483945266\n",
      "\n",
      "step 36\n",
      "observation: [-0.32229045  0.33949992 -0.30843848  0.70992553  0.07809163  0.17319965\n",
      "  1.2413473   0.0716489   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.020010359317193915\n",
      "\n",
      "step 37\n",
      "observation: [-0.24171785  0.75462496 -0.2842667   0.634463    0.10226341  0.09773716\n",
      "  1.265519   -0.00381359  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11618681821898\n",
      "\n",
      "step 38\n",
      "observation: [-0.18128838  0.5659687  -0.26613787  0.57786614  0.12039226  0.0411403\n",
      "  1.2836479  -0.06041046  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11795472941977714\n",
      "\n",
      "step 39\n",
      "observation: [-1.3596629e-01  4.2447653e-01 -2.5254124e-01  5.3541851e-01\n",
      "  1.3398889e-01 -1.3073592e-03  1.2972445e+00 -1.0285811e-01\n",
      "  3.0000000e+00]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.009969534021875845\n",
      "\n",
      "step 40\n",
      "observation: [ 0.39802527  0.3183574  -0.29234377  0.5035828   0.09418635 -0.0331431\n",
      "  1.257442   -0.13469386  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.003456746106142958\n",
      "\n",
      "step 41\n",
      "observation: [ 0.79851896  0.23876806 -0.37219566  0.47970596  0.01433446 -0.0570199\n",
      "  1.1775901  -0.15857066  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.014744787447662226\n",
      "\n",
      "step 42\n",
      "observation: [ 1.0988892   0.17907603 -0.48208457  0.46179834 -0.09555446 -0.07492751\n",
      "  1.0677012  -0.17647827  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03314508536533878\n",
      "\n",
      "step 43\n",
      "observation: [ 0.8241669  -0.36569297 -0.5645013   0.49836764 -0.17797115 -0.03835821\n",
      "  0.98528445 -0.13990897  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.036137335886264936\n",
      "\n",
      "step 44\n",
      "observation: [ 0.11812519 -0.27426973 -0.5763138   0.5257946  -0.18978368 -0.01093124\n",
      "  0.97347194 -0.112482    0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.13955197773256298\n",
      "\n",
      "step 45\n",
      "observation: [ 0.08859389 -0.20570229 -0.5851732   0.54636484 -0.19864306  0.00963899\n",
      "  0.96461254 -0.09191176  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14277133946731344\n",
      "\n",
      "step 46\n",
      "observation: [ 0.06644542 -0.15427673 -0.59181774  0.5617925  -0.2052876   0.02506666\n",
      "  0.957968   -0.07648409  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14555619971526643\n",
      "\n",
      "step 47\n",
      "observation: [ 0.04983407 -0.11570754 -0.5968011   0.57336324 -0.21027102  0.03663742\n",
      "  0.95298463 -0.06491334  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.0458216121657395\n",
      "\n",
      "step 48\n",
      "observation: [ 0.03737555  0.41321933 -0.6005387   0.5320413  -0.21400857 -0.00468452\n",
      "  0.94924706 -0.10623527  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05434769962892325\n",
      "\n",
      "step 49\n",
      "observation: [ 0.02803166  0.8099145  -0.6033419   0.4510499  -0.21681173 -0.08567597\n",
      "  0.9464439  -0.18722673  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05722011783431563\n",
      "\n",
      "step 50\n",
      "observation: [ 0.02102374  0.10743588 -0.60544425  0.4403063  -0.2189141  -0.09641956\n",
      "  0.94434154 -0.19797032  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05158400301239272\n",
      "\n",
      "step 51\n",
      "observation: [ 0.01576781 -0.4194231  -0.60702103  0.4822486  -0.22049089 -0.05447724\n",
      "  0.94276476 -0.156028    0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1496690740127561\n",
      "\n",
      "step 52\n",
      "observation: [ 0.01182586 -0.31456733 -0.6082036   0.5137053  -0.22167347 -0.02302051\n",
      "  0.94158214 -0.12457127  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.029777424120390694\n",
      "\n",
      "step 53\n",
      "observation: [-4.91130620e-01 -2.35925481e-01 -5.59090555e-01  5.37297904e-01\n",
      " -1.72560409e-01  5.72034507e-04  9.90695238e-01 -1.00978725e-01\n",
      "  0.00000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.01942846227453407\n",
      "\n",
      "step 54\n",
      "observation: [-0.36834794  0.3230559  -0.5222557   0.5049923  -0.13572562 -0.03173355\n",
      "  1.02753    -0.13328432  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11481733572301303\n",
      "\n",
      "step 55\n",
      "observation: [-0.27626097  0.24229191 -0.49462965  0.4807631  -0.10809952 -0.05596275\n",
      "  1.0551561  -0.1575135   4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.023044664271749088\n",
      "\n",
      "step 56\n",
      "observation: [-0.20719573  0.68171895 -0.47391006  0.41259122 -0.08737995 -0.12413464\n",
      "  1.0758756  -0.22568539  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.020851983719181087\n",
      "\n",
      "step 57\n",
      "observation: [-0.15539679  0.0112892  -0.4583704   0.4114623  -0.07184027 -0.12526356\n",
      "  1.0914154  -0.22681431  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03463711556881082\n",
      "\n",
      "step 58\n",
      "observation: [-0.11654759  0.5084669  -0.44671562  0.3606156  -0.06018551 -0.17611025\n",
      "  1.1030701  -0.277661    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02962303815844151\n",
      "\n",
      "step 59\n",
      "observation: [-0.0874107  -0.11864983 -0.43797454  0.3724806  -0.05144444 -0.16424526\n",
      "  1.1118112  -0.26579604  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1261475342408393\n",
      "\n",
      "step 60\n",
      "observation: [-0.06555802 -0.08898737 -0.43141875  0.38137934 -0.04488864 -0.15534653\n",
      "  1.118367   -0.25689727  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04106849695653185\n",
      "\n",
      "step 61\n",
      "observation: [-0.04916852  0.4332595  -0.4265019   0.33805338 -0.03997179 -0.19867247\n",
      "  1.1232839  -0.30022323  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15475469249837503\n",
      "\n",
      "step 62\n",
      "observation: [-0.03687639  0.32494462 -0.42281425  0.30555892 -0.03628415 -0.23116694\n",
      "  1.1269715  -0.3327177   4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.16642304300136135\n",
      "\n",
      "step 63\n",
      "observation: [-0.02765729  0.24370845 -0.42004853  0.28118807 -0.03351842 -0.25553778\n",
      "  1.1297373  -0.35708854  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.0753194769510537\n",
      "\n",
      "step 64\n",
      "observation: [-0.52074295  0.18278134 -0.36797425  0.26290995  0.01855588 -0.27381593\n",
      "  1.1818115  -0.37536666  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.059736992786226996\n",
      "\n",
      "step 65\n",
      "observation: [-0.39055723 -0.362914   -0.32891852  0.29920134  0.0576116  -0.23752452\n",
      "  1.2208673  -0.33907527  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04559046042363401\n",
      "\n",
      "step 66\n",
      "observation: [ 0.20708208 -0.2721855  -0.34962672  0.3264199   0.03690339 -0.21030597\n",
      "  1.2001591  -0.31185672  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.020026559689753736\n",
      "\n",
      "step 67\n",
      "observation: [ 0.15531155 -0.7041391  -0.36515787  0.3968338   0.02137223 -0.13989206\n",
      "  1.1846279  -0.24144281  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.018885924586003373\n",
      "\n",
      "step 68\n",
      "observation: [ 0.11648367 -0.02810434 -0.37680626  0.39964423  0.00972387 -0.13708162\n",
      "  1.1729795  -0.23863238  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.00722152134134483\n",
      "\n",
      "step 69\n",
      "observation: [ 8.7362751e-02 -5.2107823e-01 -3.8554251e-01  4.5175207e-01\n",
      "  9.8759285e-04 -8.4973797e-02  1.1642432e+00 -1.8652456e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -4.783297564817705e-05\n",
      "\n",
      "step 70\n",
      "observation: [ 0.06552207 -0.8908087  -0.39209473  0.54083294 -0.00556461  0.00410707\n",
      "  1.157691   -0.09744368  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.006591251479524079\n",
      "\n",
      "step 71\n",
      "observation: [-0.45085844 -0.6681065  -0.34700888  0.6076436   0.03952123  0.07091773\n",
      "  1.2027769  -0.03063303  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.010422764542706402\n",
      "\n",
      "step 72\n",
      "observation: [-3.3814383e-01 -1.0798892e-03 -3.1319451e-01  6.0775155e-01\n",
      "  7.3335618e-02  7.1025714e-02  1.2365912e+00 -3.0525042e-02\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11479714362638087\n",
      "\n",
      "step 73\n",
      "observation: [-2.5360787e-01 -8.0991699e-04 -2.8783372e-01  6.0783255e-01\n",
      "  9.8696403e-02  7.1106702e-02  1.2619520e+00 -3.0444050e-02\n",
      "  3.0000000e+00]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.009650397179213878\n",
      "\n",
      "step 74\n",
      "observation: [ 3.0979410e-01 -6.0743775e-04 -3.1881312e-01  6.0789329e-01\n",
      "  6.7716993e-02  7.1167447e-02  1.2309726e+00 -3.0383306e-02\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.016671280393764303\n",
      "\n",
      "step 75\n",
      "observation: [ 0.23234557 -0.50045556 -0.3420477   0.65793884  0.04448244  0.121213\n",
      "  1.207738    0.01966225  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.031138373050823377\n",
      "\n",
      "step 76\n",
      "observation: [-0.3257408  -0.37534168 -0.3094736   0.695473    0.07705652  0.15874718\n",
      "  1.2403121   0.05719642  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.03758170797293503\n",
      "\n",
      "step 77\n",
      "observation: [ 0.2556944  -0.28150627 -0.33504304  0.72362363  0.05148708  0.1868978\n",
      "  1.2147427   0.08534705  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.02601134027418881\n",
      "\n",
      "step 78\n",
      "observation: [ 0.19177079  0.2888703  -0.35422012  0.6947366   0.03231     0.15801077\n",
      "  1.1955656   0.05646002  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11891147938496358\n",
      "\n",
      "step 79\n",
      "observation: [ 0.1438281   0.21665272 -0.36860293  0.6730714   0.01792719  0.1363455\n",
      "  1.1811829   0.03479474  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11447416084076402\n",
      "\n",
      "step 80\n",
      "observation: [ 0.10787107  0.16248955 -0.37939003  0.6568224   0.00714009  0.12009654\n",
      "  1.1703957   0.01854579  3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.014240458348206746\n",
      "\n",
      "step 81\n",
      "observation: [ 0.5809033   0.12186716 -0.43748036  0.6446357  -0.05095024  0.10790983\n",
      "  1.1123054   0.00635907  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.030640921285953523\n",
      "\n",
      "step 82\n",
      "observation: [ 0.93567747  0.09140037 -0.5310481   0.63549566 -0.14451799  0.09876979\n",
      "  1.0187377  -0.00278096  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.03557237193272152\n",
      "\n",
      "step 83\n",
      "observation: [ 0.2017581   0.06855027 -0.55122393  0.6286406  -0.1646938   0.09191477\n",
      "  0.99856186 -0.00963599  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.02438434086132167\n",
      "\n",
      "step 84\n",
      "observation: [-0.34868142  0.05141271 -0.51635575  0.62349933 -0.12982565  0.08677349\n",
      "  1.03343    -0.01477726  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.028415485307387807\n",
      "\n",
      "step 85\n",
      "observation: [-0.26151106 -0.46144047 -0.49020466  0.6696434  -0.10367455  0.13291754\n",
      "  1.059581    0.03136678  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.02087855000001906\n",
      "\n",
      "step 86\n",
      "observation: [-0.1961333   0.15391965 -0.47059134  0.65425146 -0.08406122  0.11752558\n",
      "  1.0791944   0.01597482  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.007943532168908574\n",
      "\n",
      "step 87\n",
      "observation: [-0.14709997  0.6154397  -0.45588133  0.59270746 -0.06935123  0.0559816\n",
      "  1.0939044  -0.04556916  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.005015215331194657\n",
      "\n",
      "step 88\n",
      "observation: [-0.11032498  0.9615798  -0.44484884  0.4965495  -0.05831873 -0.04017638\n",
      "  1.104937   -0.14172713  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.028844060106887708\n",
      "\n",
      "step 89\n",
      "observation: [-0.08274373  1.2211848  -0.43657446  0.37443098 -0.05004435 -0.16229486\n",
      "  1.1132113  -0.26384562  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.16637876763552759\n",
      "\n",
      "step 90\n",
      "observation: [-0.0620578   0.91588867 -0.4303687   0.28284213 -0.04383857 -0.25388372\n",
      "  1.1194171  -0.35543448  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.20559027737712188\n",
      "\n",
      "step 91\n",
      "observation: [-0.04654335  0.6869165  -0.42571434  0.21415047 -0.03918424 -0.3225754\n",
      "  1.1240714  -0.42412612  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.24122042917431943\n",
      "\n",
      "step 92\n",
      "observation: [-0.03490751  0.5151874  -0.4222236   0.16263175 -0.03569349 -0.3740941\n",
      "  1.1275622  -0.47564486  4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.17725019008799803\n",
      "\n",
      "step 93\n",
      "observation: [ 0.47381938  0.3863905  -0.46960554  0.12399269 -0.08307543 -0.41273317\n",
      "  1.0801802  -0.5142839   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.16750742236322325\n",
      "\n",
      "step 94\n",
      "observation: [ 0.35536453 -0.2102071  -0.505142    0.1450134  -0.11861187 -0.39171246\n",
      "  1.0446438  -0.4932632   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.12734308620488788\n",
      "\n",
      "step 95\n",
      "observation: [ 0.2665234  -0.65765536 -0.5317943   0.21077894 -0.14526421 -0.32594693\n",
      "  1.0179914  -0.42749768  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08980351860392125\n",
      "\n",
      "step 96\n",
      "observation: [-0.30010745 -0.4932415  -0.5017836   0.26010308 -0.11525347 -0.27662277\n",
      "  1.0480021  -0.37817353  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.05924954241582952\n",
      "\n",
      "step 97\n",
      "observation: [-0.7250806  -0.36993113 -0.4292755   0.2970962  -0.04274541 -0.23962966\n",
      "  1.1205102  -0.3411804   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.048694129243109015\n",
      "\n",
      "step 98\n",
      "observation: [-1.0438105  -0.27744836 -0.3248945   0.32484102  0.06163563 -0.21188483\n",
      "  1.2248913  -0.31343558  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04459597410848844\n",
      "\n",
      "step 99\n",
      "observation: [-0.28285784 -0.20808625 -0.2966087   0.34564966  0.08992141 -0.19107619\n",
      "  1.253177   -0.29262695  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02809380837630117\n",
      "\n",
      "step 100\n",
      "observation: [-0.21214338 -0.6560647  -0.27539435  0.41125613  0.11113575 -0.12546973\n",
      "  1.2743914  -0.22702049  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.016830658787199842\n",
      "\n",
      "step 101\n",
      "observation: [-0.15910754 -0.9920485  -0.2594836   0.510461    0.12704651 -0.02626487\n",
      "  1.2903022  -0.12781563  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.038030619422449355\n",
      "\n",
      "step 102\n",
      "observation: [-0.61933064 -0.7440364  -0.19755054  0.5848646   0.18897957  0.04813876\n",
      "  1.3522352  -0.05341199  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.058336673479267825\n",
      "\n",
      "step 103\n",
      "observation: [-0.46449798 -0.05802729 -0.15110074  0.59066737  0.23542938  0.05394149\n",
      "  1.398685   -0.04760926  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.0519155647997133\n",
      "\n",
      "step 104\n",
      "observation: [ 0.15162651 -0.04352047 -0.1662634   0.5950194   0.22026671  0.05829354\n",
      "  1.3835224  -0.04325721  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.07081581724043041\n",
      "\n",
      "step 105\n",
      "observation: [-0.38628012 -0.03264035 -0.12763539  0.5982834   0.25889474  0.06155758\n",
      "  1.4221504  -0.03999318  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.06067682760410647\n",
      "\n",
      "step 106\n",
      "observation: [ 0.21028991 -0.02448026 -0.14866437  0.60073143  0.23786575  0.0640056\n",
      "  1.4011214  -0.03754516  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15366086271408608\n",
      "\n",
      "step 107\n",
      "observation: [ 0.15771744 -0.0183602  -0.16443612  0.6025675   0.222094    0.06584162\n",
      "  1.3853496  -0.03570914  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14872979504875847\n",
      "\n",
      "step 108\n",
      "observation: [ 0.11828808 -0.01377015 -0.17626493  0.6039445   0.21026519  0.06721864\n",
      "  1.3735209  -0.03433212  3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04089248918116198\n",
      "\n",
      "step 109\n",
      "observation: [ 0.08871606  0.4896724  -0.18513653  0.55497724  0.20139359  0.0182514\n",
      "  1.3646492  -0.08329936  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03891750743855119\n",
      "\n",
      "step 110\n",
      "observation: [ 0.06653704 -0.13274571 -0.19179024  0.56825185  0.19473988  0.03152597\n",
      "  1.3579955  -0.07002479  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03607746937763516\n",
      "\n",
      "step 111\n",
      "observation: [ 0.04990278  0.40044072 -0.19678052  0.5282078   0.1897496  -0.0085181\n",
      "  1.3530053  -0.11006886  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.01998406666060268\n",
      "\n",
      "step 112\n",
      "observation: [ 0.53742707  0.30033055 -0.2505232   0.4981747   0.13600689 -0.03855116\n",
      "  1.2992625  -0.14010191  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11288873488682806\n",
      "\n",
      "step 113\n",
      "observation: [ 0.4030703   0.2252479  -0.29083025  0.4756499   0.09569986 -0.06107595\n",
      "  1.2589555  -0.1626267   3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.006318557422860145\n",
      "\n",
      "step 114\n",
      "observation: [ 0.8023027   0.16893592 -0.37106052  0.45875633  0.01546959 -0.07796954\n",
      "  1.1787252  -0.1795203   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.01718424205840732\n",
      "\n",
      "step 115\n",
      "observation: [ 1.101727    0.12670195 -0.48123324  0.4460861  -0.09470312 -0.09063973\n",
      "  1.0685525  -0.1921905   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03396112583955875\n",
      "\n",
      "step 116\n",
      "observation: [ 0.8262953  -0.40497354 -0.56386274  0.48658347 -0.17733264 -0.05014238\n",
      "  0.985923   -0.15169314  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15765761176054263\n",
      "\n",
      "step 117\n",
      "observation: [ 0.6197215  -0.30373016 -0.6258349   0.5169565  -0.2393048  -0.01976936\n",
      "  0.92395085 -0.12132012  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.055603111388555844\n",
      "\n",
      "step 118\n",
      "observation: [-0.0352089  -0.22779761 -0.62231404  0.5397363  -0.2357839   0.0030104\n",
      "  0.92747176 -0.09854036  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05926910772168677\n",
      "\n",
      "step 119\n",
      "observation: [-0.02640668 -0.6708482  -0.6196734   0.60682106 -0.23314324  0.07009522\n",
      "  0.9301124  -0.03145554  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.16793449707570526\n",
      "\n",
      "step 120\n",
      "observation: [-0.01980501 -0.50313616 -0.6176928   0.6571347  -0.23116274  0.12040883\n",
      "  0.9320929   0.01885808  3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.10322896434019518\n",
      "\n",
      "step 121\n",
      "observation: [ 0.48514625 -0.37735212 -0.6662075   0.6948699  -0.27967736  0.15814404\n",
      "  0.8835783   0.05659329  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.10555160171437615\n",
      "\n",
      "step 122\n",
      "observation: [-0.13614032 -0.2830141  -0.65259343  0.7231713  -0.26606333  0.18644546\n",
      "  0.8971923   0.0848947   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09032096265674205\n",
      "\n",
      "step 123\n",
      "observation: [-0.10210524  0.28773943 -0.6423829   0.6943974  -0.25585282  0.15767151\n",
      "  0.9074028   0.05612075  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.06901238345972716\n",
      "\n",
      "step 124\n",
      "observation: [-0.07657893  0.7158046  -0.63472503  0.6228169  -0.24819492  0.08609106\n",
      "  0.9150607  -0.0154597   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05909228488793241\n",
      "\n",
      "step 125\n",
      "observation: [-0.0574342   1.0368534  -0.6289816   0.51913154 -0.24245149 -0.01759429\n",
      "  0.92080414 -0.11914504  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05876990512092506\n",
      "\n",
      "step 126\n",
      "observation: [-0.04307565  0.27764007 -0.624674    0.49136755 -0.23814392 -0.0453583\n",
      "  0.9251117  -0.14690906  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05544607125028136\n",
      "\n",
      "step 127\n",
      "observation: [-0.03230673 -0.29176995 -0.6214434   0.5205445  -0.23491326 -0.0161813\n",
      "  0.9283424  -0.11773206  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15408422206344657\n",
      "\n",
      "step 128\n",
      "observation: [-0.02423005 -0.21882746 -0.61902034  0.5424273  -0.23249024  0.00570144\n",
      "  0.9307654  -0.09584931  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1536990374199037\n",
      "\n",
      "step 129\n",
      "observation: [-0.01817254 -0.1641206  -0.6172031   0.5588394  -0.230673    0.0221135\n",
      "  0.9325826  -0.07943726  4.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05971026741958949\n",
      "\n",
      "step 130\n",
      "observation: [-0.0136294  -0.62309045 -0.6158402   0.6211484  -0.22931005  0.08442254\n",
      "  0.9339456  -0.01712821  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.04898801549629605\n",
      "\n",
      "step 131\n",
      "observation: [-0.5102221  -0.46731782 -0.56481797  0.6678802  -0.17828785  0.13115433\n",
      "  0.98496777  0.02960357  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.06373154762668148\n",
      "\n",
      "step 132\n",
      "observation: [ 0.11733346 -0.35048836 -0.5765513   0.702929   -0.1900212   0.16620317\n",
      "  0.9732344   0.06465241  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09833117266479652\n",
      "\n",
      "step 133\n",
      "observation: [ 0.0880001  -0.76286626 -0.58535135  0.77921563 -0.1988212   0.2424898\n",
      "  0.96443444  0.14093904  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11397870065509905\n",
      "\n",
      "step 134\n",
      "observation: [-0.43399993 -0.5721497  -0.5419513   0.8364306  -0.15542121  0.29970476\n",
      "  1.0078344   0.19815402  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.12269593484955024\n",
      "\n",
      "step 135\n",
      "observation: [-0.82549995 -0.4291123  -0.45940134  0.87934184 -0.07287122  0.342616\n",
      "  1.0903844   0.24106523  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.10561475490055267\n",
      "\n",
      "step 136\n",
      "observation: [-0.61912495  0.1781658  -0.39748883  0.8615253  -0.01095872  0.32479942\n",
      "  1.1522969   0.22324866  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.06960781790947455\n",
      "\n",
      "step 137\n",
      "observation: [-0.46434373  0.6336243  -0.35105446  0.7981628   0.03547565  0.26143697\n",
      "  1.1987313   0.15988623  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03181046991606795\n",
      "\n",
      "step 138\n",
      "observation: [-0.34825778  0.97521824 -0.3162287   0.70064104  0.07030143  0.16391516\n",
      "  1.2335571   0.0623644   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1175368444285867\n",
      "\n",
      "step 139\n",
      "observation: [-0.26119334  0.73141366 -0.29010934  0.62749964  0.09642076  0.09077379\n",
      "  1.2596763  -0.01077697  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11474846716098594\n",
      "\n",
      "step 140\n",
      "observation: [-0.195895    0.54856026 -0.27051985  0.57264364  0.11601026  0.03591776\n",
      "  1.2792659  -0.06563299  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1171104077497911\n",
      "\n",
      "step 141\n",
      "observation: [-0.14692126  0.4114202  -0.25582772  0.5315016   0.13070239 -0.00522426\n",
      "  1.2939581  -0.10677502  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12138680143236695\n",
      "\n",
      "step 142\n",
      "observation: [-0.11019094  0.30856514 -0.24480863  0.5006451   0.14172149 -0.03608077\n",
      "  1.304977   -0.13763154  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12600312445484557\n",
      "\n",
      "step 143\n",
      "observation: [-0.0826432   0.23142387 -0.23654431  0.4775027   0.1499858  -0.05922316\n",
      "  1.3132415  -0.16077392  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.13025794483701209\n",
      "\n",
      "step 144\n",
      "observation: [-0.0619824   0.17356789 -0.23034607  0.46014592  0.15618405 -0.07657995\n",
      "  1.3194396  -0.1781307   3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04535463953340099\n",
      "\n",
      "step 145\n",
      "observation: [-0.0464868   0.63017595 -0.22569738  0.3971283   0.16083273 -0.13959754\n",
      "  1.3240883  -0.2411483   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.04573167257269839\n",
      "\n",
      "step 146\n",
      "observation: [-0.0348651  -0.02736806 -0.22221088  0.39986512  0.16431923 -0.13686074\n",
      "  1.3275748  -0.23841149  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03505941933182025\n",
      "\n",
      "step 147\n",
      "observation: [-0.02614883 -0.52052605 -0.219596    0.45191774  0.16693412 -0.08480813\n",
      "  1.3301897  -0.18635888  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.01623085979459664\n",
      "\n",
      "step 148\n",
      "observation: [ 0.48038837 -0.39039454 -0.26763484  0.49095717  0.11889528 -0.04576868\n",
      "  1.2821509  -0.14731944  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.017925304521089185\n",
      "\n",
      "step 149\n",
      "observation: [-0.13970871 -0.2927959  -0.25366396  0.5202368   0.13286616 -0.01648909\n",
      "  1.2961218  -0.11803985  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.008743086981486884\n",
      "\n",
      "step 150\n",
      "observation: [ 0.39521846 -0.21959692 -0.2931858   0.54219645  0.09334431  0.0054706\n",
      "  1.2565999  -0.09608015  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10453944214453977\n",
      "\n",
      "step 151\n",
      "observation: [ 0.29641384 -0.16469769 -0.3228272   0.5586662   0.06370292  0.02194037\n",
      "  1.2269585  -0.07961039  3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.001966636335195787\n",
      "\n",
      "step 152\n",
      "observation: [ 0.22231038  0.37647673 -0.34505823  0.52101856  0.04147188 -0.0157073\n",
      "  1.2047275  -0.11725806  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.0025661024440577105\n",
      "\n",
      "step 153\n",
      "observation: [ 0.6667328   0.28235754 -0.4117315   0.4927828  -0.0252014  -0.04394305\n",
      "  1.1380543  -0.1454938   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10989659348656172\n",
      "\n",
      "step 154\n",
      "observation: [ 0.5000496   0.21176817 -0.46173647  0.471606   -0.07520635 -0.06511987\n",
      "  1.0880493  -0.16667062  3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.029865211515281515\n",
      "\n",
      "step 155\n",
      "observation: [ 0.3750372   0.6588261  -0.4992402   0.40572336 -0.11271007 -0.13100249\n",
      "  1.0505456  -0.23255324  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03684323072807304\n",
      "\n",
      "step 156\n",
      "observation: [ 0.2812779  -0.00588041 -0.52736795  0.40631142 -0.14083786 -0.13041444\n",
      "  1.0224178  -0.2319652   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1431156129552332\n",
      "\n",
      "step 157\n",
      "observation: [ 0.21095842 -0.00441031 -0.5484638   0.40675244 -0.1619337  -0.12997341\n",
      "  1.0013219  -0.23152417  4.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03793999894618814\n",
      "\n",
      "step 158\n",
      "observation: [ 0.15821882 -0.50330776 -0.5642857   0.45708323 -0.1777556  -0.07964264\n",
      "  0.98550004 -0.1811934   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05917385660765718\n",
      "\n",
      "step 159\n",
      "observation: [ 0.6186641  -0.3774808  -0.6261521   0.4948313  -0.239622   -0.04189456\n",
      "  0.92363364 -0.14344531  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08313463609176222\n",
      "\n",
      "step 160\n",
      "observation: [ 0.46399808 -0.7831106  -0.67255193  0.57314235 -0.2860218   0.0364165\n",
      "  0.8772338  -0.06513426  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08239785745815933\n",
      "\n",
      "step 161\n",
      "observation: [-0.15200144 -0.58733296 -0.6573518   0.63187563 -0.27082166  0.09514979\n",
      "  0.89243394 -0.00640096  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.18667611912066032\n",
      "\n",
      "step 162\n",
      "observation: [-0.11400108 -0.44049972 -0.6459517   0.6759256  -0.25942156  0.13919976\n",
      "  0.9038341   0.03764901  3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12018931358748783\n",
      "\n",
      "step 163\n",
      "observation: [ 0.4144992  -0.33037478 -0.6874016   0.7089631  -0.30087146  0.17223725\n",
      "  0.86238414  0.07068649  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11831587959495973\n",
      "\n",
      "step 164\n",
      "observation: [-0.18912561 -0.24778108 -0.66848904  0.7337412  -0.2819589   0.19701536\n",
      "  0.8812967   0.0954646   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.14746353054703976\n",
      "\n",
      "step 165\n",
      "observation: [ 0.3581558  -0.18583581 -0.70430464  0.7523248  -0.3177745   0.21559894\n",
      "  0.84548116  0.11404818  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.27146115689107425\n",
      "\n",
      "step 166\n",
      "observation: [ 0.26861686 -0.13937686 -0.7311663   0.7662625  -0.34463617  0.22953662\n",
      "  0.81861943  0.12798586  3.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2906613762222108\n",
      "\n",
      "step 167\n",
      "observation: [ 0.20146264 -0.10453264 -0.75131255  0.77671576 -0.36478245  0.23998989\n",
      "  0.7984732   0.13843913  3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18345468193829398\n",
      "\n",
      "step 168\n",
      "observation: [ 0.15109697  0.42160052 -0.7664223   0.73455566 -0.37989214  0.19782984\n",
      "  0.78336346  0.09627908  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.19980321784292643\n",
      "\n",
      "step 169\n",
      "observation: [ 0.11332273 -0.18379961 -0.77775455  0.75293565 -0.3912244   0.2162098\n",
      "  0.7720312   0.11465904  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.175204202400156\n",
      "\n",
      "step 170\n",
      "observation: [-0.41500795 -0.1378497  -0.73625374  0.76672065 -0.3497236   0.22999477\n",
      "  0.813532    0.12844402  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.12990508673438708\n",
      "\n",
      "step 171\n",
      "observation: [-0.811256   -0.10338728 -0.6551281   0.7770594  -0.26859802  0.2403335\n",
      "  0.8946576   0.13878274  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.13201783454171706\n",
      "\n",
      "step 172\n",
      "observation: [-0.60844195 -0.57754046 -0.59428394  0.8348134  -0.20775382  0.29808754\n",
      "  0.9555018   0.19653678  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11119886708005962\n",
      "\n",
      "step 173\n",
      "observation: [-0.4563315   0.06684466 -0.5486508   0.82812893 -0.16212067  0.29140308\n",
      "  1.001135    0.18985233  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.07223744316904028\n",
      "\n",
      "step 174\n",
      "observation: [-0.34224862  0.55013347 -0.51442593  0.7731156  -0.12789582  0.23638973\n",
      "  1.0353599   0.13483897  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07053897205026818\n",
      "\n",
      "step 175\n",
      "observation: [-0.25668645 -0.08739989 -0.48875728  0.7818556  -0.10222717  0.24512972\n",
      "  1.0610285   0.14357896  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.047561685610768546\n",
      "\n",
      "step 176\n",
      "observation: [-0.19251484  0.4344501  -0.4695058   0.7384106  -0.08297569  0.20168471\n",
      "  1.08028     0.10013395  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.13329246185659957\n",
      "\n",
      "step 177\n",
      "observation: [-0.14438613  0.32583755 -0.4550672   0.7058268  -0.06853707  0.16910096\n",
      "  1.0947186   0.0675502   3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.032528459669548704\n",
      "\n",
      "step 178\n",
      "observation: [ 0.3917104   0.24437818 -0.49423823  0.681389   -0.10770811  0.14466314\n",
      "  1.0555475   0.04311238  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.050961792521336216\n",
      "\n",
      "step 179\n",
      "observation: [ 0.79378283  0.18328363 -0.5736165   0.6630606  -0.18708639  0.12633477\n",
      "  0.9761692   0.02478401  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08725649475841563\n",
      "\n",
      "step 180\n",
      "observation: [ 0.5953371  -0.36253726 -0.6331502   0.69931436 -0.2466201   0.16258849\n",
      "  0.9166355   0.06103774  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.10437654854409426\n",
      "\n",
      "step 181\n",
      "observation: [ 0.44650283  0.22809704 -0.6778005   0.6765047  -0.29127038  0.1397788\n",
      "  0.87198526  0.03822804  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.13528327399291679\n",
      "\n",
      "step 182\n",
      "observation: [ 0.33487713 -0.32892722 -0.7112882   0.7093974  -0.32475808  0.17267151\n",
      "  0.8384975   0.07112076  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.12886783732827206\n",
      "\n",
      "step 183\n",
      "observation: [-0.24884216 -0.24669541 -0.686404    0.7340669  -0.2998739   0.19734105\n",
      "  0.86338174  0.0957903   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.10658343980641377\n",
      "\n",
      "step 184\n",
      "observation: [-0.18663162  0.31497845 -0.6677408   0.70256907 -0.28121072  0.16584322\n",
      "  0.8820449   0.06429246  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1083514361243513\n",
      "\n",
      "step 185\n",
      "observation: [-0.13997371 -0.26376617 -0.65374345  0.7289457  -0.26721334  0.19221982\n",
      "  0.8960423   0.09066907  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.13901925895050765\n",
      "\n",
      "step 186\n",
      "observation: [ 0.3950197  -0.19782463 -0.6932454   0.74872816 -0.3067153   0.21200229\n",
      "  0.8565403   0.11045153  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.13344762026135715\n",
      "\n",
      "step 187\n",
      "observation: [-0.20373522 -0.14836848 -0.6728719   0.763565   -0.2863418   0.22683914\n",
      "  0.87691385  0.12528838  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23010260753132425\n",
      "\n",
      "step 188\n",
      "observation: [-0.15280141 -0.11127635 -0.65759176  0.77469265 -0.27106166  0.23796678\n",
      "  0.892194    0.13641602  3.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.10593156325242373\n",
      "\n",
      "step 189\n",
      "observation: [-0.11460106  0.41654274 -0.6461317   0.73303837 -0.25960156  0.1963125\n",
      "  0.9036541   0.09476174  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.06765229937213443\n",
      "\n",
      "step 190\n",
      "observation: [-0.5859508   0.31240705 -0.5875366   0.70179766 -0.20100647  0.1650718\n",
      "  0.96224916  0.06352104  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14473014300498294\n",
      "\n",
      "step 191\n",
      "observation: [-0.4394631   0.23430529 -0.54359025  0.67836714 -0.15706016  0.14164127\n",
      "  1.0061954   0.04009051  3.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04570392016880288\n",
      "\n",
      "step 192\n",
      "observation: [ 0.17040268  0.17572896 -0.56063056  0.6607942  -0.17410043  0.12406837\n",
      "  0.9891552   0.02251762  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.03103261037677231\n",
      "\n",
      "step 193\n",
      "observation: [-0.372198    0.13179672 -0.52341074  0.64761454 -0.13688064  0.1108887\n",
      "  1.026375    0.00933794  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.03547191622588522\n",
      "\n",
      "step 194\n",
      "observation: [ 2.20851511e-01  9.88475457e-02 -5.45495868e-01  6.37729824e-01\n",
      " -1.58965781e-01  1.01003945e-01  1.00428987e+00 -5.46811731e-04\n",
      "  0.00000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05142885263153513\n",
      "\n",
      "step 195\n",
      "observation: [ 0.16563863 -0.42586434 -0.56205976  0.6803162  -0.17552964  0.14359038\n",
      "  0.987726    0.04203962  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05108398984452815\n",
      "\n",
      "step 196\n",
      "observation: [ 0.12422897  0.18060175 -0.5744827   0.66225606 -0.18795255  0.1255302\n",
      "  0.9753031   0.02397945  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15145599673735688\n",
      "\n",
      "step 197\n",
      "observation: [ 0.09317173  0.1354513  -0.58379984  0.6487109  -0.19726971  0.11198507\n",
      "  0.9659859   0.01043432  3.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.034163985382297715\n",
      "\n",
      "step 198\n",
      "observation: [-4.3012121e-01  1.0158848e-01 -5.4078770e-01  6.3855207e-01\n",
      " -1.5425760e-01  1.0182623e-01  1.0089980e+00  2.7546936e-04\n",
      "  0.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.016837901617298534\n",
      "\n",
      "step 199\n",
      "observation: [-0.3225909   0.57619137 -0.5085286   0.5809329  -0.1219985   0.04420709\n",
      "  1.0412571  -0.05734367  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.012165918024240484\n",
      "\n",
      "step 0\n",
      "observation: [ 0.          0.         -1.087085   -1.216352   -0.23936401 -1.2764785\n",
      "  0.4941422  -1.9182279   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.5615447586337103\n",
      "\n",
      "step 1\n",
      "observation: [ 0.         -0.5        -1.087085   -1.166352   -0.23936401 -1.2264786\n",
      "  0.4941422  -1.868228    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -1.3545672630030332\n",
      "\n",
      "step 2\n",
      "observation: [ 0.         -0.875      -1.087085   -1.078852   -0.23936401 -1.1389786\n",
      "  0.4941422  -1.780728    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -1.1879465674814709\n",
      "\n",
      "step 3\n",
      "observation: [-0.5        -0.65625    -1.0370849  -1.013227   -0.18936402 -1.0733535\n",
      "  0.5441422  -1.7151029   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.9720012820145102\n",
      "\n",
      "step 4\n",
      "observation: [-0.375      -0.9921875  -0.999585   -0.91400826 -0.15186402 -0.9741348\n",
      "  0.5816422  -1.6158842   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.7373366706046145\n",
      "\n",
      "step 5\n",
      "observation: [-0.28125    -1.2441406  -0.97146    -0.7895942  -0.12373902 -0.8497207\n",
      "  0.6097672  -1.4914701   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.5095514037153313\n",
      "\n",
      "step 6\n",
      "observation: [-0.2109375  -1.4331055  -0.95036626 -0.6462836  -0.10264527 -0.70641017\n",
      "  0.630861   -1.3481596   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.30885972846965226\n",
      "\n",
      "step 7\n",
      "observation: [-0.15820312 -1.5748291  -0.93454593 -0.48880073 -0.08682495 -0.54892725\n",
      "  0.64668125 -1.1906767   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.18622463322786723\n",
      "\n",
      "step 8\n",
      "observation: [-0.61865234 -1.1811218  -0.87268066 -0.37068856 -0.02495972 -0.4308151\n",
      "  0.7085465  -1.0725645   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08585857187682638\n",
      "\n",
      "step 9\n",
      "observation: [-0.46398926 -1.3858414  -0.8262818  -0.23210442  0.0214392  -0.29223096\n",
      "  0.75494546 -0.9339804   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.022287671023260944\n",
      "\n",
      "step 10\n",
      "observation: [-0.34799194 -1.539381   -0.79148257 -0.07816631  0.0562384  -0.13829285\n",
      "  0.7897446  -0.7800423   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.012085072599097794\n",
      "\n",
      "step 11\n",
      "observation: [-0.26099396 -0.6545358  -0.7653832  -0.01271274  0.0823378  -0.07283928\n",
      "  0.815844   -0.7145887   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.011075235133594987\n",
      "\n",
      "step 12\n",
      "observation: [-0.19574547 -0.9909018  -0.7458086   0.08637744  0.10191234  0.0262509\n",
      "  0.8354186  -0.61549854  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.037867342786417194\n",
      "\n",
      "step 13\n",
      "observation: [-0.6468091  -0.74317634 -0.6811277   0.16069508  0.16659325  0.10056854\n",
      "  0.90009946 -0.5411809   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.05169111507505837\n",
      "\n",
      "step 14\n",
      "observation: [ 0.01489317 -0.5573823  -0.682617    0.21643332  0.16510393  0.15630677\n",
      "  0.8986102  -0.48544264  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08503815942631215\n",
      "\n",
      "step 15\n",
      "observation: [-0.48883012 -0.4180367  -0.63373405  0.25823697  0.21398695  0.19811045\n",
      "  0.9474932  -0.44363898  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.09291346047561592\n",
      "\n",
      "step 16\n",
      "observation: [ 0.1333774  -0.31352752 -0.6470718   0.28958973  0.2006492   0.22946319\n",
      "  0.9341554  -0.41228622  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.12190819481133812\n",
      "\n",
      "step 17\n",
      "observation: [-0.39996696 -0.23514564 -0.6070751   0.3131043   0.2406459   0.25297776\n",
      "  0.97415215 -0.38877165  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12191528346503988\n",
      "\n",
      "step 18\n",
      "observation: [ 0.2000248  -0.17635924 -0.6270776   0.3307402   0.22064342  0.27061367\n",
      "  0.95414966 -0.37113574  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2228539514948466\n",
      "\n",
      "step 19\n",
      "observation: [ 0.15001859 -0.13226943 -0.6420794   0.34396717  0.20564155  0.28384063\n",
      "  0.9391478  -0.3579088   4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.10714394629328297\n",
      "\n",
      "step 20\n",
      "observation: [ 0.61251396 -0.09920207 -0.7033308   0.35388738  0.14439017  0.29376084\n",
      "  0.87789637 -0.34798858  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11275991806540551\n",
      "\n",
      "step 21\n",
      "observation: [-0.04061454 -0.07440155 -0.69926935  0.36132753  0.14845161  0.301201\n",
      "  0.8819578  -0.34054843  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.08888809133716649\n",
      "\n",
      "step 22\n",
      "observation: [-0.03046091  0.44419885 -0.69622326  0.31690764  0.1514977   0.2567811\n",
      "  0.8850039  -0.3849683   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09843274663600798\n",
      "\n",
      "step 23\n",
      "observation: [-0.02284568 -0.16685088 -0.6939387   0.33359274  0.15378228  0.2734662\n",
      "  0.8872885  -0.36828324  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1370614787602349\n",
      "\n",
      "step 24\n",
      "observation: [-0.01713426 -0.62513816 -0.6922253   0.39610654  0.1554957   0.33598\n",
      "  0.8890019  -0.3057694   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1579880238662879\n",
      "\n",
      "step 25\n",
      "observation: [ 0.4871493  -0.46885362 -0.7409402   0.4429919   0.10678077  0.38286537\n",
      "  0.84028697 -0.25888404  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.17515841358671835\n",
      "\n",
      "step 26\n",
      "observation: [ 0.865362   -0.35164022 -0.8274764   0.47815594  0.02024457  0.4180294\n",
      "  0.7537508  -0.22372003  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.19752203978107885\n",
      "\n",
      "step 27\n",
      "observation: [ 0.14902149 -0.26373017 -0.84237856  0.50452894  0.00534243  0.4444024\n",
      "  0.7388486  -0.19734702  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.21741570402326527\n",
      "\n",
      "step 28\n",
      "observation: [-0.3882339  -0.19779763 -0.80355513  0.5243087   0.04416582  0.46418217\n",
      "  0.77767205 -0.17756726  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18942602730568578\n",
      "\n",
      "step 29\n",
      "observation: [-0.29117543  0.3516518  -0.7744376   0.48914352  0.07328336  0.42901698\n",
      "  0.8067896  -0.21273243  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1641574210984456\n",
      "\n",
      "step 30\n",
      "observation: [ 0.28161845  0.26373884 -0.80259943  0.46276966  0.04512151  0.4026431\n",
      "  0.77862775 -0.23910631  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.18794611663014837\n",
      "\n",
      "step 31\n",
      "observation: [ 0.21121383 -0.30219588 -0.8237208   0.49298924  0.02400013  0.4328627\n",
      "  0.7575064  -0.20888673  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.20925585788798165\n",
      "\n",
      "step 32\n",
      "observation: [ 0.6584104  -0.2266469  -0.8895619   0.5156539  -0.04184091  0.4555274\n",
      "  0.6916653  -0.18622203  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3316021224330884\n",
      "\n",
      "step 33\n",
      "observation: [ 0.4938078  -0.16998518 -0.9389427   0.53265244 -0.09122168  0.4725259\n",
      "  0.6422845  -0.16922352  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.35194154755539253\n",
      "\n",
      "step 34\n",
      "observation: [ 0.37035584 -0.12748888 -0.97597826  0.54540133 -0.12825727  0.4852748\n",
      "  0.6052489  -0.15647464  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.36920971267344993\n",
      "\n",
      "step 35\n",
      "observation: [ 0.27776688 -0.09561666 -1.003755    0.554963   -0.15603396  0.49483645\n",
      "  0.57747227 -0.14691296  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2681068371948288\n",
      "\n",
      "step 36\n",
      "observation: [-0.29167485 -0.07171249 -0.97458744  0.56213427 -0.12686647  0.5020077\n",
      "  0.60663974 -0.13974172  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2202251631718601\n",
      "\n",
      "step 37\n",
      "observation: [-0.21875614  0.44621563 -0.9527118   0.5175127  -0.10499086  0.45738614\n",
      "  0.62851536 -0.18436328  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2875552921960683\n",
      "\n",
      "step 38\n",
      "observation: [-0.1640671   0.33466172 -0.9363051   0.48404652 -0.08858415  0.42391998\n",
      "  0.6449221  -0.21782945  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1597482561096923\n",
      "\n",
      "step 39\n",
      "observation: [-0.62305033  0.2509963  -0.8740001   0.45894688 -0.02627912  0.39882034\n",
      "  0.7072271  -0.24292909  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.10931529699222149\n",
      "\n",
      "step 40\n",
      "observation: [-0.46728775  0.6882472  -0.82727134  0.39012215  0.02044966  0.32999563\n",
      "  0.7539559  -0.3117538   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.055235922041062535\n",
      "\n",
      "step 41\n",
      "observation: [-0.3504658   1.0161854  -0.7922247   0.28850362  0.05549624  0.22837707\n",
      "  0.7890025  -0.41337234  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.047558108363856405\n",
      "\n",
      "step 42\n",
      "observation: [-0.26284936  0.26213905 -0.7659398   0.2622897   0.08178117  0.20216317\n",
      "  0.8152874  -0.43958625  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14360846065717253\n",
      "\n",
      "step 43\n",
      "observation: [-0.19713701  0.1966043  -0.7462261   0.24262927  0.10149488  0.18250275\n",
      "  0.8350011  -0.45924667  4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.03253561048747032\n",
      "\n",
      "step 44\n",
      "observation: [ 0.35214725  0.14745322 -0.7814408   0.22788396  0.06628015  0.16775742\n",
      "  0.7997864  -0.473992    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.03263085422622419\n",
      "\n",
      "step 45\n",
      "observation: [-0.23588957  0.11058991 -0.75785184  0.21682496  0.08986911  0.15669844\n",
      "  0.82337534 -0.485051    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.025337050829897307\n",
      "\n",
      "step 46\n",
      "observation: [ 0.32308283  0.08294243 -0.7901601   0.20853072  0.05756083  0.14840418\n",
      "  0.79106706 -0.49334523  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03804536165544481\n",
      "\n",
      "step 47\n",
      "observation: [ 0.24231212 -0.43779317 -0.8143914   0.25231004  0.03332962  0.1921835\n",
      "  0.76683587 -0.44956592  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07586460530619345\n",
      "\n",
      "step 48\n",
      "observation: [ 0.18173409 -0.8283449  -0.8325648   0.33514452  0.01515621  0.27501798\n",
      "  0.7486624  -0.36673144  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2136683085828892\n",
      "\n",
      "step 49\n",
      "observation: [ 1.3630056e-01 -6.2125868e-01 -8.4619480e-01  3.9727038e-01\n",
      "  1.5261503e-03  3.3714387e-01  7.3503238e-01 -3.0460557e-01\n",
      "  4.0000000e+00]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1507003149263983\n",
      "\n",
      "step 50\n",
      "observation: [ 0.6022254  -0.465944   -0.90641737  0.4438648  -0.05869639  0.38373825\n",
      "  0.6748098  -0.25801116  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.17819759267905483\n",
      "\n",
      "step 51\n",
      "observation: [-0.04833093 -0.349458   -0.90158427  0.4788106  -0.0538633   0.41868407\n",
      "  0.6796429  -0.22306536  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.24744338654439563\n",
      "\n",
      "step 52\n",
      "observation: [-0.0362482  -0.7620935  -0.8979595   0.5550199  -0.05023848  0.4948934\n",
      "  0.6832678  -0.14685601  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.40701780008727484\n",
      "\n",
      "step 53\n",
      "observation: [-0.02718615 -0.5715701  -0.89524084  0.61217695 -0.04751986  0.5520504\n",
      "  0.68598634 -0.08969901  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.4559961485753632\n",
      "\n",
      "step 54\n",
      "observation: [-0.02038961 -0.4286776  -0.8932019   0.65504473 -0.0454809   0.5949182\n",
      "  0.6880253  -0.04683125  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.39325210705937225\n",
      "\n",
      "step 55\n",
      "observation: [-0.5152922  -0.3215082  -0.84167266  0.68719554  0.00604832  0.627069\n",
      "  0.7395545  -0.01468043  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.3634175931644225\n",
      "\n",
      "step 56\n",
      "observation: [-0.38646916  0.25886884 -0.8030257   0.66130865  0.04469523  0.6011821\n",
      "  0.77820146 -0.04056731  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2882048890081665\n",
      "\n",
      "step 57\n",
      "observation: [-0.28985187  0.69415164 -0.7740406   0.5918935   0.07368042  0.53176695\n",
      "  0.80718666 -0.10998248  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.23218035028355624\n",
      "\n",
      "step 58\n",
      "observation: [ 0.2826111   0.52061373 -0.80230165  0.5398321   0.04541931  0.47970557\n",
      "  0.77892554 -0.16204385  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.19484526185997886\n",
      "\n",
      "step 59\n",
      "observation: [ 0.71195835  0.3904603  -0.8734975   0.50078607 -0.02577652  0.44065955\n",
      "  0.7077297  -0.20108987  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.17008049460916283\n",
      "\n",
      "step 60\n",
      "observation: [ 0.03396874  0.29284522 -0.87689435  0.47150156 -0.0291734   0.41137502\n",
      "  0.7043328  -0.2303744   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1583197469995388\n",
      "\n",
      "step 61\n",
      "observation: [ 0.5254766   0.21963392 -0.92944205  0.44953817 -0.08172105  0.38941163\n",
      "  0.6517852  -0.25233778  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.19355038109006203\n",
      "\n",
      "step 62\n",
      "observation: [ 0.39410743 -0.33527455 -0.96885276  0.48306563 -0.12113179  0.4229391\n",
      "  0.6123744  -0.21881033  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.21091832380851525\n",
      "\n",
      "step 63\n",
      "observation: [-0.20441943 -0.25145593 -0.9484108   0.5082112  -0.10068985  0.44808468\n",
      "  0.6328164  -0.19366474  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18112824691946777\n",
      "\n",
      "step 64\n",
      "observation: [-0.15331458  0.31140807 -0.93307936  0.47707042 -0.08535839  0.41694388\n",
      "  0.6481478  -0.22480555  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2603669936996543\n",
      "\n",
      "step 65\n",
      "observation: [-0.11498594  0.23355605 -0.9215808   0.45371482 -0.0738598   0.39358827\n",
      "  0.65964645 -0.24816115  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.24568553753621614\n",
      "\n",
      "step 66\n",
      "observation: [-0.08623945  0.17516704 -0.91295683  0.43619812 -0.06523585  0.37607157\n",
      "  0.66827035 -0.26567787  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23517478002381223\n",
      "\n",
      "step 67\n",
      "observation: [-0.06467959  0.13137527 -0.9064889   0.42306057 -0.0587679   0.36293402\n",
      "  0.67473835 -0.2788154   4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.13546484274542217\n",
      "\n",
      "step 68\n",
      "observation: [ 0.4514903   0.09853145 -0.9516379   0.41320744 -0.10391693  0.3530809\n",
      "  0.6295893  -0.28866854  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.17555436221458673\n",
      "\n",
      "step 69\n",
      "observation: [ 0.33861774 -0.42610142 -0.9854997   0.45581758 -0.1377787   0.39569104\n",
      "  0.5957275  -0.24605839  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.19569194383162083\n",
      "\n",
      "step 70\n",
      "observation: [-0.2460367  -0.31957605 -0.960896    0.48777518 -0.11317503  0.42764863\n",
      "  0.62033117 -0.21410078  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20595785309304424\n",
      "\n",
      "step 71\n",
      "observation: [-0.6845275  -0.23968205 -0.89244324  0.51174337 -0.04472228  0.45161685\n",
      "  0.68878394 -0.19013257  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.22372309735492735\n",
      "\n",
      "step 72\n",
      "observation: [-1.0133957  -0.17976153 -0.79110366  0.52971953  0.05661729  0.469593\n",
      "  0.7901235  -0.17215642  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.35095014538850466\n",
      "\n",
      "step 73\n",
      "observation: [-0.7600467  -0.13482115 -0.71509904  0.5432016   0.13262196  0.4830751\n",
      "  0.8661282  -0.15867431  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2323722639988688\n",
      "\n",
      "step 74\n",
      "observation: [-0.57003504  0.39888415 -0.6580955   0.50331324  0.18962547  0.4431867\n",
      "  0.9231317  -0.19856273  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.32479198970419965\n",
      "\n",
      "step 75\n",
      "observation: [-0.4275263   0.2991631  -0.61534286  0.47339693  0.2323781   0.41327038\n",
      "  0.9658843  -0.22847904  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3226804234036992\n",
      "\n",
      "step 76\n",
      "observation: [-0.3206447   0.22437233 -0.5832784   0.45095968  0.26444256  0.39083314\n",
      "  0.99794877 -0.25091627  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.25445601282861807\n",
      "\n",
      "step 77\n",
      "observation: [-0.7404835   0.16827925 -0.5092301   0.43413177  0.33849093  0.37400523\n",
      "  1.0719972  -0.26774418  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.24895330466272217\n",
      "\n",
      "step 78\n",
      "observation: [-0.05536265  0.12620944 -0.50369376  0.42151082  0.3440272   0.36138427\n",
      "  1.0775334  -0.28036514  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2823935061443018\n",
      "\n",
      "step 79\n",
      "observation: [-0.54152197  0.09465708 -0.4495416   0.41204512  0.39817938  0.35191858\n",
      "  1.1316856  -0.28983083  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.2700607580442842\n",
      "\n",
      "step 80\n",
      "observation: [ 0.09385851  0.07099281 -0.45892742  0.40494585  0.38879353  0.3448193\n",
      "  1.1222998  -0.29693013  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2974424538288213\n",
      "\n",
      "step 81\n",
      "observation: [ 0.07039388 -0.4467554  -0.46596682  0.44962138  0.38175416  0.38949484\n",
      "  1.1152604  -0.25225458  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.28086323876406205\n",
      "\n",
      "step 82\n",
      "observation: [ 0.05279541  0.16493346 -0.47124636  0.43312803  0.3764746   0.3730015\n",
      "  1.1099808  -0.26874793  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.23407095664236516\n",
      "\n",
      "step 83\n",
      "observation: [ 0.53959656  0.12370009 -0.525206    0.420758    0.32251495  0.36063147\n",
      "  1.0560212  -0.28111795  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.30299911337253066\n",
      "\n",
      "step 84\n",
      "observation: [ 0.40469742  0.09277507 -0.56567574  0.41148052  0.28204522  0.35135397\n",
      "  1.0155514  -0.29039544  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2819578131009637\n",
      "\n",
      "step 85\n",
      "observation: [ 0.30352306  0.0695813  -0.5960281   0.4045224   0.2516929   0.34439585\n",
      "  0.98519915 -0.29735357  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.1360318157508984\n",
      "\n",
      "step 86\n",
      "observation: [ 0.2276423   0.55218595 -0.6187923   0.34930378  0.22892867  0.28917724\n",
      "  0.9624349  -0.35257217  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.08399307400597326\n",
      "\n",
      "step 87\n",
      "observation: [ 0.17073172  0.9141395  -0.63586545  0.25788984  0.2118555   0.1977633\n",
      "  0.94536173 -0.44398612  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07173480550410495\n",
      "\n",
      "step 88\n",
      "observation: [ 0.12804879  0.18560462 -0.6486704   0.23932938  0.19905062  0.17920284\n",
      "  0.93255687 -0.4625466   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.046763756258973374\n",
      "\n",
      "step 89\n",
      "observation: [ 0.5960366   0.13920346 -0.708274    0.22540903  0.13944696  0.16528249\n",
      "  0.8729532  -0.47646692  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05093680606964081\n",
      "\n",
      "step 90\n",
      "observation: [ 0.44702744 -0.3955974  -0.7529768   0.26496878  0.09474421  0.20484222\n",
      "  0.82825047 -0.4369072   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03779223205358745\n",
      "\n",
      "step 91\n",
      "observation: [ 0.33527058  0.20330195 -0.7865038   0.24463858  0.06121716  0.18451203\n",
      "  0.7947234  -0.4572374   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12995161339194333\n",
      "\n",
      "step 92\n",
      "observation: [ 0.25145295  0.15247646 -0.8116491   0.22939093  0.03607186  0.16926439\n",
      "  0.7695781  -0.47248504  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.01192330287918415\n",
      "\n",
      "step 93\n",
      "observation: [ 0.1885897   0.61435735 -0.83050805  0.16795519  0.01721289  0.10782865\n",
      "  0.75071913 -0.53392076  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.012497894019366896\n",
      "\n",
      "step 94\n",
      "observation: [ 0.14144228 -0.03923199 -0.8446523   0.1718784   0.00306866  0.11175185\n",
      "  0.7365749  -0.5299976   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.11321161638050276\n",
      "\n",
      "step 95\n",
      "observation: [ 0.10608171 -0.02942399 -0.8552605   0.1748208  -0.00753951  0.11469425\n",
      "  0.7259667  -0.52705514  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.014856407856239023\n",
      "\n",
      "step 96\n",
      "observation: [-0.4204387  -0.022068   -0.8132166   0.1770276   0.03450437  0.11690105\n",
      "  0.7680106  -0.52484834  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03277209814930379\n",
      "\n",
      "step 97\n",
      "observation: [-0.31532905 -0.516551   -0.7816837   0.2286827   0.06603727  0.16855615\n",
      "  0.7995435  -0.47319326  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07424574132147298\n",
      "\n",
      "step 98\n",
      "observation: [-0.23649678 -0.88741326 -0.75803405  0.31742403  0.08968695  0.2572975\n",
      "  0.8231932  -0.38445196  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1296634523453344\n",
      "\n",
      "step 99\n",
      "observation: [-0.6773726  -0.66555995 -0.69029677  0.38398     0.15742421  0.32385346\n",
      "  0.8909304  -0.31789595  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.14818586166305062\n",
      "\n",
      "step 100\n",
      "observation: [-5.0802946e-01  8.3004835e-04 -6.3949382e-01  3.8389701e-01\n",
      "  2.0822714e-01  3.2377046e-01  9.4173336e-01 -3.1797895e-01\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.26546516026602585\n",
      "\n",
      "step 101\n",
      "observation: [-3.8102207e-01  6.2253623e-04 -6.0139161e-01  3.8383475e-01\n",
      "  2.4632935e-01  3.2370821e-01  9.7983557e-01 -3.1804121e-01\n",
      "  4.0000000e+00]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1553394969472945\n",
      "\n",
      "step 102\n",
      "observation: [ 2.1423344e-01  4.6690219e-04 -6.2281495e-01  3.8378808e-01\n",
      "  2.2490601e-01  3.2366154e-01  9.5841223e-01 -3.1808791e-01\n",
      "  0.0000000e+00]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12996378639182962\n",
      "\n",
      "step 103\n",
      "observation: [ 6.6067511e-01  3.5017665e-04 -6.8888247e-01  3.8375306e-01\n",
      "  1.5883851e-01  3.2362652e-01  8.9234471e-01 -3.1812292e-01\n",
      "  0.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.08680093251086345\n",
      "\n",
      "step 104\n",
      "observation: [ 0.49550632  0.5002626  -0.7384331   0.3337268   0.10928787  0.27360025\n",
      "  0.8427941  -0.3681492   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08704408000342694\n",
      "\n",
      "step 105\n",
      "observation: [ 0.37162974 -0.12480303 -0.7755961   0.34620708  0.0721249   0.28608054\n",
      "  0.8056311  -0.35566887  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.12128762635307154\n",
      "\n",
      "step 106\n",
      "observation: [ 0.2787223  -0.59360224 -0.8034683   0.40556732  0.04425267  0.34544078\n",
      "  0.7777589  -0.29630864  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1527798395848868\n",
      "\n",
      "step 107\n",
      "observation: [ 0.7090417  -0.4452017  -0.8743725   0.4500875  -0.0266515   0.38996094\n",
      "  0.7068547  -0.25178847  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.18011593587926064\n",
      "\n",
      "step 108\n",
      "observation: [ 0.03178129 -0.3339013  -0.8775506   0.48347762 -0.02982963  0.42335108\n",
      "  0.7036766  -0.21839835  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3020945720835565\n",
      "\n",
      "step 109\n",
      "observation: [ 0.02383597 -0.25042596 -0.8799342   0.5085202  -0.03221323  0.44839367\n",
      "  0.701293   -0.19335575  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.21850902388283075\n",
      "\n",
      "step 110\n",
      "observation: [-0.48212302 -0.18781947 -0.8317219   0.52730215  0.01599907  0.4671756\n",
      "  0.7495053  -0.17457381  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18870746414987336\n",
      "\n",
      "step 111\n",
      "observation: [-0.36159226  0.3591354  -0.7955627   0.49138862  0.0521583   0.43126208\n",
      "  0.7856645  -0.21048735  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.26976521387915464\n",
      "\n",
      "step 112\n",
      "observation: [-0.2711942   0.26935154 -0.7684432   0.46445346  0.07927772  0.40432692\n",
      "  0.81278396 -0.2374225   4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.12156348819913217\n",
      "\n",
      "step 113\n",
      "observation: [-0.20339565  0.7020137  -0.7481037   0.3942521   0.09961729  0.33412555\n",
      "  0.8331235  -0.30762386  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.06677602394926802\n",
      "\n",
      "step 114\n",
      "observation: [-0.15254673  1.0265102  -0.732849    0.29160106  0.11487196  0.23147453\n",
      "  0.8483782  -0.4102749   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.057769595196865295\n",
      "\n",
      "step 115\n",
      "observation: [-0.11441005  0.26988268 -0.721408    0.2646128   0.12631297  0.20448627\n",
      "  0.8598192  -0.43726316  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03621805087016801\n",
      "\n",
      "step 116\n",
      "observation: [-0.08580754  0.702412   -0.71282727  0.1943716   0.13489372  0.13424505\n",
      "  0.8683999  -0.50750434  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.014993749812405154\n",
      "\n",
      "step 117\n",
      "observation: [ 0.43564436  0.52680904 -0.7563917   0.1416907   0.09132928  0.08156416\n",
      "  0.8248355  -0.56018525  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.011914364791746668\n",
      "\n",
      "step 118\n",
      "observation: [ 0.32673326 -0.10489324 -0.789065    0.15218002  0.05865596  0.09205348\n",
      "  0.7921622  -0.54969597  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.017065485869261918\n",
      "\n",
      "step 119\n",
      "observation: [-0.25495005 -0.07866994 -0.76357     0.16004701  0.08415096  0.09992047\n",
      "  0.8176572  -0.54182893  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.034945247487101594\n",
      "\n",
      "step 120\n",
      "observation: [-0.19121253 -0.55900246 -0.7444488   0.21594726  0.10327221  0.15582071\n",
      "  0.83677846 -0.4859287   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04367498122531686\n",
      "\n",
      "step 121\n",
      "observation: [ 0.3565906  -0.41925183 -0.7801078   0.25787243  0.06761316  0.1977459\n",
      "  0.8011194  -0.44400352  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.06078511048085705\n",
      "\n",
      "step 122\n",
      "observation: [-0.23255706 -0.31443888 -0.7568521   0.28931633  0.09086887  0.22918978\n",
      "  0.8243751  -0.41255963  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08895630077758021\n",
      "\n",
      "step 123\n",
      "observation: [-0.6744178  -0.23582916 -0.6894103   0.31289923  0.15831064  0.25277272\n",
      "  0.89181685 -0.38897672  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14017360985418675\n",
      "\n",
      "step 124\n",
      "observation: [-1.0058134  -0.17687187 -0.588829    0.33058643  0.25889197  0.2704599\n",
      "  0.9923982  -0.37128952  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.292275233492983\n",
      "\n",
      "step 125\n",
      "observation: [-0.75436    -0.1326539  -0.513393    0.34385183  0.33432797  0.2837253\n",
      "  1.0678343  -0.35802415  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.21218387968042163\n",
      "\n",
      "step 126\n",
      "observation: [-0.56577003  0.40050957 -0.456816    0.30380085  0.390905    0.24367432\n",
      "  1.1244112  -0.3980751   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.33342197430827497\n",
      "\n",
      "step 127\n",
      "observation: [-0.4243275   0.30038217 -0.41438323  0.27376264  0.43333772  0.2136361\n",
      "  1.166844   -0.4281133   4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3528980133970069\n",
      "\n",
      "step 128\n",
      "observation: [-0.31824562  0.22528663 -0.38255867  0.251234    0.4651623   0.19110744\n",
      "  1.1986685  -0.450642    4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3695004935298418\n",
      "\n",
      "step 129\n",
      "observation: [-0.23868422  0.16896498 -0.35869026  0.2343375   0.48903072  0.17421095\n",
      "  1.2225369  -0.46753848  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.38307479471362893\n",
      "\n",
      "step 130\n",
      "observation: [-0.17901316  0.12672374 -0.34078893  0.22166511  0.506932    0.16153857\n",
      "  1.2404382  -0.48021084  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3938868937113098\n",
      "\n",
      "step 131\n",
      "observation: [-0.13425988  0.0950428  -0.32736295  0.21216084  0.520358    0.1520343\n",
      "  1.2538643  -0.48971513  4.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.4023511153338556\n",
      "\n",
      "step 132\n",
      "observation: [-0.1006949   0.0712821  -0.31729347  0.20503262  0.5304275   0.14490609\n",
      "  1.2639338  -0.49684334  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.29744305935701576\n",
      "\n",
      "step 133\n",
      "observation: [-0.07552118  0.55346155 -0.30974135  0.14968647  0.5379796   0.08955993\n",
      "  1.2714858  -0.55218947  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.39785732138985586\n",
      "\n",
      "step 134\n",
      "observation: [-0.05664089  0.4150962  -0.30407727  0.10817685  0.5436437   0.04805031\n",
      "  1.2771499  -0.5936991   4.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.3046634358483976\n",
      "\n",
      "step 135\n",
      "observation: [-0.04248066 -0.18867786 -0.29982919  0.12704463  0.5478918   0.0669181\n",
      "  1.281398   -0.5748313   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.257651165296829\n",
      "\n",
      "step 136\n",
      "observation: [ 0.4681395  -0.1415084  -0.34664315  0.14119548  0.50107783  0.08106893\n",
      "  1.2345841  -0.5606805   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.23719939604668558\n",
      "\n",
      "step 137\n",
      "observation: [ 0.35110462 -0.6061313  -0.3817536   0.2018086   0.46596736  0.14168206\n",
      "  1.1994736  -0.50006735  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.32830060774313496\n",
      "\n",
      "step 138\n",
      "observation: [ 0.26332846 -0.4545985  -0.40808645  0.24726845  0.43963453  0.18714191\n",
      "  1.1731408  -0.45460752  4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.18576054959260008\n",
      "\n",
      "step 139\n",
      "observation: [ 0.69749635 -0.34094885 -0.47783607  0.28136334  0.36988488  0.2212368\n",
      "  1.103391   -0.42051262  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.26176656408166754\n",
      "\n",
      "step 140\n",
      "observation: [ 0.52312225 -0.25571164 -0.5301483   0.3069345   0.31757265  0.24680796\n",
      "  1.0510789  -0.39494145  4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12288719610420835\n",
      "\n",
      "step 141\n",
      "observation: [ 0.8923417  -0.19178373 -0.6193825   0.32611287  0.2283385   0.26598632\n",
      "  0.96184474 -0.3757631   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1351985245408939\n",
      "\n",
      "step 142\n",
      "observation: [ 0.66925627 -0.6438378  -0.6863081   0.39049664  0.16141286  0.33037013\n",
      "  0.8949191  -0.3113793   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.16937329290627828\n",
      "\n",
      "step 143\n",
      "observation: [ 0.00194221 -0.48287836 -0.68650234  0.43878448  0.16121864  0.37865794\n",
      "  0.89472485 -0.26309147  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.15907741211846985\n",
      "\n",
      "step 144\n",
      "observation: [ 0.00145665  0.13784124 -0.686648    0.42500037  0.16107298  0.36487383\n",
      "  0.8945792  -0.2768756   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11865133387593842\n",
      "\n",
      "step 145\n",
      "observation: [ 0.00109249  0.6033809  -0.68675727  0.36466226  0.16096373  0.30453575\n",
      "  0.89447    -0.3372137   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.12153840676714778\n",
      "\n",
      "step 146\n",
      "observation: [ 8.1936788e-04 -4.7464304e-02 -6.8683916e-01  3.6940870e-01\n",
      "  1.6088179e-01  3.0928215e-01  8.9438802e-01 -3.3246726e-01\n",
      "  0.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09494909050867988\n",
      "\n",
      "step 147\n",
      "observation: [ 6.1452592e-04  4.6440178e-01 -6.8690062e-01  3.2296851e-01\n",
      "  1.6082034e-01  2.6284200e-01  8.9432657e-01 -3.7890744e-01\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17783776397843615\n",
      "\n",
      "step 148\n",
      "observation: [ 4.6089443e-04  3.4830132e-01 -6.8694675e-01  2.8813839e-01\n",
      "  1.6077425e-01  2.2801185e-01  8.9428049e-01 -4.1373757e-01\n",
      "  4.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1665965157040324\n",
      "\n",
      "step 149\n",
      "observation: [ 3.4567082e-04  2.6122600e-01 -6.8698126e-01  2.6201579e-01\n",
      "  1.6073968e-01  2.0188925e-01  8.9424592e-01 -4.3986017e-01\n",
      "  4.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04333148793739441\n",
      "\n",
      "step 150\n",
      "observation: [ 2.5925311e-04  6.9591951e-01 -6.8700719e-01  1.9242385e-01\n",
      "  1.6071376e-01  1.3229731e-01  8.9421999e-01 -5.0945210e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.04274954135684195\n",
      "\n",
      "step 151\n",
      "observation: [ 1.9443984e-04  2.1939624e-02 -6.8702668e-01  1.9022988e-01\n",
      "  1.6069432e-01  1.3010333e-01  8.9420056e-01 -5.1164609e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.057665186476398476\n",
      "\n",
      "step 152\n",
      "observation: [ 1.4582988e-04 -4.8354527e-01 -6.8704122e-01  2.3858441e-01\n",
      "  1.6067973e-01  1.7845787e-01  8.9418596e-01 -4.6329156e-01\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17192075612337032\n",
      "\n",
      "step 153\n",
      "observation: [ 1.09372406e-04 -3.62658948e-01 -6.87052190e-01  2.74850309e-01\n",
      "  1.60668790e-01  2.14723766e-01  8.94174993e-01 -4.27025646e-01\n",
      "  4.00000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.06264633478875054\n",
      "\n",
      "step 154\n",
      "observation: [ 8.2029306e-05  2.2800578e-01 -6.8706036e-01  2.5204971e-01\n",
      "  1.6066059e-01  1.9192319e-01  8.9416683e-01 -4.4982624e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07635511883344029\n",
      "\n",
      "step 155\n",
      "observation: [ 6.1521976e-05 -3.2899567e-01 -6.8706656e-01  2.8494930e-01\n",
      "  1.6065444e-01  2.2482276e-01  8.9416069e-01 -4.1692668e-01\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1880573326129013\n",
      "\n",
      "step 156\n",
      "observation: [ 4.6141486e-05 -2.4674675e-01 -6.8707114e-01  3.0962396e-01\n",
      "  1.6064982e-01  2.4949743e-01  8.9415604e-01 -3.9225200e-01\n",
      "  4.0000000e+00]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11619773118943591\n",
      "\n",
      "step 157\n",
      "observation: [-0.4999654  -0.18506007 -0.6370746   0.32812998  0.21064636  0.26800343\n",
      "  0.9441526  -0.37374598  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.16834769749552225\n",
      "\n",
      "step 158\n",
      "observation: [-0.8749741  -0.13879505 -0.5495772   0.34200948  0.29814377  0.28188294\n",
      "  1.03165    -0.35986647  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.24949050187219435\n",
      "\n",
      "step 159\n",
      "observation: [-0.6562305  -0.6040963  -0.48395416  0.40241912  0.36376682  0.34229258\n",
      "  1.097273   -0.29945686  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.36458788342863424\n",
      "\n",
      "step 160\n",
      "observation: [-0.9921729  -0.45307222 -0.38473687  0.44772634  0.4629841   0.3875998\n",
      "  1.1964903  -0.25414965  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.41528578140926875\n",
      "\n",
      "step 161\n",
      "observation: [-0.24412967 -0.33980417 -0.3603239   0.48170674  0.48739707  0.4215802\n",
      "  1.2209033  -0.22016922  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.407536267940713\n",
      "\n",
      "step 162\n",
      "observation: [ 0.31690276 -0.25485313 -0.39201418  0.5071921   0.4557068   0.44706553\n",
      "  1.189213   -0.19468391  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.3632008022200303\n",
      "\n",
      "step 163\n",
      "observation: [ 0.73767704 -0.19113985 -0.46578187  0.52630603  0.38193908  0.4661795\n",
      "  1.1154454  -0.17556992  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.30740958322960293\n",
      "\n",
      "step 164\n",
      "observation: [ 1.0532578  -0.14335488 -0.5711076   0.54064155  0.27661332  0.480515\n",
      "  1.0101196  -0.16123444  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.26313424219603504\n",
      "\n",
      "step 165\n",
      "observation: [ 1.2899433  -0.10751616 -0.700102    0.55139315  0.14761898  0.4912666\n",
      "  0.8811252  -0.15048282  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.24933152852929374\n",
      "\n",
      "step 166\n",
      "observation: [ 1.4674575e+00 -8.0637120e-02 -8.4684771e-01  5.5945688e-01\n",
      "  8.7323337e-04  4.9933031e-01  7.3437947e-01 -1.4241910e-01\n",
      "  0.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3673286255182573\n",
      "\n",
      "step 167\n",
      "observation: [ 1.1005931  -0.06047784 -0.95690703  0.56550467 -0.10918608  0.5053781\n",
      "  0.62432015 -0.13637131  4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.28009977867963737\n",
      "\n",
      "step 168\n",
      "observation: [ 0.32544485 -0.04535838 -0.9894515   0.57004046 -0.14173056  0.5099139\n",
      "  0.59177566 -0.13183548  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2769813797054835\n",
      "\n",
      "step 169\n",
      "observation: [-0.25591636 -0.03401878 -0.9638599   0.57344234 -0.11613893  0.5133158\n",
      "  0.61736727 -0.1284336   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.22643064510913635\n",
      "\n",
      "step 170\n",
      "observation: [-0.19193727  0.4744859  -0.94466615  0.52599376 -0.0969452   0.46586722\n",
      "  0.63656104 -0.17588219  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.15142796126053126\n",
      "\n",
      "step 171\n",
      "observation: [-0.14395295  0.8558644  -0.93027085  0.44040734 -0.08254991  0.3802808\n",
      "  0.65095633 -0.26146865  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1391711397371418\n",
      "\n",
      "step 172\n",
      "observation: [-0.10796472  0.14189832 -0.9194744   0.4262175  -0.07175343  0.36609095\n",
      "  0.6617528  -0.27565846  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.13926139525329054\n",
      "\n",
      "step 173\n",
      "observation: [ 0.41902646  0.10642374 -0.961377    0.41557512 -0.11365608  0.35544857\n",
      "  0.61985016 -0.28630084  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.17902894959884638\n",
      "\n",
      "step 174\n",
      "observation: [ 0.31426984 -0.4201822  -0.99280405  0.45759332 -0.14508307  0.3974668\n",
      "  0.58842313 -0.24428263  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.19810284177574405\n",
      "\n",
      "step 175\n",
      "observation: [-0.2642976  -0.31513664 -0.9663743   0.489107   -0.1186533   0.42898047\n",
      "  0.6148529  -0.21276896  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2623901136469109\n",
      "\n",
      "step 176\n",
      "observation: [-0.19822322 -0.7363525  -0.946552    0.56274223 -0.09883098  0.5026157\n",
      "  0.63467526 -0.1391337   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.31234142827832717\n",
      "\n",
      "step 177\n",
      "observation: [-0.6486674  -0.55226433 -0.8816852   0.6179687  -0.03396424  0.55784214\n",
      "  0.699542   -0.08390728  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.3019043870363452\n",
      "\n",
      "step 178\n",
      "observation: [-0.48650056  0.08580173 -0.8330352   0.60938853  0.01468581  0.549262\n",
      "  0.748192   -0.09248745  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2454970089157709\n",
      "\n",
      "step 179\n",
      "observation: [-0.3648754   0.5643513  -0.7965476   0.55295336  0.05117336  0.49282685\n",
      "  0.7846796  -0.14892258  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.20376517136207684\n",
      "\n",
      "step 180\n",
      "observation: [ 0.22634344  0.42326346 -0.819182    0.51062703  0.02853901  0.4505005\n",
      "  0.76204526 -0.19124892  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.21986564772820658\n",
      "\n",
      "step 181\n",
      "observation: [ 0.16975757 -0.1825524  -0.83615774  0.52888227  0.01156325  0.46875572\n",
      "  0.7450695  -0.17299369  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18701191559538277\n",
      "\n",
      "step 182\n",
      "observation: [ 0.12731819  0.36308572 -0.8488895   0.4925737  -0.00116856  0.43244717\n",
      "  0.73233765 -0.20930226  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.264314656558173\n",
      "\n",
      "step 183\n",
      "observation: [ 0.09548864  0.27231428 -0.8584384   0.46534228 -0.01071743  0.40521574\n",
      "  0.7227888  -0.23653369  4.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.11240545572193206\n",
      "\n",
      "step 184\n",
      "observation: [ 0.07161648  0.70423573 -0.86560005  0.3949187  -0.01787908  0.33479217\n",
      "  0.71562713 -0.30695724  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.11074763756534939\n",
      "\n",
      "step 185\n",
      "observation: [ 0.05371236  0.02817678 -0.87097126  0.39210102 -0.02325031  0.33197448\n",
      "  0.7102559  -0.30977494  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.11478043400081842\n",
      "\n",
      "step 186\n",
      "observation: [ 0.5402843   0.02113259 -0.9249997   0.38998777 -0.07727874  0.32986122\n",
      "  0.65622747 -0.3118882   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.22164217334431785\n",
      "\n",
      "step 187\n",
      "observation: [ 0.4052132   0.01584944 -0.96552104  0.38840282 -0.11780006  0.32827628\n",
      "  0.61570615 -0.31347314  4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1462659751211123\n",
      "\n",
      "step 188\n",
      "observation: [ 0.8039099   0.01188708 -1.045912    0.38721412 -0.19819105  0.32708758\n",
      "  0.53531516 -0.31466183  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14986956048011912\n",
      "\n",
      "step 189\n",
      "observation: [ 0.10293242  0.00891531 -1.0562053   0.3863226  -0.20848429  0.32619604\n",
      "  0.5250219  -0.31555337  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.25271235319206264\n",
      "\n",
      "step 190\n",
      "observation: [ 0.07719932  0.00668648 -1.0639251   0.38565394 -0.21620423  0.3255274\n",
      "  0.517302   -0.316222    4.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.13522383881788425\n",
      "\n",
      "step 191\n",
      "observation: [-0.44210052  0.00501486 -1.0197152   0.38515246 -0.17199418  0.32502592\n",
      "  0.56151205 -0.3167235   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.11328943851279837\n",
      "\n",
      "step 192\n",
      "observation: [-0.8315754   0.00376115 -0.9365576   0.38477632 -0.08883663  0.3246498\n",
      "  0.6446696  -0.31709963  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.20591499267976798\n",
      "\n",
      "step 193\n",
      "observation: [-6.2368155e-01  2.8208597e-03 -8.7418944e-01  3.8449425e-01\n",
      " -2.6468480e-02  3.2436770e-01  7.0703775e-01 -3.1738171e-01\n",
      "  4.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.07557399295589837\n",
      "\n",
      "step 194\n",
      "observation: [-0.46776116  0.50211567 -0.8274133   0.3342827   0.02030763  0.27415615\n",
      "  0.75381386 -0.3675933   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03784932889750511\n",
      "\n",
      "step 195\n",
      "observation: [-0.35082087  0.87658674 -0.7923313   0.24662401  0.05538972  0.18649748\n",
      "  0.78889596 -0.45525196  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12125650034805935\n",
      "\n",
      "step 196\n",
      "observation: [-0.26311564  0.65744007 -0.7660197   0.18088001  0.08170129  0.12075347\n",
      "  0.8152075  -0.520996    4.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.007750009540492515\n",
      "\n",
      "step 197\n",
      "observation: [ 0.30266327  0.49308005 -0.796286    0.13157201  0.05143496  0.07144547\n",
      "  0.7849412  -0.570304    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.007959957928120175\n",
      "\n",
      "step 198\n",
      "observation: [ 0.22699745 -0.13018997 -0.81898576  0.144591    0.02873521  0.08446446\n",
      "  0.7622414  -0.55728495  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.020939054376966075\n",
      "\n",
      "step 199\n",
      "observation: [ 0.17024809 -0.5976425  -0.8360106   0.20435525  0.01171041  0.14422871\n",
      "  0.7452166  -0.49752071  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05714692867361033\n",
      "\n",
      "step 0\n",
      "observation: [ 0.          0.          0.07752181  1.1839113  -0.6803673   0.16968624\n",
      " -0.22296166 -0.22070721  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.42615631312201485\n",
      "\n",
      "step 1\n",
      "observation: [-0.5         0.          0.12752181  1.1839113  -0.6303673   0.16968624\n",
      " -0.17296167 -0.22070721  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.32349829162483984\n",
      "\n",
      "step 2\n",
      "observation: [-0.875       0.          0.2150218   1.1839113  -0.54286724  0.16968624\n",
      " -0.08546167 -0.22070721  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.24208497862130127\n",
      "\n",
      "step 3\n",
      "observation: [-0.65625     0.5         0.2806468   1.1339113  -0.47724226  0.11968625\n",
      " -0.01983667 -0.2707072   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.23526106125877413\n",
      "\n",
      "step 4\n",
      "observation: [ 0.0078125   0.375       0.27986556  1.0964113  -0.47802353  0.08218625\n",
      " -0.02061792 -0.3082072   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.28235057276811876\n",
      "\n",
      "step 5\n",
      "observation: [ 0.5058594   0.28125     0.22927962  1.0682863  -0.52860945  0.05406125\n",
      " -0.07120386 -0.3363322   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.32786126932158677\n",
      "\n",
      "step 6\n",
      "observation: [ 0.37939453 -0.2890625   0.19134016  1.0971925  -0.5665489   0.0829675\n",
      " -0.10914331 -0.30742595  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.307979844697637\n",
      "\n",
      "step 7\n",
      "observation: [-0.2154541  -0.21679688  0.21288557  1.1188723  -0.5450035   0.10464719\n",
      " -0.0875979  -0.28574628  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2847042361944469\n",
      "\n",
      "step 8\n",
      "observation: [-0.16159058  0.33740234  0.22904463  1.085132   -0.5288444   0.07090695\n",
      " -0.07143884 -0.3194865   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.21991188339173942\n",
      "\n",
      "step 9\n",
      "observation: [-0.62119293  0.25305176  0.29116392  1.0598269  -0.46672514  0.04560178\n",
      " -0.00931955 -0.34479168  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.27722276433485693\n",
      "\n",
      "step 10\n",
      "observation: [-0.4658947   0.18978882  0.3377534   1.0408479  -0.42013568  0.0266229\n",
      "  0.03726992 -0.36377054  2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.18954692671083048\n",
      "\n",
      "step 11\n",
      "observation: [ 0.15057898  0.14234161  0.3226955   1.0266138  -0.43519357  0.01238873\n",
      "  0.02221202 -0.37800473  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.202024883821694\n",
      "\n",
      "step 12\n",
      "observation: [ 0.11293423 -0.3932438   0.31140208  1.0659381  -0.446487    0.05171311\n",
      "  0.0109186  -0.33868033  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.20795976891382367\n",
      "\n",
      "step 13\n",
      "observation: [ 0.08470067  0.20506716  0.30293202  1.0454315  -0.45495707  0.0312064\n",
      "  0.00244853 -0.35918707  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.21713966979406937\n",
      "\n",
      "step 14\n",
      "observation: [ 0.06352551 -0.34619963  0.29657945  1.0800514  -0.4613096   0.06582636\n",
      " -0.00390402 -0.32456708  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.18154324473707115\n",
      "\n",
      "step 15\n",
      "observation: [-0.45235586 -0.25964972  0.34181505  1.1060164  -0.41607404  0.09179133\n",
      "  0.04133157 -0.29860213  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.199131234657537\n",
      "\n",
      "step 16\n",
      "observation: [ 0.1607331  -0.19473729  0.32574174  1.1254901  -0.43214732  0.11126506\n",
      "  0.02525826 -0.2791284   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.26007927934911745\n",
      "\n",
      "step 17\n",
      "observation: [ 0.6205498  -0.14605297  0.26368675  1.1400955  -0.49420232  0.12587036\n",
      " -0.03679672 -0.2645231   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2595501351789945\n",
      "\n",
      "step 18\n",
      "observation: [-0.03458763 -0.10953972  0.2671455   1.1510494  -0.49074355  0.13682432\n",
      " -0.03333796 -0.25356913  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3593264647890214\n",
      "\n",
      "step 19\n",
      "observation: [-0.02594073 -0.0821548   0.2697396   1.1592649  -0.4881495   0.14503981\n",
      " -0.03074389 -0.24535364  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3592561300989524\n",
      "\n",
      "step 20\n",
      "observation: [-0.01945554 -0.0616161   0.27168515  1.1654265  -0.48620394  0.15120143\n",
      " -0.02879833 -0.23919204  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.35925817676400673\n",
      "\n",
      "step 21\n",
      "observation: [-0.01459166 -0.04621207  0.2731443   1.1700476  -0.48474476  0.15582263\n",
      " -0.02733917 -0.23457083  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.24586168208302703\n",
      "\n",
      "step 22\n",
      "observation: [-0.01094374  0.46534094  0.27423868  1.1235136  -0.4836504   0.10928854\n",
      " -0.02624479 -0.28110492  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3386579993136826\n",
      "\n",
      "step 23\n",
      "observation: [-0.00820781  0.3490057   0.27505946  1.088613   -0.4828296   0.07438796\n",
      " -0.02542401 -0.3160055   2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.23253355840834833\n",
      "\n",
      "step 24\n",
      "observation: [-0.00615586  0.7617543   0.27567506  1.0124376  -0.48221403 -0.00178747\n",
      " -0.02480843 -0.39218092  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.33555676341254725\n",
      "\n",
      "step 25\n",
      "observation: [-0.00461689  0.5713157   0.27613673  0.955306   -0.48175234 -0.05891903\n",
      " -0.02434674 -0.44931248  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.25478523997396035\n",
      "\n",
      "step 26\n",
      "observation: [-0.00346267  0.92848676  0.276483    0.86245733 -0.48140606 -0.15176772\n",
      " -0.02400047 -0.54216117  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2608812335597956\n",
      "\n",
      "step 27\n",
      "observation: [-0.002597    0.19636509  0.2767427   0.8428208  -0.48114637 -0.17140423\n",
      " -0.02374077 -0.5617977   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2870725679164281\n",
      "\n",
      "step 28\n",
      "observation: [-0.00194775  0.64727384  0.27693748  0.77809346 -0.48095158 -0.23613161\n",
      " -0.023546   -0.62652504  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2862472988648915\n",
      "\n",
      "step 29\n",
      "observation: [-0.00146081 -0.01454464  0.27708358  0.7795479  -0.48080552 -0.23467714\n",
      " -0.02339991 -0.6250706   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.38563115163247685\n",
      "\n",
      "step 30\n",
      "observation: [-1.0956101e-03 -1.0908479e-02  2.7719313e-01  7.8063875e-01\n",
      " -4.8069596e-01 -2.3358630e-01 -2.3290353e-02 -6.2397975e-01\n",
      "  2.0000000e+00]\n",
      "action:      0  com:com active\n",
      "reward:      -0.38517061877104963\n",
      "\n",
      "step 31\n",
      "observation: [-8.2170754e-04 -8.1813587e-03  2.7727529e-01  7.8145689e-01\n",
      " -4.8061377e-01 -2.3276816e-01 -2.3208182e-02 -6.2316161e-01\n",
      "  2.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.2641106509329139\n",
      "\n",
      "step 32\n",
      "observation: [-6.1628065e-04 -5.0613600e-01  2.7733693e-01  8.3207047e-01\n",
      " -4.8055214e-01 -1.8215455e-01 -2.3146555e-02 -5.7254803e-01\n",
      "  0.0000000e+00]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20612736657304537\n",
      "\n",
      "step 33\n",
      "observation: [-0.50046223 -0.37960202  0.32738313  0.8700307  -0.43050593 -0.14419435\n",
      "  0.02689967 -0.5345878   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1310213784867822\n",
      "\n",
      "step 34\n",
      "observation: [-0.87534666 -0.2847015   0.4149178   0.89850086 -0.34297127 -0.11572421\n",
      "  0.11443433 -0.50611764  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.18581252564433562\n",
      "\n",
      "step 35\n",
      "observation: [-0.65651    -0.21352613  0.4805688   0.91985345 -0.27732027 -0.09437159\n",
      "  0.18008533 -0.48476505  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.06849695800187668\n",
      "\n",
      "step 36\n",
      "observation: [-0.4923825   0.3398554   0.52980703  0.88586795 -0.22808202 -0.12835713\n",
      "  0.22932358 -0.5187506   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.16020827383472147\n",
      "\n",
      "step 37\n",
      "observation: [-0.36928686  0.25489154  0.56673574  0.86037874 -0.19115333 -0.15384628\n",
      "  0.26625228 -0.54423976  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.15663438096449256\n",
      "\n",
      "step 38\n",
      "observation: [-0.27696514  0.19116867  0.59443223  0.8412619  -0.16345681 -0.17296316\n",
      "  0.29394877 -0.5633566   2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03921035511934521\n",
      "\n",
      "step 39\n",
      "observation: [-0.20772387 -0.3566235   0.61520463  0.8769243  -0.14268443 -0.1373008\n",
      "  0.31472117 -0.5276942   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.018167397243647854\n",
      "\n",
      "step 40\n",
      "observation: [-0.6557929  -0.26746762  0.6807839   0.903671   -0.07710513 -0.11055404\n",
      "  0.38030046 -0.5009475   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.014260788388812541\n",
      "\n",
      "step 41\n",
      "observation: [ 0.00815533 -0.20060071  0.6799684   0.9237311  -0.07792067 -0.09049397\n",
      "  0.37948492 -0.4808874   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.02221309493905966\n",
      "\n",
      "step 42\n",
      "observation: [ 0.5061165  -0.15045054  0.62935674  0.93877614 -0.12853232 -0.07544892\n",
      "  0.32887328 -0.46584237  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.02791992188146989\n",
      "\n",
      "step 43\n",
      "observation: [ 0.37958738 -0.6128379   0.591398    1.00006    -0.16649106 -0.01416512\n",
      "  0.29091454 -0.40455857  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.06101654923634022\n",
      "\n",
      "step 44\n",
      "observation: [ 0.7846905  -0.45962843  0.51292896  1.0460228  -0.24496011  0.03179772\n",
      "  0.21244548 -0.35859573  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.12957455185837044\n",
      "\n",
      "step 45\n",
      "observation: [ 1.0885179  -0.34472132  0.40407717  1.0804949  -0.3538119   0.06626985\n",
      "  0.1035937  -0.3241236   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.15705909491330627\n",
      "\n",
      "step 46\n",
      "observation: [ 0.31638843 -0.258541    0.37243834  1.106349   -0.38545075  0.09212396\n",
      "  0.07195485 -0.2982695   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1414456692769977\n",
      "\n",
      "step 47\n",
      "observation: [-0.2627087  -0.19390574  0.3987092   1.1257396  -0.35917988  0.11151452\n",
      "  0.09822573 -0.27887893  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.12102917994945846\n",
      "\n",
      "step 48\n",
      "observation: [-0.19703151  0.3545707   0.41841236  1.0902826  -0.3394767   0.07605746\n",
      "  0.11792888 -0.314336    0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.1054299584004305\n",
      "\n",
      "step 49\n",
      "observation: [-1.4777364e-01  7.6592803e-01  4.3318972e-01  1.0136898e+00\n",
      " -3.2469934e-01 -5.3534418e-04  1.3270624e-01 -3.9092880e-01\n",
      "  0.0000000e+00]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09841888413710641\n",
      "\n",
      "step 50\n",
      "observation: [-0.11083023  0.07444602  0.44427273  1.0062451  -0.31361634 -0.00797995\n",
      "  0.14378926 -0.3983734   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.09725087842611517\n",
      "\n",
      "step 51\n",
      "observation: [-0.08312267  0.55583453  0.452585    0.95066166 -0.30530408 -0.0635634\n",
      "  0.15210153 -0.45395684  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.20052055477669053\n",
      "\n",
      "step 52\n",
      "observation: [-0.062342    0.41687587  0.4588192   0.90897405 -0.29906985 -0.10525098\n",
      "  0.15833573 -0.49564445  2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.0941530897725252\n",
      "\n",
      "step 53\n",
      "observation: [-0.0467565  -0.18734309  0.46349487  0.9277084  -0.29439422 -0.08651668\n",
      "  0.16301137 -0.47691014  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1214555861618391\n",
      "\n",
      "step 54\n",
      "observation: [ 0.46493262 -0.14050731  0.4170016   0.9417591  -0.3408875  -0.07246594\n",
      "  0.11651812 -0.4628594   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.24502870699165474\n",
      "\n",
      "step 55\n",
      "observation: [ 0.34869948 -0.10538049  0.38213164  0.95229715 -0.37575743 -0.0619279\n",
      "  0.08164817 -0.45232135  2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.20714117595876247\n",
      "\n",
      "step 56\n",
      "observation: [ 0.7615246  -0.07903536  0.3059792   0.96020067 -0.4519099  -0.05402436\n",
      "  0.00549571 -0.4444178   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.21301653741097426\n",
      "\n",
      "step 57\n",
      "observation: [ 0.07114345 -0.05927652  0.29886484  0.96612835 -0.45902422 -0.04809671\n",
      " -0.00161864 -0.43849015  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.21567050986593395\n",
      "\n",
      "step 58\n",
      "observation: [ 0.05335759 -0.5444574   0.2935291   1.0205741  -0.46436     0.00634903\n",
      " -0.00695439 -0.3840444   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.3215890508207837\n",
      "\n",
      "step 59\n",
      "observation: [ 0.04001819 -0.40834305  0.28952727  1.0614084  -0.4683618   0.04718333\n",
      " -0.01095621 -0.34321013  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.2229565846935831\n",
      "\n",
      "step 60\n",
      "observation: [ 0.03001364  0.19374272  0.2865259   1.0420341  -0.4713632   0.02780906\n",
      " -0.01395758 -0.36258438  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.2743481410938031\n",
      "\n",
      "step 61\n",
      "observation: [ 0.52251023  0.14530703  0.23427488  1.0275034  -0.52361417  0.01327836\n",
      " -0.0662086  -0.3771151   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.37553252513413476\n",
      "\n",
      "step 62\n",
      "observation: [ 0.89188266  0.10898028  0.14508662  1.0166054  -0.61280245  0.00238033\n",
      " -0.15539686 -0.38801312  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.39654767146790904\n",
      "\n",
      "step 63\n",
      "observation: [ 0.16891201  0.08173521  0.1281954   1.0084319  -0.6296937  -0.00579319\n",
      " -0.17228808 -0.39618665  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.41646352033944173\n",
      "\n",
      "step 64\n",
      "observation: [ 0.12668401  0.5613014   0.11552701  0.95230174 -0.64236206 -0.06192333\n",
      " -0.18495648 -0.4523168   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.4278441075309096\n",
      "\n",
      "step 65\n",
      "observation: [ 0.095013   -0.07902394  0.10602571  0.9602041  -0.65186334 -0.05402093\n",
      " -0.19445777 -0.44441438  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.3731810622703978\n",
      "\n",
      "step 66\n",
      "observation: [-0.42874023 -0.05926796  0.14889973  0.9661309  -0.60898936 -0.04809414\n",
      " -0.15158375 -0.4384876   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2794591141270877\n",
      "\n",
      "step 67\n",
      "observation: [-0.8215552  -0.04445097  0.23105524  0.970576   -0.52683383 -0.04364904\n",
      " -0.06942823 -0.43404248  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.31805234367420976\n",
      "\n",
      "step 68\n",
      "observation: [-0.6161664  -0.03333823  0.2926719   0.97390985 -0.46521717 -0.04031522\n",
      " -0.00781159 -0.43070868  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18327638942696395\n",
      "\n",
      "step 69\n",
      "observation: [-0.4621248   0.47499633  0.33888438  0.9264102  -0.4190047  -0.08781485\n",
      "  0.03840089 -0.4782083   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1531147146501053\n",
      "\n",
      "step 70\n",
      "observation: [-0.3465936  -0.14375275  0.37354374  0.94078547 -0.38434535 -0.07343958\n",
      "  0.07306025 -0.46383303  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.14110716514481755\n",
      "\n",
      "step 71\n",
      "observation: [-0.25994518  0.39218545  0.39953825  0.9015669  -0.3583508  -0.11265812\n",
      "  0.09905477 -0.5030516   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.235007127597316\n",
      "\n",
      "step 72\n",
      "observation: [-0.1949589   0.2941391   0.41903415  0.87215304 -0.33885494 -0.14207202\n",
      "  0.11855066 -0.53246546  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23206651326376349\n",
      "\n",
      "step 73\n",
      "observation: [-0.14621918  0.2206043   0.43365607  0.8500926  -0.32423303 -0.16413246\n",
      "  0.13317257 -0.5545259   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23078041029132806\n",
      "\n",
      "step 74\n",
      "observation: [-0.10966438  0.16545323  0.4446225   0.8335473  -0.31326658 -0.18067779\n",
      "  0.14413902 -0.5710712   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23033297181197399\n",
      "\n",
      "step 75\n",
      "observation: [-0.08224829  0.12408993  0.45284733  0.82113826 -0.30504176 -0.19308677\n",
      "  0.15236385 -0.58348024  2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1125489317716186\n",
      "\n",
      "step 76\n",
      "observation: [-0.06168621 -0.40693256  0.45901594  0.86183155 -0.29887313 -0.15239352\n",
      "  0.15853247 -0.54278696  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.13335893386562245\n",
      "\n",
      "step 77\n",
      "observation: [ 0.45373535 -0.3051994   0.4136424   0.8923515  -0.34424666 -0.12187357\n",
      "  0.11315893 -0.51226705  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.25289110500950285\n",
      "\n",
      "step 78\n",
      "observation: [ 0.3403015  -0.22889957  0.37961227  0.9152414  -0.37827682 -0.09898362\n",
      "  0.07912878 -0.48937708  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.18042947518116556\n",
      "\n",
      "step 79\n",
      "observation: [ 0.25522614  0.32832533  0.35408965  0.8824089  -0.4037994  -0.13181615\n",
      "  0.05360617 -0.5222096   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.19020900757881443\n",
      "\n",
      "step 80\n",
      "observation: [ 0.1914196  -0.25375602  0.33494768  0.9077845  -0.4229414  -0.10644055\n",
      "  0.03446421 -0.496834    0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.157639936859689\n",
      "\n",
      "step 81\n",
      "observation: [-0.3564353  -0.190317    0.37059122  0.9268162  -0.38729787 -0.08740885\n",
      "  0.07010774 -0.4778023   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.14516951665589073\n",
      "\n",
      "step 82\n",
      "observation: [-0.26732647  0.35726225  0.39732388  0.89109    -0.36056522 -0.12313507\n",
      "  0.09684039 -0.5135285   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23842988583593144\n",
      "\n",
      "step 83\n",
      "observation: [-0.20049486  0.2679467   0.41737336  0.8642953  -0.34051573 -0.14992975\n",
      "  0.11688987 -0.5403232   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23484507890821224\n",
      "\n",
      "step 84\n",
      "observation: [-0.15037115  0.20096001  0.43241048  0.8441993  -0.3254786  -0.17002575\n",
      "  0.13192698 -0.5604192   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.23298330156775118\n",
      "\n",
      "step 85\n",
      "observation: [-0.11277836  0.15072002  0.4436883   0.8291273  -0.31420076 -0.18509775\n",
      "  0.14320482 -0.5754912   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2320520592310275\n",
      "\n",
      "step 86\n",
      "observation: [-0.08458377  0.11304001  0.45214668  0.8178233  -0.30574238 -0.19640175\n",
      "  0.1516632  -0.5867952   2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.16405510233153167\n",
      "\n",
      "step 87\n",
      "observation: [ 0.43656218  0.08478001  0.40849045  0.8093453  -0.3493986  -0.20487975\n",
      "  0.10800698 -0.5952732   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.17202934592422386\n",
      "\n",
      "step 88\n",
      "observation: [ 0.32742164 -0.436415    0.3757483   0.8529868  -0.3821408  -0.16123825\n",
      "  0.07526482 -0.5516317   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.1715661427379645\n",
      "\n",
      "step 89\n",
      "observation: [ 0.24556622 -0.8273112   0.35119167  0.93571794 -0.4066974  -0.07850713\n",
      "  0.0507082  -0.4689006   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.14098205381777007\n",
      "\n",
      "step 90\n",
      "observation: [-0.31582534 -0.62048346  0.3827742   0.99776626 -0.37511486 -0.01645878\n",
      "  0.08229073 -0.40685225  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.12991401735458816\n",
      "\n",
      "step 91\n",
      "observation: [-0.23686899 -0.96536255  0.40646112  1.0943025  -0.35142797  0.08007748\n",
      "  0.10597763 -0.31031597  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.10371462866914799\n",
      "\n",
      "step 92\n",
      "observation: [-0.67765176 -0.7240219   0.4742263   1.1667048  -0.2836628   0.15247966\n",
      "  0.1737428  -0.23791379  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.07879433855846896\n",
      "\n",
      "step 93\n",
      "observation: [-0.5082388  -0.04301645  0.52505016  1.1710063  -0.23283891  0.15678132\n",
      "  0.22456668 -0.23361214  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1635186829816063\n",
      "\n",
      "step 94\n",
      "observation: [-0.38117912 -0.03226234  0.56316805  1.1742326  -0.194721    0.16000755\n",
      "  0.26268458 -0.2303859   2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04023991000085559\n",
      "\n",
      "step 95\n",
      "observation: [-0.28588432  0.47580326  0.5917565   1.1266522  -0.16613257  0.11242723\n",
      "  0.29127303 -0.27796623  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1268248866466953\n",
      "\n",
      "step 96\n",
      "observation: [-0.21441326  0.35685244  0.6131978   1.090967   -0.14469124  0.07674198\n",
      "  0.31271437 -0.31365147  2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.0265362056505299\n",
      "\n",
      "step 97\n",
      "observation: [-0.16080993 -0.23236068  0.62927884  1.1142031  -0.12861025  0.09997804\n",
      "  0.32879534 -0.2904154   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12736774345565852\n",
      "\n",
      "step 98\n",
      "observation: [-0.12060745 -0.17427051  0.6413396   1.1316302  -0.1165495   0.1174051\n",
      "  0.3408561  -0.27298835  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.018033385477406856\n",
      "\n",
      "step 99\n",
      "observation: [-0.09045559  0.36929712  0.65038514  1.0947005  -0.10750394  0.08047539\n",
      "  0.34990165 -0.30991808  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.010152190588708549\n",
      "\n",
      "step 100\n",
      "observation: [-0.06784169  0.77697283  0.6571693   1.0170032  -0.10071977  0.0027781\n",
      "  0.35668582 -0.38761535  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.024288256003754338\n",
      "\n",
      "step 101\n",
      "observation: [ 0.44911873  0.58272964  0.6122574   0.9587302  -0.14563164 -0.05549486\n",
      "  0.31177396 -0.4458883   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1419946241880723\n",
      "\n",
      "step 102\n",
      "observation: [ 0.33683905  0.4370472   0.5785735   0.9150255  -0.17931555 -0.09919959\n",
      "  0.27809003 -0.48959303  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.07496839230937657\n",
      "\n",
      "step 103\n",
      "observation: [ 0.25262928  0.82778543  0.5533106   0.83224696 -0.20457847 -0.18197812\n",
      "  0.2528271  -0.5723716   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.20953000297449478\n",
      "\n",
      "step 104\n",
      "observation: [ 0.18947196  0.62083906  0.5343634   0.770163   -0.22352567 -0.24406204\n",
      "  0.23387992 -0.6344555   2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.11441881246831016\n",
      "\n",
      "step 105\n",
      "observation: [ 0.14210397 -0.0343707   0.520153    0.7736001  -0.23773608 -0.24062496\n",
      "  0.21966952 -0.6310184   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.09706124914047982\n",
      "\n",
      "step 106\n",
      "observation: [ 0.10657798 -0.52577806  0.5094952   0.8261779  -0.24839388 -0.18804716\n",
      "  0.20901172 -0.5784406   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.0754590894106557\n",
      "\n",
      "step 107\n",
      "observation: [ 0.07993349 -0.89433354  0.50150186  0.91561127 -0.25638723 -0.09861381\n",
      "  0.20101838 -0.48900726  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.046100907682455915\n",
      "\n",
      "step 108\n",
      "observation: [-0.4400499  -0.67075014  0.54550683  0.9826863  -0.21238223 -0.03153879\n",
      "  0.24502337 -0.42193225  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.03315211373122694\n",
      "\n",
      "step 109\n",
      "observation: [-0.33003742 -0.00306261  0.5785106   0.98299253 -0.1793785  -0.03123253\n",
      "  0.27802712 -0.42162597  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12487027799713676\n",
      "\n",
      "step 110\n",
      "observation: [-0.24752806 -0.00229695  0.6032634   0.98322225 -0.15462568 -0.03100284\n",
      "  0.3027799  -0.4213963   2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.03556924898625915\n",
      "\n",
      "step 111\n",
      "observation: [ 0.31435394 -0.00172272  0.571828    0.9833945  -0.18606108 -0.03083056\n",
      "  0.2713445  -0.42122403  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14489050779215432\n",
      "\n",
      "step 112\n",
      "observation: [ 2.3576547e-01 -1.2920366e-03  5.4825145e-01  9.8352367e-01\n",
      " -2.0963763e-01 -3.0701362e-02  2.4776797e-01 -4.2109481e-01\n",
      "  2.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05817147740579547\n",
      "\n",
      "step 113\n",
      "observation: [ 0.1768241   0.49903098  0.530569    0.9336206  -0.22732003 -0.08060446\n",
      "  0.23008557 -0.4709979   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1718111247080476\n",
      "\n",
      "step 114\n",
      "observation: [ 0.13261807  0.37427324  0.5173072   0.89619327 -0.24058184 -0.11803178\n",
      "  0.21682376 -0.50842524  2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07200002540444518\n",
      "\n",
      "step 115\n",
      "observation: [ 0.09946355 -0.21929508  0.5073609   0.91812277 -0.2505282  -0.09610227\n",
      "  0.2068774  -0.48649573  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17290273213485796\n",
      "\n",
      "step 116\n",
      "observation: [ 0.07459766 -0.16447131  0.49990112  0.9345699  -0.25798798 -0.07965514\n",
      "  0.19941764 -0.4700486   2.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.05014956317223208\n",
      "\n",
      "step 117\n",
      "observation: [-0.44405174 -0.12335348  0.5443063   0.94690526 -0.21358278 -0.0673198\n",
      "  0.2438228  -0.45771325  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1358724106975989\n",
      "\n",
      "step 118\n",
      "observation: [-0.3330388  -0.09251511  0.57761014  0.9561568  -0.18027891 -0.05806828\n",
      "  0.2771267  -0.44846174  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12673264123075062\n",
      "\n",
      "step 119\n",
      "observation: [-0.2497791  -0.06938633  0.60258806  0.9630954  -0.155301   -0.05112965\n",
      "  0.3021046  -0.4415231   2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.036916624034468916\n",
      "\n",
      "step 120\n",
      "observation: [ 0.31266567 -0.05203975  0.5713215   0.9682994  -0.18656756 -0.04592567\n",
      "  0.27083802 -0.43631914  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.052575522421673244\n",
      "\n",
      "step 121\n",
      "observation: [ 0.23449925  0.4609702   0.5478716   0.92220235 -0.21001749 -0.09202269\n",
      "  0.24738811 -0.48241615  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.04756992257996151\n",
      "\n",
      "step 122\n",
      "observation: [-0.32412556  0.34572765  0.5802841   0.88762957 -0.17760493 -0.12659545\n",
      "  0.27980065 -0.51698893  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.06459295228674644\n",
      "\n",
      "step 123\n",
      "observation: [ 0.25690582  0.25929573  0.55459356  0.8617     -0.20329551 -0.15252504\n",
      "  0.25411007 -0.5429185   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.06441171131749215\n",
      "\n",
      "step 124\n",
      "observation: [ 0.19267938 -0.3055282   0.53532565  0.89225286 -0.22256346 -0.12197221\n",
      "  0.23484214 -0.51236564  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.16598823638596855\n",
      "\n",
      "step 125\n",
      "observation: [ 0.14450952 -0.22914615  0.5208747   0.91516745 -0.23701441 -0.09905759\n",
      "  0.22039118 -0.48945105  2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.0954191487475415\n",
      "\n",
      "step 126\n",
      "observation: [ 0.60838217 -0.1718596   0.46003646  0.93235344 -0.29785264 -0.08187164\n",
      "  0.15955298 -0.4722651   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.09088980213391905\n",
      "\n",
      "step 127\n",
      "observation: [-0.04371339 -0.12889472  0.4644078   0.9452429  -0.2934813  -0.06898216\n",
      "  0.1639243  -0.45937562  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08430442297514674\n",
      "\n",
      "step 128\n",
      "observation: [-0.03278504 -0.59667104  0.4676863   1.00491    -0.29020277 -0.00931506\n",
      "  0.16720282 -0.3997085   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.0830086835775382\n",
      "\n",
      "step 129\n",
      "observation: [-0.02458878  0.05249672  0.47014517  0.9996603  -0.2877439  -0.01456473\n",
      "  0.1696617  -0.4049582   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.05599101161687459\n",
      "\n",
      "step 130\n",
      "observation: [-0.51844156  0.03937254  0.52198935  0.99572307 -0.23589975 -0.01850199\n",
      "  0.22150585 -0.40889543  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1392758635163115\n",
      "\n",
      "step 131\n",
      "observation: [-0.3888312   0.02952941  0.56087244  0.99277014 -0.19701663 -0.02145493\n",
      "  0.26038897 -0.4118484   2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.033602275305321755\n",
      "\n",
      "step 132\n",
      "observation: [-0.29162338  0.52214706  0.5900348   0.9405554  -0.16785428 -0.07366963\n",
      "  0.28955132 -0.46406308  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.13404165713075328\n",
      "\n",
      "step 133\n",
      "observation: [-0.21871755  0.3916103   0.6119065   0.90139437 -0.14598253 -0.11283066\n",
      "  0.31142306 -0.50322413  2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.025291747379784194\n",
      "\n",
      "step 134\n",
      "observation: [-0.16403815 -0.20629229  0.6283104   0.9220236  -0.12957871 -0.09220143\n",
      "  0.3278269  -0.48259488  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.029813994803472778\n",
      "\n",
      "step 135\n",
      "observation: [-0.12302862  0.3452808   0.6406132   0.8874955  -0.11727585 -0.12672952\n",
      "  0.34012973 -0.517123    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.022206530731509494\n",
      "\n",
      "step 136\n",
      "observation: [-0.09227146 -0.24103941  0.64984035  0.91159946 -0.10804871 -0.10262557\n",
      "  0.3493569  -0.493019    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.011420480005494633\n",
      "\n",
      "step 137\n",
      "observation: [-0.0692036  -0.6807796   0.65676075  0.97967744 -0.10112835 -0.03454762\n",
      "  0.35627726 -0.42494106  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10947672244826875\n",
      "\n",
      "step 138\n",
      "observation: [-0.0519027  -0.51058465  0.661951    1.0307359  -0.09593807  0.01651085\n",
      "  0.3614675  -0.3738826   2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.00849543580684755\n",
      "\n",
      "step 139\n",
      "observation: [-0.03892702  0.1170615   0.6658437   1.0190297  -0.09204537  0.0048047\n",
      "  0.36536023 -0.38558877  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10795921636706153\n",
      "\n",
      "step 140\n",
      "observation: [-0.02919527  0.08779612  0.6687632   1.0102501  -0.08912585 -0.00397491\n",
      "  0.36827976 -0.39436838  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.011225370840566948\n",
      "\n",
      "step 141\n",
      "observation: [-0.02189645  0.5658471   0.67095286  0.95366544 -0.0869362  -0.06055962\n",
      "  0.3704694  -0.45095307  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.028913077252569433\n",
      "\n",
      "step 142\n",
      "observation: [ 0.48357767  0.4243853   0.62259513  0.9112269  -0.13529396 -0.10299815\n",
      "  0.32211164 -0.4933916   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14761195415838935\n",
      "\n",
      "step 143\n",
      "observation: [ 0.36268324  0.31828898  0.5863268   0.879398   -0.1715623  -0.13482705\n",
      "  0.2858433  -0.5252205   2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05132235623873317\n",
      "\n",
      "step 144\n",
      "observation: [ 0.27201244 -0.26128325  0.55912554  0.90552634 -0.19876353 -0.10869873\n",
      "  0.25864205 -0.4990922   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.08038876321811506\n",
      "\n",
      "step 145\n",
      "observation: [ 0.70400935 -0.19596244  0.48872462  0.92512256 -0.26916447 -0.08910248\n",
      "  0.18824112 -0.47949594  0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.14389423432075182\n",
      "\n",
      "step 146\n",
      "observation: [ 1.028007   -0.14697184  0.3859239   0.93981975 -0.37196517 -0.0744053\n",
      "  0.08544043 -0.46479875  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.21451556706032224\n",
      "\n",
      "step 147\n",
      "observation: [ 0.7710053   0.38977113  0.30882338  0.90084267 -0.4490657  -0.11338241\n",
      "  0.0083399  -0.5037759   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.26551615633486964\n",
      "\n",
      "step 148\n",
      "observation: [ 0.5782539  -0.20767166  0.250998    0.9216098  -0.5068911  -0.09261525\n",
      " -0.04948549 -0.4830087   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.40872135311226176\n",
      "\n",
      "step 149\n",
      "observation: [ 0.43369046 -0.15575375  0.20762894  0.93718517 -0.5502601  -0.07703988\n",
      " -0.09285454 -0.46743333  2.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.2881336088562019\n",
      "\n",
      "step 150\n",
      "observation: [-0.17473216 -0.11681531  0.22510216  0.9488667  -0.5327869  -0.06535834\n",
      " -0.07538132 -0.4557518   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.22380442546946028\n",
      "\n",
      "step 151\n",
      "observation: [-0.6310491  -0.08761148  0.28820708  0.95762783 -0.469682   -0.0565972\n",
      " -0.01227641 -0.44699064  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.28088495922018775\n",
      "\n",
      "step 152\n",
      "observation: [-0.47328684 -0.06570861  0.33553576  0.9641987  -0.42235333 -0.05002633\n",
      "  0.03505228 -0.4404198   2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.2516920340092992\n",
      "\n",
      "step 153\n",
      "observation: [-0.35496512 -0.04928146  0.37103227  0.9691269  -0.3868568  -0.04509819\n",
      "  0.07054879 -0.43549165  2.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.1700064115065708\n",
      "\n",
      "step 154\n",
      "observation: [ 0.23377615 -0.03696109  0.34765464  0.97282296 -0.41023442 -0.04140208\n",
      "  0.04717118 -0.43179554  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.1442006600047842\n",
      "\n",
      "step 155\n",
      "observation: [-0.3246679  -0.02772082  0.38012144  0.97559506 -0.37776762 -0.03863\n",
      "  0.07963797 -0.42902344  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.13239502186858562\n",
      "\n",
      "step 156\n",
      "observation: [-0.24350092  0.4792094   0.40447155  0.9276741  -0.35341755 -0.08655094\n",
      "  0.10398806 -0.4769444   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.14208222017852432\n",
      "\n",
      "step 157\n",
      "observation: [-0.18262568  0.85940707  0.4227341   0.8417334  -0.33515498 -0.17249164\n",
      "  0.12225062 -0.5628851   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.19412503878418402\n",
      "\n",
      "step 158\n",
      "observation: [ 0.36303073  0.6445553   0.38643104  0.7772779  -0.37145805 -0.23694716\n",
      "  0.08594755 -0.6273406   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.21431083357535707\n",
      "\n",
      "step 159\n",
      "observation: [ 0.27227306 -0.01658354  0.35920373  0.77893627 -0.39868534 -0.23528881\n",
      "  0.05872025 -0.6256823   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.33042677979262847\n",
      "\n",
      "step 160\n",
      "observation: [ 0.20420478 -0.01243766  0.33878323  0.78018004 -0.41910583 -0.23404504\n",
      "  0.03829977 -0.6244385   2.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.20212095820243642\n",
      "\n",
      "step 161\n",
      "observation: [-0.3468464  -0.00932824  0.3734679   0.78111285 -0.3844212  -0.23311222\n",
      "  0.0729844  -0.62350565  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.16173044307303927\n",
      "\n",
      "step 162\n",
      "observation: [-0.26013482 -0.50699615  0.39948136  0.83181244 -0.3584077  -0.18241261\n",
      "  0.09899789 -0.57280606  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.10430968460730415\n",
      "\n",
      "step 163\n",
      "observation: [-0.6951011  -0.38024715  0.4689915   0.86983716 -0.2888976  -0.14438789\n",
      "  0.168508   -0.53478134  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1694833795290858\n",
      "\n",
      "step 164\n",
      "observation: [-0.5213258  -0.28518534  0.52112406  0.8983557  -0.23676501 -0.11586936\n",
      "  0.22064058 -0.50626284  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14799823659165615\n",
      "\n",
      "step 165\n",
      "observation: [-0.39099437 -0.21388902  0.5602235   0.9197446  -0.19766557 -0.09448045\n",
      "  0.25974002 -0.48487392  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.044835211749248094\n",
      "\n",
      "step 166\n",
      "observation: [-0.2932458   0.33958325  0.58954805  0.8857863  -0.168341   -0.12843877\n",
      "  0.2890646  -0.5188322   0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.06223989165974833\n",
      "\n",
      "step 167\n",
      "observation: [ 0.28006566  0.25468743  0.5615415   0.8603175  -0.19634756 -0.15390752\n",
      "  0.26105803 -0.544301    0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.1771742464344618\n",
      "\n",
      "step 168\n",
      "observation: [ 0.21004926  0.19101557  0.5405366   0.84121597 -0.2173525  -0.17300908\n",
      "  0.2400531  -0.56340253  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.18943298773903092\n",
      "\n",
      "step 169\n",
      "observation: [ 0.15753694  0.14326169  0.5247829   0.8268898  -0.23310618 -0.18733525\n",
      "  0.22429942 -0.5777287   2.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08191416756793922\n",
      "\n",
      "step 170\n",
      "observation: [ 0.1181527  -0.39255375  0.51296765  0.8661452  -0.24492146 -0.14807987\n",
      "  0.21248414 -0.5384733   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.05560252950314927\n",
      "\n",
      "step 171\n",
      "observation: [-0.41138548 -0.2944153   0.5541062   0.8955867  -0.2037829  -0.11863834\n",
      "  0.25362268 -0.5090318   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03207201031639619\n",
      "\n",
      "step 172\n",
      "observation: [-0.3085391  -0.7208115   0.5849601   0.9676679  -0.172929   -0.0465572\n",
      "  0.2844766  -0.43695065  0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.010014062575504792\n",
      "\n",
      "step 173\n",
      "observation: [-0.7314043  -0.5406086   0.6581005   1.0217288  -0.09978856  0.00750367\n",
      "  0.35761702 -0.38288978  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10432773249723924\n",
      "\n",
      "step 174\n",
      "observation: [-0.5485532  -0.40545645  0.71295583  1.0622743  -0.04493324  0.04804931\n",
      "  0.41247237 -0.34234414  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.0008242662493462193\n",
      "\n",
      "step 175\n",
      "observation: [-0.41141492  0.19590765  0.75409734  1.0426836  -0.00379175  0.02845855\n",
      "  0.45361385 -0.3619349   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.0020454212902049533\n",
      "\n",
      "step 176\n",
      "observation: [-0.3085612   0.64693075  0.7849535   0.9779905   0.02706437 -0.03623453\n",
      "  0.48446998 -0.426628    0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.007183339552621554\n",
      "\n",
      "step 177\n",
      "observation: [ 2.6857910e-01  4.8519805e-01  7.5809556e-01  9.2947072e-01\n",
      "  2.0646406e-04 -8.4754333e-02  4.5761207e-01 -4.7514778e-01\n",
      "  0.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.029687815476660585\n",
      "\n",
      "step 178\n",
      "observation: [ 0.20143433  0.8638985   0.7379521   0.8430809  -0.01993697 -0.17114419\n",
      "  0.43746862 -0.5615376   0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.035800530933865676\n",
      "\n",
      "step 179\n",
      "observation: [ 0.15107575  0.1479239   0.72284454  0.8282885  -0.03504454 -0.18593659\n",
      "  0.42236105 -0.57633     0.        ]\n",
      "action:      2  com:no_com\n",
      "reward:      -0.04810934775558835\n",
      "\n",
      "step 180\n",
      "observation: [ 0.6133068   0.11094293  0.66151386  0.81719416 -0.09637523 -0.19703087\n",
      "  0.36103037 -0.58742434  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.04440425446854221\n",
      "\n",
      "step 181\n",
      "observation: [ 0.4599801  -0.4167928   0.6155158   0.8588735  -0.14237323 -0.1553516\n",
      "  0.31503236 -0.545745    0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.03677325688919411\n",
      "\n",
      "step 182\n",
      "observation: [ 0.34498507 -0.8125946   0.5810173   0.9401329  -0.17687175 -0.07409213\n",
      "  0.28053385 -0.4644856   0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.14127864528085743\n",
      "\n",
      "step 183\n",
      "observation: [ 0.25873882 -0.6094459   0.5551435   1.0010775  -0.20274562 -0.01314754\n",
      "  0.25465998 -0.403541    2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.04965520354603972\n",
      "\n",
      "step 184\n",
      "observation: [ 0.19405411  0.04291554  0.53573805  0.99678594 -0.22215104 -0.01743909\n",
      "  0.23525456 -0.40783253  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.05689026778245849\n",
      "\n",
      "step 185\n",
      "observation: [ 0.14554058 -0.46781334  0.52118397  1.0435673  -0.2367051   0.02934225\n",
      "  0.2207005  -0.3610512   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.04320491442288245\n",
      "\n",
      "step 186\n",
      "observation: [-0.39084455 -0.35086     0.56026846  1.0786533  -0.19762063  0.06442825\n",
      "  0.25978497 -0.3259652   0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.029987316287389404\n",
      "\n",
      "step 187\n",
      "observation: [-0.29313344  0.23685499  0.5895818   1.0549678  -0.16830729  0.04074275\n",
      "  0.2890983  -0.3496507   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.009805999907948995\n",
      "\n",
      "step 188\n",
      "observation: [-0.71985006  0.17764124  0.6615668   1.0372037  -0.09632228  0.02297862\n",
      "  0.3610833  -0.36741483  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10188535709958438\n",
      "\n",
      "step 189\n",
      "observation: [-0.53988755  0.13323094  0.71555555  1.0238806  -0.04233353  0.00965553\n",
      "  0.41507205 -0.38073793  2.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.10000350626118223\n",
      "\n",
      "step 190\n",
      "observation: [-4.0491566e-01  9.9923201e-02  7.5604713e-01  1.0138882e+00\n",
      " -1.8419647e-03 -3.3679002e-04  4.5556363e-01 -3.9073023e-01\n",
      "  2.0000000e+00]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.004158201220628885\n",
      "\n",
      "step 191\n",
      "observation: [-0.30368674  0.5749424   0.78641576  0.956394    0.02852671 -0.05783103\n",
      "  0.4859323  -0.44822448  0.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.005228096738669134\n",
      "\n",
      "step 192\n",
      "observation: [-0.22776505 -0.0687932   0.8091923   0.96327335  0.05130322 -0.05095171\n",
      "  0.50870883 -0.44134516  0.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.013852739106258779\n",
      "\n",
      "step 193\n",
      "observation: [-0.1708238   0.4484051   0.8262747   0.91843283  0.06838559 -0.09579222\n",
      "  0.52579117 -0.48618567  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.12334322460450722\n",
      "\n",
      "step 194\n",
      "observation: [-0.12811784  0.33630383  0.8390865   0.88480246  0.08119738 -0.1294226\n",
      "  0.53860295 -0.51981604  2.        ]\n",
      "action:      4  com:no_com\n",
      "reward:      -0.05012550475578641\n",
      "\n",
      "step 195\n",
      "observation: [-0.09608839  0.75222784  0.8486953   0.8095797   0.09080622 -0.2046454\n",
      "  0.5482118  -0.59503883  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.17776013651113667\n",
      "\n",
      "step 196\n",
      "observation: [-0.07206629  0.5641709   0.8559019   0.75316256  0.09801285 -0.26106247\n",
      "  0.55541843 -0.65145594  1.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.07489428654100716\n",
      "\n",
      "step 197\n",
      "observation: [-0.05404972 -0.07687183  0.8613069   0.7608498   0.10341782 -0.2533753\n",
      "  0.56082344 -0.6437687   0.        ]\n",
      "action:      1  com:no_com\n",
      "reward:      -0.08610795614252505\n",
      "\n",
      "step 198\n",
      "observation: [-0.5405373  -0.05765387  0.9153606   0.76661515  0.15747155 -0.24760991\n",
      "  0.61487716 -0.63800335  0.        ]\n",
      "action:      0  com:com active\n",
      "reward:      -0.19839670526745157\n",
      "\n",
      "step 199\n",
      "observation: [-0.40540296 -0.0432404   0.9559009   0.7709392   0.19801185 -0.24328586\n",
      "  0.65541744 -0.63367933  1.        ]\n",
      "action:      3  com:no_com\n",
      "reward:      -0.08829063761567983\n",
      "\n",
      "that all - part 4\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from gym import Wrapper\n",
    "from copy import deepcopy\n",
    "from pettingzoo.mpe import simple_speaker_listener_v3\n",
    "from stable_baselines3 import PPO\n",
    "\n",
    "# speaker action-to-index dict\n",
    "SPEAKER_DISCRETE_ACTIONS = {\n",
    "\n",
    "    'A': 0,\n",
    "    'B': 1,\n",
    "    'C': 2,\n",
    "    'nothing' : 3,\n",
    "\n",
    "}\n",
    "\n",
    "# listener action-to-index dict\n",
    "LISTENER_DISCRETE_ACTIONS = {\n",
    "    'nothing': 0,\n",
    "    'left':    1,\n",
    "    'right':   2,\n",
    "    'down':    3,\n",
    "    'up':      4\n",
    "}\n",
    "\n",
    "from gym.spaces import Discrete, Box\n",
    "\n",
    "\"\"\"#### Custom gym Wrapper\n",
    "\"\"\"\n",
    "class ListenerOnlyCOMWrapper(Wrapper):\n",
    "\n",
    "    def __init__(self, env, com_allways_on = False, com_reward = -0.1):\n",
    "        super().__init__(env)\n",
    "        self.com_allways_on = com_allways_on\n",
    "        self.com_reward = com_reward\n",
    "\n",
    "        # reset to skip speaker before new game\n",
    "        self.obs = self.reset()\n",
    "\n",
    "        # set single agent list\n",
    "        self.agents = self.agents[1:]\n",
    "\n",
    "        self.observation_space = self.get_observation_space()\n",
    "\n",
    "        #set action space (listener)\n",
    "        self.action_space = env.action_spaces[self.agents[0]]\n",
    "\n",
    "    def get_observation_space(self):\n",
    "        return Box(low=-np.inf,high=np.inf , shape= (len(self.obs),), dtype=np.float32)\n",
    "\n",
    "    def reset(self):\n",
    "        super().reset()\n",
    "\n",
    "\n",
    "        # skip speaker action\n",
    "        self.__step_speaker()\n",
    "        self.obs,_,_,_ = self.env.last()\n",
    "        self.obs = self.fix_obs(self.obs, self.com_allways_on)\n",
    "        return self.obs\n",
    "\n",
    "    def step(self, action):\n",
    "        # CHANGE ACTION AS NEEDED\n",
    "        com = action == 0\n",
    "        super().step(action)\n",
    "        ob, _, done, _ = self.env.last()  # do listener action\n",
    "\n",
    "        # skip speaker action - if com==True, speaker 'speaks' else - 0 speak will be performed\n",
    "        self.__step_speaker()\n",
    "\n",
    "        step_rets = self.env.last()\n",
    "\n",
    "        if self.com_allways_on:  com = True\n",
    "\n",
    "        self.obs = self.fix_obs(step_rets[0], com)\n",
    "\n",
    "        reward_com = 0\n",
    "        if com:\n",
    "            # get - minus-reward for com use - adjust reward by initializing\n",
    "            if not self.com_allways_on: reward_com = self.com_reward\n",
    "\n",
    "        return (self.obs,step_rets[1] + reward_com, step_rets[2], step_rets[3])\n",
    "\n",
    "    def fix_obs(self, obs, com):\n",
    "        if len(obs)<5: return obs\n",
    "        obs = np.array(obs[:-2])\n",
    "        point = int(obs[-1])\n",
    "        if not com or point==3:\n",
    "            obs[-1] = 0\n",
    "            return obs\n",
    "        try:\n",
    "            x = obs[2+point*2]\n",
    "            y = obs[3+point*2]\n",
    "        except:\n",
    "            print('out of bounds')\n",
    "        myradians = math.atan2(y, x)\n",
    "        deg = int(math.degrees(myradians))\n",
    "        if deg<0:\n",
    "            deg+=360\n",
    "        output = deg // 45\n",
    "        obs[-1] = (output+1)\n",
    "        return obs\n",
    "\n",
    "    def __step_speaker(self):\n",
    "        goal_color, _, done, _ = self.env.last()\n",
    "\n",
    "        # speaker is done before the listener.\n",
    "        if done:\n",
    "            return\n",
    "\n",
    "        # step with the correct action type\n",
    "        if self.env.unwrapped.continuous_actions:\n",
    "            super().step(goal_color)\n",
    "        else:\n",
    "            super().step(np.argmax(goal_color))\n",
    "\n",
    "env = simple_speaker_listener_v3.env(max_cycles=200, continuous_actions=False)\n",
    "env = ListenerOnlyCOMWrapper(env, com_allways_on=False)\n",
    "print(f'custom wrapped environment: {env}')\n",
    "\n",
    "#train PPO agent method\n",
    "def train_model(env):\n",
    "    env_copy = deepcopy(env)\n",
    "    env_copy.reset()\n",
    "    model = PPO(\"MlpPolicy\", env_copy, verbose=1)\n",
    "    for batch in range(200):\n",
    "        print(f\"batch: {batch}\")\n",
    "        model.learn(total_timesteps=8000, n_eval_episodes=40)\n",
    "        env_copy.reset()\n",
    "    return model\n",
    "\n",
    "def get_ppo_agent(is_com_active):\n",
    "    if is_com_active:\n",
    "        model_file_name = 'S-L_PPO_base'\n",
    "    else:\n",
    "        model_file_name = 'S-L_PPO_com10'\n",
    "    try:\n",
    "        model = PPO.load(model_file_name)\n",
    "    except:\n",
    "        model = train_model(env)\n",
    "        model.save(model_file_name)\n",
    "    return model\n",
    "\n",
    "# run (n_iter) episodes after training and render\n",
    "def run(env,model,n_iter):\n",
    "    for m in range(n_iter):\n",
    "        observation = env.reset()\n",
    "        env.render()\n",
    "        for i in range(env.unwrapped.max_cycles):\n",
    "            # choose an action and execute\n",
    "            action = model.predict(observation)\n",
    "            action = action[0]\n",
    "            print(f'step {i}')\n",
    "            print(f'observation: {observation}')\n",
    "            observation, reward, done, info = env.step(action)\n",
    "\n",
    "            # log everything\n",
    "            c = \"no_com\"\n",
    "            if action == 0 or env.com_allways_on: c = \"com active\"\n",
    "            print(f'action:      {action}  com:{c}')\n",
    "            print(f'reward:      {reward}')\n",
    "            print()\n",
    "\n",
    "            # if done, the episode is complete. no more actions can be taken\n",
    "            if done:\n",
    "                break\n",
    "\n",
    "            env.render()\n",
    "        env.close()\n",
    "\n",
    "model = get_ppo_agent(env.com_allways_on)\n",
    "\n",
    "run(env,model,5)\n",
    "\n",
    "print(\"that all - part 4\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1361a98-458c-414b-a5f5-77b43c3b0218",
   "metadata": {},
   "source": [
    "## Method Weaknesses, Issues and Failures\n",
    "\n",
    "- Adding complexity, to a complexed problem. 5 stages, all must be reasonable for a good communication model\n",
    "- sending the wrong message or too much information can result in worse performence\n",
    "- Does not worth the effort when a central 'brain' available\n",
    "- Implicit COM - domain knowlege (and ussualy obs space of other agents) dependant\n",
    "- Communication (explicit) is exposed to Noise, Jamming, and Spoofing\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}