2024-07-15 ODD working group weekly meeting (197)

[Igata-ctrl]

Attendee

• Aniket Salvi (Technical University of Munich)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke MITSUDOME (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On June 16, 2024, 446/642 regular and prototype scenarios succeeded.

  • 126/141 public road bus scenarios were successful.
  • 103/220 pull over scenarios were successful.
  • 185/198 pull out scenarios were successful.
  • 18/83 prototype scenarios including low speed ODD were successful. Berkay kindly another scenario.
  • The overall result this week was similar to the May 13, 2024 result.

• On June 23, 2024, 442/642 regular and prototype scenarios succeeded.

  • 127/141 public road bus scenarios were successful.
  • 114/220 pull over scenarios were successful.
  • 182/198 pull out scenarios were successful.
  • 19/83 prototype scenarios including low speed ODD were successful.
  • The overall result this week was similar to last week.

• On June 30, 2024, No simulation result was obtained due to SimulationError:AutowareError.

  • Identical to the simulation result of May 19 and 26.
    • Message : Simulator waited for the Autoware state to transition to WaitingForRoute, but time is up. The current Autoware state is EMERGENCY.
    • The term "Emergency" hints that this error is associated with the Autonomous Emergency Braking (AEB) module of Autoware.

• On July 7, 2024, 445/642 regular and prototype scenarios succeeded.

  • 127/141 public road bus scenarios were successful. 100% identical to the June 23 result.
  • 116/220 pull over scenarios were successful.
  • 186/198 pull out scenarios were successful.
  • 16/83 prototype scenarios including low speed ODD were successful.

• Based on the discussion last week, following changes have been applied to Autoware Evaluator settings.

  • "UC-F-16_ArvDpt_PullOver_zero#2" suite has been excluded from the weekly simulation scenario set, as the default speed limit and the Automated Emergency Brake feature of Autoware.universe may take some more time to be addressed. The total number of the scenarios including the parameter variation is now 422.
  • "test scenarios" suite has been renamed to "Low Speed Vehicles" suite to reflect the actual content of it.
  • The simulation logs expire in 45 days as opposed to 60 days, the old setting in order to save/manage storage space on AWS.

• On July 14, 2024, 329/422 regular and prototype scenarios succeeded.

  • 127/141 public road bus scenarios were successful. 100% identical to July 7 and June 23 result.
    • UC-23-004-0001-With2Obstacles-ShalunRS : EGO doesn't stop against the detected NPS from the left.
    • UC-LD-001-0004-1-Gebze, UC-LD-001-0004-2-Gebze : NPC's unsafe behavior.
    • UC-PB-001-0003-Hsinchu : Failed in obstacle avoidance (insufficient lateral space).
    • UC-PB-005-0001-(1|2|3)-Hsinchu : EGO doesn't start after the cross traffic passed.
  • 184/198 pull out scenarios were successful. Identical to June16 result except for one case.
    • UC-v2-F-15-00003_003_case01 : Failed when EGO conducts the expected backup.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : It looks like the EGO destination is set behind start position (failed cases).
    • UC-v2-F-15-00007_002_case03_cmn_general : EGO doesn't reach the destintion.
    • UC-v2-F-15-00007_004_case01_cmn_general : Log size limit exceeded (for only one case).
    • UC-v2-F-15-00101_001_case01_cmn_general : It looks like start planner (red wall) is preventing EGO from starting in a complex curve.
    • UC-v2-F-15-00103_001_case01_cmn_general : Obstacle stop prevents EGO from proceeding (may better modify NPC1 position).
  • 18/83 Low Speed Vehicles were successful.
    • Test-dense-urban_ITRI(2|3|4) : It looks like EGO cannot use the space in the opposite lane to avoid NPC/Peds.
    • Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC is the round about.
    • Test_MRM_Fault_Injection1 : It looks like MRM not conducted upon the fault injection.
    • UC-NTR-001-0001_case(1|2), UC-NTR-001-0003_case(1|2), UC-NTR-001-0005 : EGO's "reach destination" condition is not met though the rude NPS behaves as intended.
    • UC-NTR-001-0007 : It looks like something is wrong with NPC initial position.
    • UC-NTR-002-0001_case(1|2), UC-NTR-002-0002_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot over take NPC using the space outside of the lane.
    • UC-NTR-002-0004 : NPC2 collides with EGO albeit the "isBLIND:false" setting.
    • UC-NTR-003-0001 : It looks like EGO's obstacle stop is too late.
    • UC-NTR-003-0002 : Fails with excess deceleration condition when EGO detects the oncoming NPC.
    • UC-NTR-003-0003 : EGO cannot avoid the collision in the intersection (too late).
    • UC-NTR-004-0001 : It looks like WGO is not trying to avoid the parked NPCs within the lane.
    • (19 more failure cases to be analyzed.)
    • UC-NTR-005-0001 : Simulation does not start (time out).
    • UC-VRU-001-0002 : Fails with excess deceleration condition when EGO detects the pedestrian (jay walker).
    • UC-VRU-002-0001_case(1|2) : EGO cannot pass the parked NPC using the space outside of the lane (obstacle stop).
    • UC-VRU-002-0002_case(1|2) : Fails with excess deceleration condition when EGO tries to pass the bicycle.
    • UC-VRU-002-0003_case(1|2) : EGO cannot overtake the bicycle due to insufficient lane width.
    • UC-VRU-002-0004_case(1|2) : Fails with excess deceleration condition when EGO detects the bicycle (jay biker).
    • UC-VRU-002-0005 : EGO acceleration seems insufficient when it tries to overtake bicycles.

• Some of the failures above may be solved by modifying the size of the EGO vehicle.

• [recap of the discussions until last week]

• After a brief hearing form TIER IV CI/CD team, it seems that the storage is becoming the bottle neck with Autoware Evaluator (for AWF organization use). There are two settings to save the storage space.

  • [Log Expiration] : Currently set to 60 days. We can revise this to 30 days, etc.
  • [Disable Logging on Success] : By disabling this, only the failed logs will be stored.

• In order to change the above settings of weekly simulations, you need to go to [Catalog] -> (ex) public road bus -> INTEGRATION tab -> select catalog -> ACTIONS -> Edit. You need the Github access token of the integrated repository.

• We can also exclude the stably succeeding scenarios from the weekly test list to save the storage space.

• After the discussion today, following settings have been applied.

  • [Log Expiration] : Changed from 60 to 45 days.
  • [Disable Logging on Success] : Not disabled for now. (We may sometimes need to play back the successful cases, too.)

• Following changes will be applied to the Autoware Evaluator settings by next week.

  • "pull over scenario suite" ("UC-F-16_ArvDpt_PullOver_zero#2") will be excluded from the weekly list as it may take some time for the speed limit/AEB issue to be addressed.
  • "test scenarios" suite to be renamed to "Low Speed Vehicles" for clarification.

• Did additional Q&A on Salvi's paper/poster "Online Identification of Operational Design Domains of Automated Driving System Features" which was presented in IV24 including the AD safety models defined in IEEE2846. His other papers are available form his personal page at Fraunhofer.

• Comments have been added to Leo Drive Scenario Catalog v0.7 - DRAFT.

  • Discussion topics include:
    • Autoware can detect animals as small as 15cm, but it requires additional learning/tuning.
    • If the detected objects fall in "unknown" category, it may lead to a sudden braking. In low speed ODDs, it may not be a big issue as the maximum speed is low (<20kph).
    • Parking module is included in current Autoware.universe.

• In addition to the discussion last week, @Igata-ctrl is investigating the cloud storage quota. It is associated to how long you would like to keep the simulation result in terms of rosbag.

• Answers to some of the questions from the meeting last week.

  • Comfortable braking threshold (deceleration and jerk) -> See the updated vehicle performance page.
  • Cloud quota for the weekly simulation -> The monthly total DURATION should not exceed 50 hours. You can check it in the Evaluator screen. -> @Igata-ctrl to double check that it is OK to simply add up the time in the DURATION column in the Evaluator Reports page.

• A TUM researcher Aniket Salvi, M. Sc. introduced his paper/poster "Online Identification of Operational Design Domains of Automated Driving System Features" which was presented in IV24.

Aniket Salvi, M. Sc.
Research Engineer
Engineering of Software-Defined Mobility
Fraunhofer-Institut für Kognitive Systeme IKS

• @Igata-ctrl joined the Reference Design WG meeting on May 29, 2024 and learned about the low speed ODDs of ROBEFF Technology's cargo vehicle and Kingwaytek's shuttle bus. -> Try to take some time to comment on this use case spreadsheet.

• Also discussed the followings:

  • Comfortable braking threshold (deceleration and jerk)
  • Cloud quota for the weekly simulation (TBC)
  • Expect @mitsudome-r to streamline the registration process to the Evaluator
  • Is there an easy way to download multiple scenarios at once? -> Unfortunately, no, as each scenario is associated to certain map and its version.

• Recap of the discussion items with Safety Pool.

  • [Map] We can convert OpenDRIVE Map to Lanelet2, but have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead ) proposed in May TSC that Autoware should accept OpenDRIVE maps in a direct fashion as it solves all these issues, while we cn also solve these by post processing after conversion.
    • Speed limit, lane change flag and LHT/RHT are lost through conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings.
  • [Scenario] Autoware requires the EGO's destination to work, while Safety Pool scenarios do not restrict the Ego vehicle (SUT) nor define destination. We need to consider some “roaming” mode for Autoware to operate without the destination. Also, some autonomous driving scenarios do require EGO destination to represent certain use cases.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios are not too many, or the on-premise simulation set up is used.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible scenarios are quite limited (651 as opposed to one million). Safety Pool is kindly looking into it.

• Found an interesting urban mixed traffic example "iino" in Japan, which uses Autoware. Reaching out to this company if they can attend AWF ODD WG to share their experience.

  • News of its demonstration experiment in a congested commercial avenue in Japan.
  • Developer web page (Gekidan Iino).
  • Additional information from the developer in Japanese. (It uses the infrastructure cameras to understand the flow of the people.)
  • Past article (2018) which states that it uses Autoware.

• Discussion over Urban-Area-Scenarios by Berkay (Leo Drive).

• Continued discussion of the Low Speed Vehicle ODD over the video of National Taiwan University campus.

• Brief discussion on the proposed agenda regarding the Add docs for guiding the developers on finding, diagnosing, and fixing scenarios in Autoware

• In the AWF TSC on May 14, Mohammad Musa (Safety Assurance WG/Deepen.ai) recommended to consider switching to OpenDRIVE as it is the default map format with AD simulation companies including Applied Intuition, Foretellix and Cognata along with OpenSCENARIO scenario format. May need to discuss it internally first, revisiting the discussion when Autoware chose Lanelet2.

• On April 30, @Igata-ctrl and @mitsudome-r joined the meeting with Safety Pool (Deepen.ai + WMG) and discussed that the following issues need to be addressed.

  • [Map] We can now convert OpenDRIVE maps of Safety Pool to Lanelet2 for Autoware using CommRoad Scenario Designer, but the following issues need to be ironed out. One option is to make Autoware accept OpenDRIVE maps instead of Lanelet2.
    • Following information is lost through the conversion.
      • Speed limit, lane change tag, LHT/RHT, etc.
    • The converted Lanelet2 map contains too many unnecessary points.
  • [File format] Safety Pool uses .xosc for OpenSCENARIO files, while Autoware Scenario Simulator v2 on Evaluator (cloud integrated CI/CD pipeline) uses .yaml file.
  • [Autoware] Autoware requires the EGO destination to operate, while no EGO destination is specified in Safety Pool scenarios. May need to consider a sort of roaming mode.
  • [Simulation Tool] Safety Pool uses esmini simulator (OpenSCENARIO player). Safety Pool is kindly sending us the link of the recording of a recent Webinar covering this.
  • [Access] After registering to Safety Pool, @Igata-ctrl can access the limited number (651 out of one million) of scenarios in Safety Pool database. Safety Pool is kindly looking into this issue.

• @Igata-ctrl prototyped three dense urban scenarios based on the use cases. May need to clarify a few small issues with TIER IV CI/CD team.

• Daniel Shih, the lead of Reference Design WG joined to discuss the low speed (controlled area) ODD. Once it's finalized, it will be merged to the ODD definitions spread sheet.

• @Igata-ctrl shared a tip how to set arbitrary trajectory for NPCs with @Berkay54 on Discord.

• @Igata-ctrl found an example scenario which uses the FollowTrajectoryAction. It seems there are some caveats around the initial settings of NPCs.

• @Igata-ctrl created a couple of example breadboard use cases for the dense urban ODD.

• @Igata-ctrl is continuing communication with Safety Pool. Currently, Map conversion and compatibility issues are discussed. A meeting has been set on April 30 to discuss how many scenarios will be available for Autoware and how.

• As for the "too many points" issue with the converted Lanelet2 map, I learned from TIER IV FMS & map team that this tool maybe helpful to reduce the number of the unnecessary points.

• Due to a recent bug fix of the scenario simulator v2 regarding the maximum speed setting, it turned out some of the scenarios need to be modified. Berkay who ported the basic TIER IV scenarios to AWF is kindly working on the fix now.

• As for the dense urban traffic ODD which Autoware Labs is going to work on, here's the link to the video of typical dense urban traffic which Fatih showed in the TSC. This would be a good start point for the new use case/scenarios.

• @Igata-ctrl has created a step-by-step memo "Map conversion using CommonRoad Scenario Designer (OpenDRIVE-_Lanelet2)-110324-055327" with a help from Ata (Leo Drive) who is familiar with the CommonRoad Scenario Designer also the author of the documentation. A example of converted Lanelet2 map is uploaded to the folder. The sample map has been uploaded to to the Evaluator (CI/CD pipeline) and an example scenario was created on it. Known map issues which require modification are as follows.

  • Linestrings contain too many points (16,000 points in the area map) -> Vector Map Builder response is very slow
  • Max speed was set to 10 kph -> Modified to 100 kph
  • Line strings between lanes were set as "not-lane changeable" -> Set to "lane changeable"

• Regarding the scenario compatibility, Safety Pool scenarios are in .xosc file and Autoware/Evaluator (CI/CD pipeline) uses .yaml file for more flexible handling of the parameters. Though stand alone Scenario Simulator v2 can accept .xosc file, Evaluator cannot. We may perhaps need another converter or some modification of Evaluator side. @Igata-ctrl is inquiring Safety Pool approximately how many scenarios will be available for Autoware in order to identify what level of automation is desired here.

• [Stephen] Is there any way to obtain the ODD information from the suspended EV project of Apple's (Titan)?

• As a preparation for the Autoware Lab discussion, @Igata-ctrl would like to add "(RoboTaxi in) dense traffic area" to the ODD definitions table. Would like to further edit this table during the meeting.

• Safety Pool presented the outline of Safety Pool.

  • The presentation will be shared in the ODD WG Google Drive.
  • A very large scale (half to one million) scenario database and the eco system is already established.
  • Users can search for appropriate scenarios by tag searches and add scenarios to the database.
  • The scenarios of Safety Pool use OpenDRIVE map.
  • @Igata-ctrl shared the sample scenario and the Lanelet2 map with Safety Pool for analysis.
  • Asked Mohammad and Siddartha to join again next week to make a full demonstration of Safety Pool.

Action Item

• @Igata-ctrl to join Reference Working Group meeting on May 29 to discuss the low speed vehicle ODDs from robeff and Kingwaytek both in Taiwan.

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time