Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

<1s(?) build time #1602

Open
wants to merge 61 commits into
base: master
Choose a base branch
from
Open

Conversation

MikeBusuttil
Copy link

@MikeBusuttil MikeBusuttil commented Dec 29, 2024

  • replaced standard regex with RE2
  • removed code optimization (for development with FAST=1)
  • removed unused scons tools
  • refactored redundant SG_ regex

This should resolve issues #1180 and #1528 by reporting <1s build time when running

export FAST=1 && scons -c && time scons -j40

Results:

image

Thanks to:

  • @nelsonjchen for showing me Tunshell which was needed to debug the RE2 brew install (since I don't have a Mac)
  • @bdbaddog for the scons speed-up tip

@github-actions github-actions bot added the can related to CAN tools, aka opendbc/can/ label Dec 29, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing to opendbc! In order for us to review your PR as quickly as possible, check the following:

  • Convert your PR to a draft unless it's ready to review
  • Read the contributing docs
  • Before marking as "ready for review", ensure:
    • the goal is clearly stated in the description
    • all the tests are passing
    • include a route or your device' dongle ID if relevant

Comment on lines +20 to +21
RE2 val_regexp(R"(VAL_ (\w+) (\w+) (.*))");
RE2 val_split_regexp(R"((([0-9]) \"(.+?)\"))");
Copy link
Author

@MikeBusuttil MikeBusuttil Dec 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes:

  • no tests match \s*[-+]? so this was removed
  • RE2 doesn't support a variable number of matching groups (the reason for the small refactor here and below)

@MikeBusuttil MikeBusuttil mentioned this pull request Dec 29, 2024
@MikeBusuttil MikeBusuttil marked this pull request as ready for review December 31, 2024 04:50
@adeebshihadeh
Copy link
Contributor

adeebshihadeh commented Jan 1, 2025

This does improve the speed, but doesn't quite get to <1s. Ran a bunch of times on my machine, and it's consistently ~2.2s:

batman@workstation-adeeb:~/src/opendbc$ export FAST=1 && scons -c && time scons -j40
Cannot read termcap database;
using dumb terminal settings.
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed opendbc/can/common.os
Removed opendbc/can/parser.os
Removed opendbc/can/packer.os
Removed opendbc/can/packer_pyx.cpp
Removed opendbc/can/parser_pyx.cpp
Removed opendbc/dbc/honda_crv_executive_2016_can_generated.dbc
Removed opendbc/dbc/honda_civic_touring_2016_can_generated.dbc
Removed opendbc/dbc/honda_crv_touring_2016_can_generated.dbc
Removed opendbc/dbc/acura_ilx_2016_can_generated.dbc
Removed opendbc/dbc/subaru_global_2017_generated.dbc
Removed opendbc/dbc/tesla_radar_continental_generated.dbc
Removed opendbc/dbc/toyota_rav4_prime_generated.dbc
Removed opendbc/dbc/nissan_leaf_2018_generated.dbc
Removed opendbc/dbc/honda_accord_2018_can_generated.dbc
Removed opendbc/dbc/honda_fit_hybrid_2018_can_generated.dbc
Removed opendbc/dbc/toyota_nodsu_pt_generated.dbc
Removed opendbc/dbc/gm_global_a_powertrain_generated.dbc
Removed opendbc/dbc/subaru_outback_2019_generated.dbc
Removed opendbc/dbc/honda_crv_ex_2017_body_generated.dbc
Removed opendbc/dbc/honda_odyssey_exl_2018_generated.dbc
Removed opendbc/dbc/honda_clarity_hybrid_2018_can_generated.dbc
Removed opendbc/dbc/honda_civic_ex_2022_can_generated.dbc
Removed opendbc/dbc/tesla_radar_bosch_generated.dbc
Removed opendbc/dbc/acura_rdx_2020_can_generated.dbc
Removed opendbc/dbc/subaru_global_2020_hybrid_generated.dbc
Removed opendbc/dbc/toyota_tnga_k_pt_generated.dbc
Removed opendbc/dbc/honda_crv_ex_2017_can_generated.dbc
Removed opendbc/dbc/chrysler_ram_hd_generated.dbc
Removed opendbc/dbc/honda_insight_ex_2019_can_generated.dbc
Removed opendbc/dbc/acura_rdx_2018_can_generated.dbc
Removed opendbc/dbc/subaru_forester_2017_generated.dbc
Removed opendbc/dbc/honda_civic_hatchback_ex_2017_can_generated.dbc
Removed opendbc/dbc/honda_odyssey_extreme_edition_2018_china_can_generated.dbc
Removed opendbc/dbc/hyundai_kia_mando_corner_radar_generated.dbc
Removed opendbc/dbc/chrysler_ram_dt_generated.dbc
Removed opendbc/dbc/hyundai_kia_mando_front_radar_generated.dbc
Removed opendbc/dbc/honda_pilot_2023_can_generated.dbc
Removed opendbc/dbc/nissan_x_trail_2017_generated.dbc
Removed opendbc/dbc/toyota_new_mc_pt_generated.dbc
Removed opendbc/dbc/honda_fit_ex_2018_can_generated.dbc
Removed opendbc/dbc/chrysler_pacifica_2017_hybrid_generated.dbc
Removed opendbc/dbc/subaru_outback_2015_generated.dbc
scons: done cleaning targets.
Cannot read termcap database;
using dumb terminal settings.
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o opendbc/can/common.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O0 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/common.cc
g++ -o opendbc/can/dbc.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O0 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/dbc.cc
g++ -o opendbc/can/parser.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O0 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/parser.cc
g++ -o opendbc/can/packer.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O0 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/packer.cc
cythonize opendbc/can/packer_pyx.pyx
cythonize opendbc/can/parser_pyx.pyx
python3 opendbc/dbc/generator/generator.py
Compiling /home/batman/src/opendbc/opendbc/can/packer_pyx.pyx because it changed.
[1/1] Cythonizing /home/batman/src/opendbc/opendbc/can/packer_pyx.pyx
Compiling /home/batman/src/opendbc/opendbc/can/parser_pyx.pyx because it changed.
[1/1] Cythonizing /home/batman/src/opendbc/opendbc/can/parser_pyx.pyx
g++ -o opendbc/can/packer_pyx.o -c -std=c++1z -g -fPIC -O0 -Wunused -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -Wno-#warnings -Wno-shadow -Wno-deprecated-declarations -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 -I.venv/lib/python3.11/site-packages/numpy/_core/include opendbc/can/packer_pyx.cpp
g++ -o opendbc/can/parser_pyx.o -c -std=c++1z -g -fPIC -O0 -Wunused -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -Wno-#warnings -Wno-shadow -Wno-deprecated-declarations -I. -I/usr/lib/include -I/opt/homebrew/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 -I.venv/lib/python3.11/site-packages/numpy/_core/include opendbc/can/parser_pyx.cpp
g++ -o opendbc/can/libdbc.so -shared opendbc/can/dbc.os opendbc/can/parser.os opendbc/can/packer.os opendbc/can/common.os -Lopendbc/can -L/opt/homebrew/lib
g++ -o opendbc/can/packer_pyx.so -pthread -shared -Wl,-rpath=/home/batman/src/opendbc/opendbc/can opendbc/can/packer_pyx.o -Lopendbc/can -L/opt/homebrew/lib -Lopendbc/can -ldbc
g++ -o opendbc/can/parser_pyx.so -pthread -shared -Wl,-rpath=/home/batman/src/opendbc/opendbc/can opendbc/can/parser_pyx.o -Lopendbc/can -L/opt/homebrew/lib -Lopendbc/can -ldbc -lre2
scons: done building targets.

real    0m2.189s

Aside from that, this approach adds both a fairly large dependency and goes from 1 -> 2 cases.

  • is there some alternative single header regex library?
  • how do I find out about FAST=1?
  • what do I lose with FAST=1? i.e. why wouldn't I want to always build one way or another?

I'm not sold that we can't do this with minimal to no tradeoffs (such as having to check in a precompiled header when the regex changes).

@MikeBusuttil
Copy link
Author

MikeBusuttil commented Jan 1, 2025

Yo @adeebshihadeh

is there some alternative single header regex library?

  • Yep, I found ctre among several that I considered but originally decided against it because the API is different and run-time performance wasn't as good as RE2. I'll make a branch that uses ctre instead of re2 and ping you.

I'm not sold that we can't do this with minimal to no tradeoffs (such as having to check in a precompiled header when the regex changes).

  • Regex is no longer the bottleneck thanks to RE2. The upper bound for time saved by further optimizing the regex is 0.05s:
    • Compiling dbc.cc with RE2 takes 0.95(1)s
    • Compiling dbc.cc without regex takes 0.9(5)s (using this modified dbc.cc).
  • The bottleneck now is the Cython. It takes 0.53(2)s to cythonize (convert to c++) parser_pyx.pyx and then another 0.97(3)s to compile parser_pyx.cpp. This might just be the price of Cython but I'll dig in and report back.

Note: 0.9(5)s means 0.9s ± 0.05s which means between 0.85 and 0.95 seconds. 0.95(1) means between 0.949...0.951

@adeebshihadeh adeebshihadeh marked this pull request as draft January 2, 2025 23:21
@MikeBusuttil MikeBusuttil changed the title <1s build time ~2s build time Jan 10, 2025
@MikeBusuttil
Copy link
Author

Update @adeebshihadeh :

  • I don't believe there exists a better regex library for our purposes than RE2. Most regex libraries are quite bulky and optimize for run-time performance over build-time performance. The CTRE library was the only single header library I found and then implemented here but do not recommend in our case since compiling dbc.cc using it takes over 3s on my machine. There are some alternatives we could look into:
    • The standard rust regex appears to be fast if we want to add rust as a dependency
    • Building our own regex library to suit our purposes could work
    • Building our own trie-based (not regex-based) single-header library
  • see my report here for what I recommend for next steps

Co-authored-by: William Deegan <[email protected]>
@MikeBusuttil MikeBusuttil changed the title ~2s build time <1s(?) build time Jan 13, 2025
@MikeBusuttil MikeBusuttil marked this pull request as ready for review January 13, 2025 05:36
@MikeBusuttil
Copy link
Author

MikeBusuttil commented Jan 13, 2025

Yo @adeebshihadeh can you run this again and let me know how fast it is on your machine? I trimmed an additional 25% off the runtime with a SConstruct optimization (that might be worth implementing in other repos you guys maintain)

MikeBusuttil and others added 3 commits January 13, 2025 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
can related to CAN tools, aka opendbc/can/
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants