Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Set spark.rapids.sql.allowMultipleJars to NEVER by default #9870

Open
thirtiseven opened this issue Nov 28, 2023 · 0 comments
Open

[FEA] Set spark.rapids.sql.allowMultipleJars to NEVER by default #9870

thirtiseven opened this issue Nov 28, 2023 · 0 comments
Assignees
Labels
feature request New feature or request

Comments

@thirtiseven
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
#9654 Added a new config spark.rapids.sql.allowMultipleJars, which is used to detect if there are multiple plugin/cudf/jni jars on the classpath when initializing RapidsDriverPlugin and RapidsExecutorPlugin and throw an exception if there are.

Currently it's default value is set to SAME_REVISION, which means that jars with the same revision are allowed by this detection. Ideally, we should set it to NEVER by default to avoid the following problems:

Additionally, we should also provide a way to detect submoudle jars in the classpath, like adding a new config mode RECURSIVE to find out the path of submoudle jars.

Describe the solution you'd like
The reason we didn't set the default value to NEVER is that it may require some breaking changes in CI, to solve it, we can:

  • Find any multiple jar usage from the log, and try to remove them.
  • Set spark.rapids.sql.allowMultipleJars=NEVER in CI jobs and fix failure if any.
  • Set default value to NEVER in code.
@thirtiseven thirtiseven added the ? - Needs Triage Need team to review and classify label Nov 28, 2023
@mattahrens mattahrens added feature request New feature or request and removed ? - Needs Triage Need team to review and classify labels Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants