[FEA] Set spark.rapids.sql.allowMultipleJars
to NEVER by default
#9870
Labels
feature request
New feature or request
Is your feature request related to a problem? Please describe.
#9654 Added a new config
spark.rapids.sql.allowMultipleJars
, which is used to detect if there are multiple plugin/cudf/jni jars on the classpath when initializingRapidsDriverPlugin
andRapidsExecutorPlugin
and throw an exception if there are.Currently it's default value is set to SAME_REVISION, which means that jars with the same revision are allowed by this detection. Ideally, we should set it to NEVER by default to avoid the following problems:
Additionally, we should also provide a way to detect submoudle jars in the classpath, like adding a new config mode RECURSIVE to find out the path of submoudle jars.
Describe the solution you'd like
The reason we didn't set the default value to
NEVER
is that it may require some breaking changes in CI, to solve it, we can:spark.rapids.sql.allowMultipleJars=NEVER
in CI jobs and fix failure if any.The text was updated successfully, but these errors were encountered: