Skip to content

Commit

Permalink
[CELEBORN-1682] Add java tools.jar into classpath for JVM quake
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Add java tools.jar into classpath for JVM quake.

### Why are the changes needed?
Meet below issue with `celeborn.worker.jvmQuake.enabled=true`, see #2061
```
24/11/03 15:51:08,453 ERROR [main] Worker: Initialize worker failed.
java.lang.NoClassDefFoundError: sun/jvmstat/monitor/HostIdentifier
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm$lzycompute(JVMQuake.scala:180)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm(JVMQuake.scala:179)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor$lzycompute(JVMQuake.scala:185)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor(JVMQuake.scala:184)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.org$apache$celeborn$service$deploy$worker$monitor$JVMQuake$$getLastExitTime(JVMQuake.scala:192)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake.start(JVMQuake.scala:66)
    at org.apache.celeborn.service.deploy.worker.Worker.<init>(Worker.scala:360)
    at org.apache.celeborn.service.deploy.worker.Worker$.main(Worker.scala:1041)
    at org.apache.celeborn.service.deploy.worker.Worker.main(Worker.scala)
Caused by: java.lang.ClassNotFoundException: sun.jvmstat.monitor.HostIdentifier
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 9 more
```

Related code:
https://github.com/apache/celeborn/blob/c12e8881ab5b03e39a8b53d038a283351bf5906c/project/JDKTools.scala#L58-L75

Similar issue: vladimirvivien/jmx-cli#4

After copy the `tools.jar` into worker-jars, the issue got resolved.

It is better that to involve the `tools.jar` automatically without copy.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
<img width="1202" alt="image" src="https://github.com/user-attachments/assets/af8f6c0d-9123-4a73-93b5-69836c5f826d">

Closes #2874 from turboFei/jdk_tools.

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: SteNicholas <[email protected]>
  • Loading branch information
turboFei authored and SteNicholas committed Nov 4, 2024
1 parent c12e888 commit d4044c5
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
2 changes: 1 addition & 1 deletion bin/celeborn-class
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ if [ ! -d "$CELEBORN_JARS_DIR" ]; then
echo "You need to build CELEBORN with the target \"package\" before running this program." 1>&2
exit 1
else
CELEBORN_CLASSPATH="$CELEBORN_CONF_DIR:$HADOOP_CONF_DIR:$CELEBORN_JARS_DIR/*"
CELEBORN_CLASSPATH="$CELEBORN_CONF_DIR:$HADOOP_CONF_DIR:$CELEBORN_JARS_DIR/*:$JAVA_TOOLS_JAR"
fi

# Turn off posix mode since it does not allow process substitution
Expand Down
12 changes: 12 additions & 0 deletions sbin/load-celeborn-env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,24 @@ if [ -n "${JAVA_HOME}" ]; then
else
if [ "$(command -v java)" ]; then
export JAVA="java"
JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | awk '{print $3}')
else
echo "JAVA_HOME is not set" >&2
exit 1
fi
fi

# Find the java tools.jar
if [ -f "${JAVA_HOME}/lib/tools.jar" ]; then
export JAVA_TOOLS_JAR="${JAVA_HOME}/lib/tools.jar"
else
if [ -f "${JAVA_HOME}/../lib/tools.jar" ]; then
export JAVA_TOOLS_JAR="${JAVA_HOME}/../lib/tools.jar"
else
echo "WARNING: cannot locate tools.jar. Expected to find it in either ${JAVA_HOME}/lib/tools.jar or ${JAVA_HOME}/../lib/tools.jar"
fi
fi

# Get log directory
if [ "$CELEBORN_LOG_DIR" = "" ]; then
export CELEBORN_LOG_DIR="${CELEBORN_HOME}/logs"
Expand Down

0 comments on commit d4044c5

Please sign in to comment.