Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-1031] SBT correct the LICENSE and NOTICE for shaded client jars #1967

Closed
wants to merge 11 commits into from
Closed

[CELEBORN-1031] SBT correct the LICENSE and NOTICE for shaded client jars #1967

Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
19270c6
license
cxzl25 Oct 10, 2023
098056a
contains
cxzl25 Oct 18, 2023
40bd06e
custom strategy
cxzl25 Nov 1, 2023
bd7f4d2
use sLog
cxzl25 Nov 1, 2023
2f0ec39
remove path
cxzl25 Nov 1, 2023
e7fecb4
META-INF/license/
cxzl25 Nov 3, 2023
acf13b4
preferProject
cxzl25 Nov 3, 2023
9ae48b9
discard root path license file
cxzl25 Nov 6, 2023
4db8623
Update project/CelebornBuild.scala
cfmcgrady Nov 6, 2023
cd2db07
Update project/CelebornBuild.scala
cfmcgrady Nov 6, 2023
1ad6634
Update project/CelebornBuild.scala
cfmcgrady Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions project/CelebornBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -690,6 +690,8 @@ trait SparkClientProjects {

(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
case PathList(ps@_*) if ps.last == "LICENSE" || ps.last == "NOTICE" => MergeStrategy.preferProject
case PathList(ps@_*) if Assembly.isLicenseFile(ps.last) => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case m if m.toLowerCase(Locale.ROOT).startsWith("meta-inf/native-image") => MergeStrategy.discard
Expand Down Expand Up @@ -881,6 +883,8 @@ trait FlinkClientProjects {

(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
case PathList(ps@_*) if ps.last == "LICENSE" || ps.last == "NOTICE" => MergeStrategy.preferProject
case PathList(ps@_*) if Assembly.isLicenseFile(ps.last) => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case m if m.toLowerCase(Locale.ROOT).startsWith("meta-inf/native-image") => MergeStrategy.discard
Expand Down Expand Up @@ -958,11 +962,10 @@ object MRClientProjects {

(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") => MergeStrategy.discard
// For netty-3.x.y.Final.jar
case m if m.startsWith("META-INF/license/") => MergeStrategy.discard
case m if m == "META-INF/LICENSE.txt" => MergeStrategy.discard
case m if m == "META-INF/NOTICE.txt" => MergeStrategy.discard
case m if m == "LICENSE.txt" => MergeStrategy.discard
case m if m == "NOTICE.txt" => MergeStrategy.discard
case PathList(ps@_*) if ps.last == "LICENSE" || ps.last == "NOTICE" => MergeStrategy.preferProject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
case PathList(ps@_*) if ps.last == "LICENSE" || ps.last == "NOTICE" => MergeStrategy.preferProject
case "LICENSE" | "NOTICE" => MergeStrategy.discard
case PathList(ps@_*) if ps.last == "LICENSE" || ps.last == "NOTICE" => MergeStrategy.preferProject

we should discard the NOTICE/LICENSE files from the scala-library for mr-shade client.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

$ ./build/sbt -Pmr celeborn-client-mr-shaded/assembly
$ jar tf client-mr/mr-shaded/target/scala-2.12/celeborn-client-mr-shaded_2.12-0.4.0-SNAPSHOT.jar | grep -i 'license\|notice'
LICENSE
META-INF/LICENSE
META-INF/NOTICE
META-INF/licenses/
META-INF/licenses/LICENSE-protobuf.txt
NOTICE

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding this problem.
I have now changed to use preferProject only for META-INF/LICENSE and META-INF/NOTICE.

case PathList(ps@_*) if Assembly.isLicenseFile(ps.last) => MergeStrategy.discard
// Drop all proto files that are not needed as artifacts of the build.
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case m if m.toLowerCase(Locale.ROOT).startsWith("meta-inf/native-image") => MergeStrategy.discard
Expand Down
Loading