Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CELEBORN-1156][BUILD] SBT publish support #2129

Closed
wants to merge 9 commits into from

Conversation

cfmcgrady
Copy link
Contributor

@cfmcgrady cfmcgrady commented Dec 4, 2023

What changes were proposed in this pull request?

As title

Why are the changes needed?

As title

Does this PR introduce any user-facing change?

Yes, the user can publish shade clients via SBT

How was this patch tested?

docker run -d -p 8081:8081 sonatype/nexus3
export SONATYPE_SNAPSHOTS_URL=http://192.168.3.46:8081/repository/maven-snapshots/
export SONATYPE_RELEASES_URL=http://192.168.3.46:8081/repository/maven-releases/
export ASF_USERNAME=admin
export ASF_PASSWORD=123456
  • Publish the shade client for Spark 3.5:
./build/sbt -Pspark-3.4 celeborn-client-spark-3-shaded/publish
截屏2023-12-08 下午10 22 07
  • Publish the shade client for Flink 1.18:
$ ./build/sbt -Pflink-1.18 celeborn-client-flink-1_18-shaded/publish
截屏2023-12-08 下午10 25 28
  • Publish the shade client for MapReduce:
$ ./build/sbt -Pmr celeborn-client-mr-shaded/publish
截屏2023-12-08 下午10 25 47

Copy link

codecov bot commented Dec 7, 2023

Codecov Report

Attention: 108 lines in your changes are missing coverage. Please review.

Comparison is base (2504b50) 46.56% compared to head (c4fc46b) 46.03%.
Report is 3 commits behind head on main.

Files Patch % Lines
...leborn/common/network/sasl/CelebornSaslClient.java 0.00% 44 Missing ⚠️
...eborn/common/network/sasl/SaslClientBootstrap.java 0.00% 33 Missing ⚠️
...apache/celeborn/common/network/sasl/SaslUtils.java 0.00% 10 Missing ⚠️
.../celeborn/common/network/sasl/SaslCredentials.java 0.00% 6 Missing ⚠️
...born/common/network/sasl/SaslTimeoutException.java 0.00% 6 Missing ⚠️
...eleborn/common/network/client/TransportClient.java 0.00% 5 Missing ⚠️
...cala/org/apache/celeborn/common/CelebornConf.scala 77.78% 2 Missing ⚠️
...born/common/network/protocol/TransportMessage.java 0.00% 1 Missing ⚠️
...he/celeborn/common/network/util/TransportConf.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2129      +/-   ##
==========================================
- Coverage   46.56%   46.03%   -0.52%     
==========================================
  Files         175      180       +5     
  Lines       11377    11492     +115     
  Branches     1019     1027       +8     
==========================================
- Hits         5296     5289       -7     
- Misses       5732     5852     +120     
- Partials      349      351       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cfmcgrady cfmcgrady changed the title [CELEBORN-XXXX][WIP] SBT publish support [CELEBORN-1156][BUILD] SBT publish support Dec 8, 2023
Comment on lines 291 to 295
| SONATYPE_HOST | Sonatype repository host address, default is "oss.sonatype.org" |
| SONATYPE_USERNAME | Sonatype repository username |
| SONATYPE_PASSWORD | Sonatype repository password |
| SONATYPE_SNAPSHOTS_URL | Sonatype repository URL for snapshot version releases, default is "https://oss.sonatype.org/content/repositories/snapshots" |
| SONATYPE_RELEASES_URL | Sonatype repository URL for official release versions, default is "https://oss.sonatype.org/service/local/staging/deploy/maven2" |
Copy link
Member

@pan3793 pan3793 Dec 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is SONATYPE_HOST redundant? I suppose SONATYPE_RELEASES_URL and SONATYPE_SNAPSHOTS_URL are sufficient.

For Apache release, the default URL should be https://repository.apache.org/service/local/staging/deploy/maven2 and https://repository.apache.org/content/repositories/snapshots

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can eliminate SONATYPE_USERNAME, SONATYPE_PASSWORD and simply pick up ASF_USERNAME, ASF_PASSWORD, as they were used in source/binary release process

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For Apache release, the default URL should be https://repository.apache.org/service/local/staging/deploy/maven2 and https://repository.apache.org/content/repositories/snapshots

do you mean the default values(SONATYPE_SNAPSHOTS_URL/SONATYPE_RELEASES_URL) should be Apache URLs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we did such thing in another apache projects

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can eliminate SONATYPE_USERNAME, SONATYPE_PASSWORD and simply pick up ASF_USERNAME, ASF_PASSWORD, as they were used in source/binary release process

done

project/build.properties Outdated Show resolved Hide resolved
For example:
```shell
export SONATYPE_SNAPSHOTS_URL=http://192.168.3.46:8081/repository/maven-snapshots/
export SONATYPE_RELEASES_URL=http://192.168.3.46:8081/repository/maven-releases/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the tailing slash matter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've confirmed in a local environment that the presence or absence of a trailing slash doesn't affect the outcome.

@cfmcgrady
Copy link
Contributor Author

thanks, merging to main(v0.4.0).

@cfmcgrady cfmcgrady closed this in 41df4eb Dec 15, 2023
@cfmcgrady cfmcgrady deleted the sbt-publish branch December 15, 2023 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants