Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rsync tls toomanyfiles #1291

Merged

Conversation

tesshuflower
Copy link
Contributor

Describe what this PR does

Fixes rsync-tls issue with argments too long when many files are in the root of the source PVC

Is there anything that requires special attention?

  • Made a copy of all files into /tmp/filelist.txt and then pass that as --files-from arg to rsync
  • Needed to add -r flag to rsync (this flag is normally implied by -a, but gets turned off by default when using --files-from)
  • added e2e tests that create 21k files
    • used long names to reduce the number of files needed to create the issue. Hopefully this doesn't cause perf issues with the actual e2e tests in CI. In my own env the sync does take a lot longer, but the compare pvc checks actually were surprisingly fast.
    • Also added some files on the source with subdirs and files/dirs that start with .

Related issues:

Fixes: #1290

- doing ls -A /data/* will produce error
"Argument list too long" with the data used in this test

Signed-off-by: Tesshu Flower <[email protected]>
Copy link

sonarqubecloud bot commented Jun 4, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@tesshuflower
Copy link
Contributor Author

/cc @JohnStrunk

@openshift-ci openshift-ci bot requested a review from JohnStrunk June 5, 2024 12:30
Copy link
Member

@JohnStrunk JohnStrunk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link
Contributor

openshift-ci bot commented Jun 5, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JohnStrunk, tesshuflower

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [JohnStrunk,tesshuflower]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@JohnStrunk
Copy link
Member

/override DCO

Copy link
Contributor

openshift-ci bot commented Jun 5, 2024

@JohnStrunk: Overrode contexts on behalf of JohnStrunk: DCO

In response to this:

/override DCO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot openshift-merge-bot bot merged commit 3ebf860 into backube:main Jun 5, 2024
30 checks passed
@tesshuflower tesshuflower deleted the rsync-tls-toomanyfiles branch June 5, 2024 14:43
tesshuflower pushed a commit to tesshuflower/volsync that referenced this pull request Jun 5, 2024
Rsync tls toomanyfiles

(cherry picked from commit 3ebf860)
tesshuflower pushed a commit to tesshuflower/volsync that referenced this pull request Jun 5, 2024
Rsync tls toomanyfiles

(cherry picked from commit 3ebf860)
tesshuflower pushed a commit to tesshuflower/volsync that referenced this pull request Jun 5, 2024
Rsync tls toomanyfiles

(cherry picked from commit 3ebf860)
Signed-off-by: Tesshu Flower <[email protected]>
tesshuflower pushed a commit to tesshuflower/volsync that referenced this pull request Jun 5, 2024
Rsync tls toomanyfiles

(cherry picked from commit 3ebf860)
Signed-off-by: Tesshu Flower <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rsync-tls - fails to sync if many files in the root of the source PVC
2 participants