Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mpes process files from memory #532

Merged
merged 3 commits into from
Dec 19, 2024
Merged

Conversation

rettigl
Copy link
Member

@rettigl rettigl commented Dec 17, 2024

Processing directly from the files for mpes is rather slow if reading from a filesystem with poor seeking performance (e.g. hdd, network drives). To overcome this, we load all files completely into memory for each binning step, and then access them from there.

@coveralls
Copy link
Collaborator

Pull Request Test Coverage Report for Build 12371852256

Details

  • 17 of 19 (89.47%) changed or added relevant lines in 1 file are covered.
  • 3 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.03%) to 91.759%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sed/loader/mpes/loader.py 17 19 89.47%
Files with Coverage Reduction New Missed Lines %
sed/binning/numba_bin.py 3 87.74%
Totals Coverage Status
Change from base Build 12349863088: -0.03%
Covered Lines: 6525
Relevant Lines: 7111

💛 - Coveralls

@rettigl rettigl requested a review from zain-sohail December 17, 2024 11:21
@rettigl
Copy link
Member Author

rettigl commented Dec 17, 2024

@zain-sohail, this is mpes-specific, maybe you can give it a brief look and commend/approve?

@zain-sohail
Copy link
Member

would this still work with smaller memory setups?

@rettigl
Copy link
Member Author

rettigl commented Dec 17, 2024

would this still work with smaller memory setups?

We are talking about additional ~100 mbyte memory per compute core, compared to several GByte for the actual binning volume, so typically only an increase of 10% or less in memory consumption. You can always compensate with using less cores if memory is an issue.

Copy link
Member

@zain-sohail zain-sohail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested. LGTM!

@rettigl rettigl merged commit 644cfa1 into main Dec 19, 2024
6 checks passed
@rettigl rettigl deleted the mpes_process_files_from_memory branch December 19, 2024 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants