Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Op store split metadata #74

Merged
merged 3 commits into from
Jan 16, 2025
Merged

Op store split metadata #74

merged 3 commits into from
Jan 16, 2025

Conversation

ThetaSinner
Copy link
Member

@ThetaSinner ThetaSinner commented Jan 16, 2025

The op store should ingest only op data. Hashes should be computed when data arrives so that we aren't trusting somebody else to do our hashing. The rest of this PR is consequences from trying to achieve that.

Within the op store, we want to be able to store more fields than the op contains. Like stored_at which I'm using on the gossip PR for incremental sync. That means we need to keep the op data model and the model the op store uses internally as separate types.

The big fiddle here was that I couldn't use the hash function that was added with the fetch queue inside the mem op store without destroying the DHT tests. Those need predictable IDs to get the ops into different parts of the DHT to test the location logic. I could spend time trying to find op data that places the ops in the right locations but predictable IDs seemed so much simpler for testing. So I'm just using the first 32 bytes of the op data as the id. If there isn't enough op data, then pad with 0.

@ThetaSinner ThetaSinner marked this pull request as ready for review January 16, 2025 14:44
@ThetaSinner ThetaSinner requested a review from a team January 16, 2025 14:44
Copy link
Contributor

@jost-s jost-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

That hashing works okay, I think. I was thinking of putting the whole MemOpStore under #[cfg(test)), but we need a replacement for it. Were it test code, we could use some actual hashing algorithm.

crates/api/src/fetch.rs Show resolved Hide resolved
@ThetaSinner
Copy link
Member Author

Yeah I'm thinking of it as test code. We could and probably should have a more realistic host implementation for integration tests. Once we have tx5 integrated and the top level crate created, I'm happy to add an op store that behaves in a more realistic way

@ThetaSinner ThetaSinner merged commit 84239d1 into main Jan 16, 2025
5 checks passed
@ThetaSinner ThetaSinner deleted the op-store-split-metadata branch January 16, 2025 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants