Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Back off improvement #60

Merged
merged 104 commits into from
Jan 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
ea8db34
feat: implement basic fetch queue type
jost-s Nov 20, 2024
43c5f25
refactor: move trait to api & clean up
jost-s Nov 27, 2024
69c7ba5
feat: add fetch task skeleton
jost-s Nov 27, 2024
06d207e
feat: add batching to fetch task
jost-s Nov 28, 2024
169fefb
docs: udpate fetch queue description
jost-s Nov 28, 2024
1042220
style: fix lints
jost-s Nov 28, 2024
d06a6c1
refactor: factorize fetch queue
jost-s Nov 30, 2024
b85f33e
refactor: add config and builder
jost-s Dec 1, 2024
140537b
feat: add request loop logic & test
jost-s Dec 2, 2024
db9de86
refactor: index by op id instead of agent id
jost-s Dec 3, 2024
1f47a0f
feat: cancel pause when ops are added
jost-s Dec 3, 2024
72e8298
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 3, 2024
9d268bd
refactor: fetch ops individually with a simple queue
jost-s Dec 4, 2024
97ca843
refactor: rename FetchQueue to Fetch
jost-s Dec 4, 2024
f8161cd
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 4, 2024
ff42a22
refactor: rename package to fetch
jost-s Dec 4, 2024
d37fed6
docs: update fetch specs
jost-s Dec 4, 2024
ffb0a9b
test: add test for parallel request count
jost-s Dec 4, 2024
014348f
test: add multi op multi agent test
jost-s Dec 5, 2024
765715d
fix lints
jost-s Dec 5, 2024
4c042f1
refactor: move fetch into core crate
jost-s Dec 5, 2024
5749858
refactor(fetch): reinstate loop waking mechanism
jost-s Dec 5, 2024
4ce895b
fix(fetch): pause bf re-adding op to give other requests a chance to …
jost-s Dec 5, 2024
7518203
test: fix flaky tests
jost-s Dec 5, 2024
a3a005f
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 5, 2024
129f3b9
style: fix lints
jost-s Dec 5, 2024
9109961
style: fix docs
jost-s Dec 5, 2024
865e7b4
test: increase timeout for slow windows
jost-s Dec 5, 2024
9f57aaf
test: increase timeout - that clearly is too long even for slow windows
jost-s Dec 5, 2024
1c0bd1a
feat: add cool down list for unresponsive agents
jost-s Dec 6, 2024
596a161
fix: shorten check intervals in tests
jost-s Dec 6, 2024
6e2db73
test: fix race cond & add unhappy fetch test
jost-s Dec 6, 2024
51683c6
style: lint fix
jost-s Dec 6, 2024
e3e332d
feat(fetch): add debug to fetch trait
jost-s Dec 7, 2024
b0bdb5c
refactor(fetch): use channel as fetch queue
jost-s Dec 7, 2024
568acca
refactor(fetch): replace index map by simple hash set
jost-s Dec 7, 2024
0cbfb06
feat: re-insert fetch requests into the channel & add queue test
jost-s Dec 8, 2024
ca63d4e
test: update multi agent test
jost-s Dec 8, 2024
8bb9e3c
refactor: split fetch task out into separate fn
jost-s Dec 8, 2024
77f628b
build: move rand to workspace deps
jost-s Dec 9, 2024
f0dba0d
refactor: maange cool-down list
jost-s Dec 9, 2024
25ac885
refactor: remove config from fetch struct
jost-s Dec 9, 2024
32c54d9
refactor: inner struct
jost-s Dec 9, 2024
22dca83
style: fix lints
jost-s Dec 9, 2024
962f987
refactor: inner struct somemore
jost-s Dec 9, 2024
57d7f2b
refactor: combine state objects in one mutex
jost-s Dec 9, 2024
5b9e87f
Merge branch 'main' into feat/fetch-pool
jost-s Dec 10, 2024
4373ea2
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 10, 2024
23e0801
Update crates/core/src/factories/core_fetch.rs
jost-s Dec 10, 2024
259aac5
refactor(fetch): default to 120 secs as cool down interval
jost-s Dec 10, 2024
a627ef8
refactor: improve op adding efficiency
jost-s Dec 10, 2024
d13f217
refactor: try_send instead of send to fetch queue to prevent dead-lock
jost-s Dec 10, 2024
3ba5768
rename task spawner
jost-s Dec 10, 2024
d4a056a
refactor: convert eprintln to tracing
jost-s Dec 10, 2024
fb12f22
Merge branch 'feat/fetch-pool' of https://github.com/holochain/kitsun…
jost-s Dec 10, 2024
e430dfd
refactor: encapsulate cool down list into its own struct
jost-s Dec 10, 2024
fec7863
refactor: use actual transport interface
jost-s Dec 10, 2024
872fc8c
refactor: increase fetch queue buffer to 16_384
jost-s Dec 11, 2024
5fd4e84
refactor: move task spawning back into constructor
jost-s Dec 11, 2024
dd79851
fix: use actual mod name
jost-s Dec 11, 2024
2a156d5
feat: add drop impl to inner to abort fetch tasks
jost-s Dec 11, 2024
9add269
refactor: replace tokio mutexes by std ones
jost-s Dec 11, 2024
f90981e
refactor: add space id & peer store to factory creator
jost-s Dec 11, 2024
ee7fe7d
style: clippy
jost-s Dec 11, 2024
2ea7ec0
feat: add peer store to fetch factory
jost-s Dec 11, 2024
05acb31
test: add agents to peer store
jost-s Dec 11, 2024
ab36f8b
refactor: move agent builder to test utils
jost-s Dec 11, 2024
6fc127a
refactor: order tests some more
jost-s Dec 11, 2024
e0c1166
refactor: merge Inner into CoreFetch
jost-s Dec 11, 2024
94bbc7c
refactor: release lock on ops after adding elements
jost-s Dec 11, 2024
613f6cc
refactor: remove op/agent from ops to fetch set when re-inserting int…
jost-s Dec 11, 2024
9ac3743
refactor: simplify instant comparison
jost-s Dec 11, 2024
287be35
refactor: move peer url getter to separate fn
jost-s Dec 11, 2024
85b7249
doc: update text
jost-s Dec 11, 2024
341f109
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 12, 2024
c2c224d
refactor: replace try_send by send when adding ops to fetch queue
jost-s Dec 12, 2024
e27beed
feat(fetch): add proto definition for ops to fetch
jost-s Dec 13, 2024
19122f7
test: adapt tests to op message
jost-s Dec 13, 2024
0e9d9e7
refactor: re-insert ops into queue only when peer is responsive
jost-s Dec 13, 2024
ec06ac8
refactor: add helpers for ser/de op ids
jost-s Dec 13, 2024
b395812
refactor: simplify op id from impls
jost-s Dec 14, 2024
5b1e7f9
Merge branch 'main' of https://github.com/holochain/kitsune2 into fea…
jost-s Dec 14, 2024
1d1bd6d
style: append line break to fetch.proto
jost-s Dec 14, 2024
5bd044e
style: add empty line to separate test sections
jost-s Dec 14, 2024
3c3e60c
test: de-flakify fetch queue test
jost-s Dec 14, 2024
43f2111
test: test that ops are removed from set when agent is not in peer store
jost-s Dec 14, 2024
8f0b94d
refactor: rename AgentBuild to AgentBuilder
jost-s Dec 16, 2024
c024fad
refactor: simplify enc/dec test
jost-s Dec 16, 2024
e41dbb1
test: fix typos
jost-s Dec 16, 2024
b20c5c7
refactor: put unresponsive agents on exponential back off
jost-s Dec 17, 2024
3f0f7c8
refactor: check for max back off at the end
jost-s Dec 17, 2024
abda1a9
refactor: set vec capacity for fetch tasks
jost-s Dec 18, 2024
d7bc3b5
refactor: simplify request retention
jost-s Dec 18, 2024
bba7e78
Merge branch 'main' of https://github.com/holochain/kitsune2 into ref…
jost-s Dec 19, 2024
d70a6da
refactor: rename fn is_agent_backing_off to is_agent_on_back_off
jost-s Dec 19, 2024
ff61632
fix: remove request when max back off interval expired
jost-s Dec 19, 2024
1236525
refactor: change back_off_interval to 20 s
jost-s Dec 19, 2024
5499de9
test: increase timeout due to flakiness
jost-s Dec 19, 2024
59880ef
wip
jost-s Dec 19, 2024
ac416d3
Merge branch 'main' of https://github.com/holochain/kitsune2 into ref…
jost-s Dec 20, 2024
c184187
fix: remove all requests from set when agent has been backed off max
jost-s Dec 19, 2024
7cae7e2
fix: the lints
jost-s Dec 20, 2024
47760c3
refactor: move back off into separate module
jost-s Dec 20, 2024
f49e64a
Merge branch 'main' into refactor/improve-fetch-back-off-backon
jost-s Dec 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 95 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ kitsune2_api = { version = "0.0.1-alpha", path = "crates/api" }
async-channel = "2.3.1"
# this is used by bootstrap_srv as the http server implementation.
axum = { version = "0.7.9", default-features = false }
# used by fetch module to back off requests to unresponsive agents
backon = "1.3"
# debugging is far easier when you can see short byte arrays
# as base64 instead of decimal u8s.
base64 = "0.22.1"
Expand Down
1 change: 1 addition & 0 deletions crates/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ categories = ["network-programming"]
edition = "2021"

[dependencies]
backon = { workspace = true }
bytes = { workspace = true }
ed25519-dalek = { workspace = true }
futures = { workspace = true }
Expand Down
Loading
Loading