Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unload model stop background #122

Merged
merged 2 commits into from
Nov 13, 2023
Merged

Unload model stop background #122

merged 2 commits into from
Nov 13, 2023

Conversation

tikikun
Copy link
Contributor

@tikikun tikikun commented Nov 13, 2023

credit to #97 for the unloading some changes i made.

  • We will not implement a seperate function for the unload due to the fact that i might need to sync upstream from llama cpp very regularly, so direct implementation inside the background stop function will show me error everytime sync upstream
  • Add stop background process
  • Background process dependent entirely on the model_load value

@tikikun tikikun added the P0: critical Mission critical label Nov 13, 2023
@tikikun tikikun added this to the Nitro v0.2 milestone Nov 13, 2023
@tikikun tikikun self-assigned this Nov 13, 2023
This was linked to issues Nov 13, 2023
feat: Add API to unload model #86
Closed
feat: stop background processing of slots when models are stopped #118
Closed
@tikikun tikikun merged commit f49bd78 into main Nov 13, 2023
8 checks passed
@tikikun tikikun mentioned this pull request Nov 13, 2023
Closed
@hiro-v hiro-v deleted the unload-model-stop-background branch November 13, 2023 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tikikun tikikun

Labels
P0: critical Mission critical
Projects
None yet
Milestone
Nitro v0.2
Development

Successfully merging this pull request may close these issues.

feat: stop background processing of slots when models are stopped feat: Add API to unload model
1 participant
@tikikun