Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable StagingEngine in test_engine_paged_cache_model.py #50

Merged
merged 3 commits into from
Nov 6, 2023

Conversation

masahi
Copy link
Member

@masahi masahi commented Nov 6, 2023

Expected output

$ python serve/tests/test_engine_paged_cache_model.py  --local-id vicuna-v1-7b-q0f16 --max-num-batched-tokens 2560 --max-input-len 256   --use-staging-engine
Prompt = 'Hello, my name is', generated text = 'Dr. David C. Preston, and I am a chiropractor serving the residents of the'
Prompt = 'The president of the United States is', generated text = 'a man who has been accused of sexual assault by more than a dozen women, and who has b'
Prompt = 'The capital of France is', generated text = 'Paris.
The currency of France is the Euro.
The official language of France is French.'
Prompt = 'The future of AI is', generated text = 'bright, and it’s up to us to ensure that it benefits everyone. As we continue to'

Currently using SyncEngine (without --use-staging-engine) is broken:

Prompt = 'Hello, my name is', generated text = ''
Prompt = 'The president of the United States is', generated text = ''
Prompt = 'The capital of France is', generated text = ''
Prompt = 'The future of AI is', generated text = 'DraParisbright. man., David who
 and C hasThe it. been currency’ Pr accused ofseston of France up, sexual is to and assault the us I by Euro to am more. ensure a than
 that chi aThe itrop dozen official benefitsractor women language everyone serving, of. the and France As residents who is we of has French continue the b. to'

@masahi masahi merged commit 84297df into octoml:batch-serving Nov 6, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant