-
-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[V1][Bugfix] Fix data item ordering in mixed-modality inference #12259
Conversation
Signed-off-by: Roger Wang <[email protected]>
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: Roger Wang <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM if you don't plan to change https://github.com/vllm-project/vllm/pull/12259/files#r1923428521
Signed-off-by: Roger Wang <[email protected]>
Due to v1 re-arch and design of
get_input_embeddings
being not modality-aware, the order of embeddings inencoder_outputs
needs to match the order of multimodal inputs (MultiModalKwargs
) in the current batch.However, batching
MultiModalKwargs
when there are multiple modalities in the same batch will not preserve their original order, and this PR adds a temporary workaround for this issue.