Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Kernel] Flashinfer correctness fix for v0.1.3 #7319

Merged
merged 11 commits into from
Aug 12, 2024
Merged

[Kernel] Flashinfer correctness fix for v0.1.3 #7319

merged 11 commits into from
Aug 12, 2024

Conversation

LiuXiaoxuanPKU
Copy link
Collaborator

@LiuXiaoxuanPKU LiuXiaoxuanPKU commented Aug 8, 2024 •
edited
Loading

Reported by @felixzhu555 , is_profile_run is buggy in flashinfer backend, which will fail flashinfer v0.1.3. This PR fixes this, and update CI flashinfer version to v1.0.3.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 8, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

comaniac
comaniac reviewed Aug 8, 2024
View reviewed changes
.buildkite/test-pipeline.yaml Outdated
@@ -61,7 +61,7 @@ steps:
- tests/basic_correctness
commands:
# This flashinfer installation will fail on AMD ROCm, so it is set as optional.
- pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.2/flashinfer-0.1.2+cu121torch2.4-cp310-cp310-linux_x86_64.whl || true
- pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.3/flashinfer-0.1.3+cu124torch2.4-cp310-cp310-linux_x86_64.whl || true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just update the Dockerfile and remove all of these.

@LiuXiaoxuanPKU
Copy link
Collaborator Author

/ready

@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 8, 2024
comaniac
comaniac approved these changes Aug 9, 2024
View reviewed changes
Copy link
Collaborator

@comaniac comaniac left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a nit

vllm/attention/backends/flashinfer.py Outdated
self.paged_kv_indptr = torch.zeros(batch_size + 1,
device=self.device)
else:
# Only use flashinfer in the non-profile run
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is a bit confusing as we are in the FlashInfer backend. Can we say more about why we skip the following logic in profile run and what's the outcome?

@LiuXiaoxuanPKU LiuXiaoxuanPKU enabled auto-merge (squash) August 9, 2024 02:43
@comaniac
Copy link
Collaborator

comaniac commented Aug 9, 2024

CI failure seems like a real bug

[2024-08-09T04:52:05Z]   File "/usr/local/lib/python3.10/dist-packages/flashinfer/prefill.py", line 791, in begin_forward
--
  | [2024-08-09T04:52:05Z]     self._wrapper.begin_forward(
  | [2024-08-09T04:52:05Z] RuntimeError: CHECK_EQ(paged_kv_indptr.size(0), batch_size + 1) failed. 1 vs 257

@exceedzhang
Copy link

@LiuXiaoxuanPKU It's bugs
image

@LiuXiaoxuanPKU LiuXiaoxuanPKU merged commit ec2affa into vllm-project:main Aug 12, 2024
68 checks passed
sfc-gh-mkeralapura pushed a commit to sfc-gh-mkeralapura/vllm that referenced this pull request Aug 12, 2024
@LiuXiaoxuanPKU @sfc-gh-mkeralapura
[Kernel] Flashinfer correctness fix for v0.1.3 (vllm-project#7319)
74494eb
kylesayrs pushed a commit to neuralmagic/vllm that referenced this pull request Aug 17, 2024
@LiuXiaoxuanPKU @kylesayrs
[Kernel] Flashinfer correctness fix for v0.1.3 (vllm-project#7319)
9a16370
fialhocoelho pushed a commit to opendatahub-io/vllm that referenced this pull request Aug 22, 2024
@LiuXiaoxuanPKU @fialhocoelho
[Kernel] Flashinfer correctness fix for v0.1.3 (vllm-project#7319)
cf541f7
@LiuXiaoxuanPKU LiuXiaoxuanPKU deleted the flashinfer-correct branch September 17, 2024 04:29
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
@LiuXiaoxuanPKU @Alvant
[Kernel] Flashinfer correctness fix for v0.1.3 (vllm-project#7319)
163a730 
Signed-off-by: Alvant <[email protected]>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@LiuXiaoxuanPKU
[Kernel] Flashinfer correctness fix for v0.1.3 (vllm-project#7319)
d5a9805
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@comaniac comaniac comaniac approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@LiuXiaoxuanPKU @comaniac @exceedzhang