[Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0

[ywang96]

This is a follow up of #12259. When there are multiple modalities involved, embedding merge are based on different assumptions in V0 and V1, therefore get_input_embeddings does not work with V0 and will produce a bug when there are multiple modalities in a batch.

Fortunately, since embedding generation and LM forward pass are executed together inside the VLM forward pass, it gives us a good way to separate the logics of the two code paths.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[ywang96]

Repro script:

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from vllm.assets.video import VideoAsset

def get_llm(): 
    model_name = "llava-hf/llava-onevision-qwen2-7b-ov-hf"
    llm = LLM(
        model=model_name,
        max_num_seqs=5,
        max_num_batched_tokens=32768,
        enable_prefix_caching=False,
    )
    return llm

def get_multi_modal_input(modality):
    if modality == "image":
        # Input image and question
        image = ImageAsset("cherry_blossom") \
            .pil_image.convert("RGB")
        img_question = "What is the content of this image?"

        return {
            "data": image,
            "question": img_question,
        }

    if modality == "video":
        # Input video and question
        video = VideoAsset(name="sample_demo_1.mp4",
                           num_frames=4).np_ndarrays
        vid_question = "Why is this video funny?"

        return {
            "data": video,
            "question": vid_question,
        }

    msg = f"Modality {modality} is not supported."
    raise ValueError(msg)

if __name__ == "__main__":

    modalities = ["image", "video", "image"]
    inputs = []

    for i in range(len(modalities)):
        modality = modalities[i]
        if modality == "image":
            placeholder = "<image>"
        elif modality == "video":
            placeholder = "<video>"

        mm_input = get_multi_modal_input(modality)
        data = mm_input["data"]
        question = mm_input["question"]
        prompt = f"<|im_start|>user {placeholder}\n{question}<|im_end|> \
        <|im_start|>assistant\n"

        inputs.append(
            {
                "prompt": prompt,
                "multi_modal_data": {
                    modality: data
                },
            }
        )

    llm = get_llm()
    params = SamplingParams(max_tokens=16, temperature=0.0)
    outputs = llm.generate(inputs, sampling_params=params)

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

Running this on main:

The image shows a view of cherry blossoms in the foreground with their pink petals
The video you've shared is not a video, but rather an image. It
The image shows a close-up view of cherry blossoms with a blurred background,

This branch:

The image shows a view of cherry blossoms in the foreground with their pink petals
The humor in the video comes from the child's exaggerated actions and the playful scenario
The image shows a view of cherry blossoms in the foreground with their pink petals

[ywang96]

Also FYI @HwwwwwwwH since you're working on MiniCPM-O, this PR might be relevant to you.

[Isotr0py]

Overall LGTM, just a nit, PTAL!