[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM

[HwwwwwwwH]

This PR aims to adapt and support all the features of MiniCPM-V and MiniCPM-o. It is designed to be compatible with various modalities (image, video, audio), different model versions (2.0, 2.5, 2.6, o), and diverse input types (raw, embeddings), while maintaining support for LORA, which might require significant effort.

Below is the roadmap for this PR:

• Refactor the input processor code of MiniCPM-V for MultiModalInputsV2 of vLLM.
  • Support for image and video inputs.
  • Support for image embeddings inputs.
  • Support for video embeddings inputs.
  • Previous supports for MiniCPM-o.
  • Verify LORA support.
• Adapt new features of MiniCPM-o.
  • Support for audio and audio embeddings inputs.
  • Support for image and audio interleave inputs.
  • Support for audio outputs (using hidden states).
  • Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

This PR is still in development. Once I complete the support for audio, I will request to merge. I'll get this work done ASAP.

FIX #12162

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[ywang96]

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

[HwwwwwwwH]

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

Thank you for suggestion! I'll start these two RFCs tomorrow.

[HwwwwwwwH]

@DarkLight1337 I think I might need some help for verifying LoRA support. Should I do any changes for it?

[DarkLight1337]

@jeejeelee can help with this. Please keep in mind though that currently LoRA is only supported for the language part of multi-modal models.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @HwwwwwwwH.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork