How to use the checkpoint

[JaYzZ]

I find it interesting that the checkpoint in your cvpr24 results do not have in_proj_weight, only in_proj_bias in nn.MultiHeadAttention parameters, which block me from using and evaluating them.

In addition, what is the corresponding config yaml file for them? The readme seems unmatched with the code and ckpt.

[Kimsure]

you need to check model_vpt.py since the attn.in_proj_weight has been chunked as q_proj_weight, k_proj_weight, v_proj_weight to only finetune query and value, where in_proj_bias was not modified. This might help you.