ValueError: weight is on the meta device, we need a value to put in on 0.

[Draculair]

I tried finetune llava-med using qlora, and I used tokenizer, model, processor, context_len = load_pretrained_model( model_path, model_base, model_name, load_8bit=False, load_4bit=True, device="cuda") to load the model, but the following error was reported:

Traceback (most recent call last):
  File "/home/wqruan/LLaVA-Med/finetune.py", line 15, in <module>
    tokenizer, model, processor, context_len = load_pretrained_model(
  File "/home/wqruan/LLaVA-Med/llava/model/builder.py", line 31, in load_pretrained_model
    model = LlavaMistralForCausalLM.from_pretrained(
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3773, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/accelerate/big_modeling.py", line 371, in dispatch_model
    attach_align_device_hook_on_blocks(
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/accelerate/hooks.py", line 506, in attach_align_device_hook_on_blocks
    add_hook_to_module(module, hook)
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/accelerate/hooks.py", line 155, in add_hook_to_module
    module = hook.init_hook(module)
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/accelerate/hooks.py", line 253, in init_hook
    set_module_tensor_to_device(module, name, self.execution_device)
  File "/home/wqruan/miniconda3/envs/llava-med/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 267, in set_module_tensor_to_device
    raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
ValueError: weight is on the meta device, we need a `value` to put in on 0.

This only appears when load_4bit=True, can anyone helps?

[heavy0ung]

https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b/discussions/1

this may help your problem.