[BUG] GPU memory used is much more in v0.2.7 than v0.2.5 while quantizing models.

[GodHforever]

The model I used is llama3-8B.
The only difference in the quantisation process is the versions, 0.2.7 and 0.2.5.
My gpu memory size is 16g, and I found that version 0.2.7 had problems with the memory being full, but 0.2.5 was able to quantise without any problems.
Has anyone else had similar problems?

[GodHforever]

After debugging with the new version code, I found that some memory was not released in module2inspect.
clear_memory() may be useful for this problem.
The code is awq/quantize/quantizer.py: _compute_best_scale

int_w_output = self._module_forward(x, module2inspect, kwargs)
clear_memory()

Using this API here, memory consumption is reduced by about half
Will any one fix this or I just submit a patch?

[wrsIt]

Hello, I’ve encountered a similar issue. Could you please elaborate further on the solution? I wasn’t able to locate the code you mentioned.（T^T）