[QUESTION] How to perform inference on trained model?

[sleepingcat4]

I have followed-thru a tutorial posted by Roboflow. While it was clear and helpful for learning PaliGemma, I had been struggling to figure out, how inference on the trained model (.npz) file can be done. Can someone provide some pointers and help?

https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-finetune-paligemma-on-detection-dataset.ipynb?ref=blog.roboflow.com#scrollTo=TGDFTYVnY4zn

[Naziyashaik09]

@sleepingcat4 Am also looking for the same have you got anything on this.
Please let me know if you got anything.

[sleepingcat4]

@Naziyashaik09 I've actually solved this problem.

[Naziyashaik09]

@sleepingcat4 Please can you share me the code or the notebook link.

[akolesnikoff]

The main colab has examples of how to tune the model and make predictions (aka inference): https://colab.research.google.com/github/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/finetune_paligemma.ipynb.

Is there something specific that is unclear?

[gadhane]

@Naziyashaik09 I've actually solved this problem.

Can you please share on how you address the issue? I tried the one provided on the original code, but it is not working. We would appreciate if you could give some ideas or if possible share the code. Thank you.

[sleepingcat4]

It's not difficult. Change the model and tokenizer paths to your trained model and tokenizer path in the beginning and done. Although if you want to just keep the inference code that's slightly tricky. Since, PaliGemma uses Jax sharding which needs to be kept unless you can load the entire model on your GPU or Colab. @gadhane @Naziyashaik09

[sleepingcat4]

@akolesnikoff actually no. Unfortunately, off-the boat it is not apparent we just need to change the model and tokenizer paths LOL!

[EtremelyKeep]

The main colab has examples of how to tune the model and make predictions (aka inference): https://colab.research.google.com/github/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/finetune_paligemma.ipynb.

Is there something specific that is unclear?

the dataset of fine-tuning paligemma is missing, and the longcap100 dataset is now inaccessible from kaggle.