Forward_Hooks

[jansoe]

Hey Qubvel and Community,

first of all some compliments to this nice library.

I was wondering if anyone of you has experience in using forward_hooks in pytorch to extract the features from the encoder instead of overriding the forward method of a model? That is, one would specify the layers for skip-connections somehow by such a FeatureExtractor: https://gist.github.com/fkodom/27ed045c9051a39102e8bcf4ce31df76#file-feature_extractor_hook-py
Do you see some obvious disadvantages?
I'm asking because there is a model with some quite complicated flow where I'm trying to rewrite/overwrite the forward method.

Thanx for any comments,
Jan

[qubvel]

Hi, there are some disadvantages of your example from my point of view:

1. you have to store features in model (I think you should not do it, because it is not layer and if you save state_dict, features would be saved too)
2. rewriting forward method gives an ability to support encoder "depth" (not use some of the satges)

[jansoe]

regarding 1: without understanding what makes it into the state_dict, the _features are apparently not included there
regarding 2: indeed, but this would be something I'm willing to sacrifice

But do you know if there are some more general problems, i.e. gradient flow or performance isssues?

[qubvel]

You are right, features will not be included in state dict, cause they are not nn.Module.

I think this could work, but still doesn't sure it is a good idea to store intermediate tensors in state of the model. At least they need additional memory.

If you try it, please, let me know about result)

[jansoe]

I will try the next days and report here