📝 Add Example Notebook for Foundation Models

[adamshephard]

• Add Example Notebook to explain how to use Foundation Models from timm module in TIAToolbox.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[GeorgeBatch]

I just realised that in my code, like in this example notebook, we do not explicitly put the model into the .eval() mode.
The same is true for the older example notebooks on which I based my code:

• extract_deep_features() and extract_composition_features() in https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/examples/full-pipelines/slide-graph.ipynb
• extract_deep_features() and extract_composition_features() in https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/examples/inference-pipelines/slide-graph.ipynb

I got worried that I was using models in train mode and that I would need to recompute all the features I computed over the last couple of weeks, but then I checked the source code and realised that we rely on the _infer_batch() function, which runs model.eval() for all models that use it (line 173):

tiatoolbox/tiatoolbox/models/architecture/vanilla.py

Lines 148 to 178 in 4a1940d

	def _infer_batch(
	model: nn.Module,
	batch_data: torch.Tensor,
	device: str,
	) -> dict[str, np.ndarray]:
	"""Run inference on an input batch.
	Contains logic for forward operation as well as i/o aggregation.
	Args:
	model (nn.Module):
	PyTorch defined model.
	batch_data (torch.Tensor):
	A batch of data generated by
	`torch.utils.data.DataLoader`.
	device (str):
	Transfers model to the specified device. Default is "cpu".
	"""
	img_patches_device = batch_data.to(device=device).type(
	torch.float32,
	) # to NCHW
	img_patches_device = img_patches_device.permute(0, 3, 1, 2).contiguous()
	# Inference mode
	model.eval()
	# Do not compute the gradient (not training)
	with torch.inference_mode():
	output = model(img_patches_device)
	# Output should be a single tensor or scalar
	return output.cpu().numpy()

I think this behaviour is as it should be, but maybe adding a message specifying that the models are being put into eval mode would be beneficial.

[shaneahmed]

Please remember to update https://github.com/TissueImageAnalytics/tiatoolbox/blob/feature-extractor-example/examples/README.md

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.90%. Comparing base (442bd3f) to head (44e2897).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #887   +/-   ##
========================================
  Coverage    99.90%   99.90%           
========================================
  Files           69       69           
  Lines         8715     8715           
  Branches      1149     1149           
========================================
  Hits          8707     8707           
  Misses           3        3           
  Partials         5        5           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shaneahmed]

Thanks @adamshephard . Please can you re-run the Jupyter notebook. This will update the title on the last image. Please update the feature_extraction.png with the new title. Please crop it to the image area before uploading.

[shaneahmed]

I just realised that in my code, like in this example notebook, we do not explicitly put the model into the .eval() mode. The same is true for the older example notebooks on which I based my code:

• extract_deep_features() and extract_composition_features() in https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/examples/full-pipelines/slide-graph.ipynb
• extract_deep_features() and extract_composition_features() in https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/examples/inference-pipelines/slide-graph.ipynb

I got worried that I was using models in train mode and that I would need to recompute all the features I computed over the last couple of weeks, but then I checked the source code and realised that we rely on the _infer_batch() function, which runs model.eval() for all models that use it (line 173):

tiatoolbox/tiatoolbox/models/architecture/vanilla.py

Lines 148 to 178 in 4a1940d

	def _infer_batch(
	model: nn.Module,
	batch_data: torch.Tensor,
	device: str,
	) -> dict[str, np.ndarray]:
	"""Run inference on an input batch.
	Contains logic for forward operation as well as i/o aggregation.
	Args:
	model (nn.Module):
	PyTorch defined model.
	batch_data (torch.Tensor):
	A batch of data generated by
	`torch.utils.data.DataLoader`.
	device (str):
	Transfers model to the specified device. Default is "cpu".
	"""
	img_patches_device = batch_data.to(device=device).type(
	torch.float32,
	) # to NCHW
	img_patches_device = img_patches_device.permute(0, 3, 1, 2).contiguous()
	# Inference mode
	model.eval()
	# Do not compute the gradient (not training)
	with torch.inference_mode():
	output = model(img_patches_device)
	# Output should be a single tensor or scalar
	return output.cpu().numpy()

I think this behaviour is as it should be, but maybe adding a message specifying that the models are being put into eval mode would be beneficial.

Thanks @GeorgeBatch. As I am now finalising release v1.6.0, we can address this in the next release. Feel free to raise an issue or a PR for this.