Tests: remove reference values; improve test verbosity (#114)

* tests: rm references
* tests/README: better docs;
* utils: less verbosing
* tests: README: improvements in doc

.gitignore

@@ -25,6 +25,10 @@
 ./output
 ./Output
 
+# test files
+./pytest_cache
+reference
+
 # vggish checkpoints
 models/vggish/hub/checkpoints/vggish-10086976.pth
 models/vggish/hub/checkpoints/vggish_pca_params-970ea276.pth

tests/README.md

@@ -7,42 +7,46 @@ The test might not cover all possible use-cases yet all minimal working examples
 as well as google colab notebooks must be tested.
 
 Now, the tests focus on comparing the shapes and the values of the output features (with some tolerance).
-These are compared with the commited references in `./tests/*/reference/`.
-
 We expect that the same code on different machines/setups might output different feature values
 slightly exceeding the tolerance.
-It is ok if you get the same values compared to the code state before your changes.
-Also feel free to recalculate the references on your setup by removing the `./tests/*/reference/` folders and
-toggling `TO_MAKE_REF` to `True` in each file.
+This makes comparing values a bit tricky.
 
+For this reason, we use the following approach:
+First, you should make reference values for the tests on the `master` branch, before you make changes.
+Then, make changes you want (or checkout a branch) and run the tests which will compare the outputs of the
+new code to the references from `master`.
+These should pass if nothing is broken.
 
+# Ok, how to run tests?
 
-# How to run tests?
+First, checkout to `master` and run tests with `TO_MAKE_REF` set to `True` in each file `tests/*/test_*.py`
+for which you want to run a test (Find & Replace works for well me).
+Then, checkout to the branch you want to test and run tests with `TO_MAKE_REF` set to `False`.
 
-You may use the conda environment that was installed locally or
-the [Docker](https://v-iashin.github.io/video_features/meta/docker) container.
+To run tests for a specific model (`vggish`), you may use the following command:
+```bash
+# conda activate torch_zoo
+pytest tests/vggish
+```
 
 ```bash
 # conda activate video_features
 pytest
 ```
 
 Also, remember that running the code with `show_pred` should yield something reasonable.
+You may use the conda environment that was installed locally or
+the [Docker](https://v-iashin.github.io/video_features/meta/docker) container.
 
 # How to make a test?
 
 **New test for an old model**
 Just add another row to the decorator `@pytest.mark.parametrize`.
-Then, comment it and run the old tests and make sure they pass.
-Next, uncomment the new line, toggle `TO_MAKE_REF` to `True`, remove the corresponding `reference` folder,
+Next, remove the corresponding `reference` folder, toggle `TO_MAKE_REF` to `True`,
 run the tests to make new references, and toggle `TO_MAKE_REF` back to `False`.
-Note, the reference files might be >100MB in size making it hard to commit to github.
-
+Finally, rerun the tests to make sure that they pass.
 
 **New model**
 Pick an implemented test from another model that is most similar to the new model and build on top of it.
-Make sure to patch the config for the 'import API tests', otherwise you will get the same output
-every time passing all tests.
-Also, toggle `TO_MAKE_REF` to `True` and during the first run it will create reference values and
-remove `reference` folder if it already exists because it will raise an error.
-Note, the reference files might be >100MB in size making it hard to commit to github.
+Also, toggle `TO_MAKE_REF` to `True`, remove the corresponding `reference` folder (if exists),
+run the tests to make new references.

tests/utils.py

@@ -34,7 +34,9 @@ def make_ref_path(feature_type, file_key, **patch_kwargs):
 
 
 def make_ref(args, video_path: Path, data, save_path):
-    assert not save_path.exists(), 'Do you make another ref????'
+    assert not save_path.exists(), 'Making another reference? ' \
+                                   f'If so, delete the old dir: "{save_path.parent}"' \
+                                   ' or toggle `TO_MAKE_REF` to False in the file you are testing.'
     save_path.parent.mkdir(exist_ok=True)
     to_save = {
         'args': args,
@@ -125,9 +127,9 @@ def base_test_script(feature_type: str, Extractor, to_make_ref: bool, **patch_kw
         if to_make_ref:
             make_ref(args, patch_kwargs['video_paths'], feat_out, ref_path)
         feat_ref = torch.load(ref_path)['data']
-        print(k)
-        print(feat_out - feat_ref)
+        # print(k)
+        # print(feat_out - feat_ref)
         # compare shapes
-        assert feat_out.shape == feat_ref.shape
+        assert feat_out.shape == feat_ref.shape, f'feat_out: {feat_out.shape}\nfeat_ref: {feat_ref.shape}'
         # compare values
-        assert all_close(feat_out, feat_ref)
+        assert all_close(feat_out, feat_ref), f'feat_out: {feat_out}\nfeat_ref: {feat_ref}'