Startup fails non-deterministically due to transient dependency on both bson and pymongo

[holgerschmeisky]

First of all, thank you very much for the great work! :)

Description
Flair fails to start up under certain conditions yielding this error

  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/flair/datasets.py", line 9, in <module>
    import pymongo
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pymongo/__init__.py", line 77, in <module>
    from pymongo.collection import ReturnDocument
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pymongo/collection.py", line 20, in <module>
    from bson.code import Code
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/bson/code.py", line 18, in <module>
    from bson.py3compat import abc, string_type, PY3, text_type
ImportError: cannot import name 'abc'

The error is described here as well in the bson project: py-bson/bson#82

To Reproduce
The error happens non-deterministically if dependencies get installed in the wrong order, to fix it I just restarted my build twice and on third attempt it worked.

Environment
flair-0.4.4

Root Cause
The root cause is that Flair depends on both pymongo and bson (via hyperopt, see output of pipdeptree below). pymongo brings its own version of a bson library, so if the order of installing the dependencies is wrong, the bson library gets overwritten leading to the above error.

Suggested Fix
PyMongo is only used by MongoDataset (https://github.com/zalandoresearch/flair/blob/master/flair/datasets.py#L1186), I'd suggest to move it to its own PyPI module or a Gist.

pipdeptree output for flair:

flair==0.4.4
  - bpemb [required: >=0.2.9, installed: 0.3.0]
    - gensim [required: Any, installed: 3.8.1]
      - numpy [required: >=1.11.3, installed: 1.16.4]
      - scipy [required: >=0.18.1, installed: 1.3.1]
        - numpy [required: >=1.13.3, installed: 1.16.4]
      - six [required: >=1.5.0, installed: 1.12.0]
      - smart-open [required: >=1.8.1, installed: 1.8.4]
        - boto [required: >=2.32, installed: 2.49.0]
        - boto3 [required: Any, installed: 1.9.50]
          - botocore [required: >=1.12.50,<1.13.0, installed: 1.12.253]
            - docutils [required: >=0.10,<0.16, installed: 0.15.2]
            - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
            - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
              - six [required: >=1.5, installed: 1.12.0]
            - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
          - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
            - botocore [required: >=1.3.0,<2.0.0, installed: 1.12.253]
              - docutils [required: >=0.10,<0.16, installed: 0.15.2]
              - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
              - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
                - six [required: >=1.5, installed: 1.12.0]
              - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
        - requests [required: Any, installed: 2.22.0]
          - certifi [required: >=2017.4.17, installed: 2019.9.11]
          - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
          - idna [required: >=2.5,<2.9, installed: 2.8]
          - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
    - numpy [required: Any, installed: 1.16.4]
    - requests [required: Any, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.9.11]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
    - sentencepiece [required: Any, installed: 0.1.83]
    - tqdm [required: Any, installed: 4.31.1]
  - deprecated [required: >=1.2.4, installed: 1.2.6]
    - wrapt [required: >=1.10,<2, installed: 1.11.2]
  - gensim [required: >=3.4.0, installed: 3.8.1]
    - numpy [required: >=1.11.3, installed: 1.16.4]
    - scipy [required: >=0.18.1, installed: 1.3.1]
      - numpy [required: >=1.13.3, installed: 1.16.4]
    - six [required: >=1.5.0, installed: 1.12.0]
    - smart-open [required: >=1.8.1, installed: 1.8.4]
      - boto [required: >=2.32, installed: 2.49.0]
      - boto3 [required: Any, installed: 1.9.50]
        - botocore [required: >=1.12.50,<1.13.0, installed: 1.12.253]
          - docutils [required: >=0.10,<0.16, installed: 0.15.2]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
            - six [required: >=1.5, installed: 1.12.0]
          - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
        - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
          - botocore [required: >=1.3.0,<2.0.0, installed: 1.12.253]
            - docutils [required: >=0.10,<0.16, installed: 0.15.2]
            - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
            - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
              - six [required: >=1.5, installed: 1.12.0]
            - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
      - requests [required: Any, installed: 2.22.0]
        - certifi [required: >=2017.4.17, installed: 2019.9.11]
        - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
        - idna [required: >=2.5,<2.9, installed: 2.8]
        - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - hyperopt [required: >=0.1.1, installed: 0.2.1]
    - bson [required: Any, installed: 0.5.8]
      - python-dateutil [required: >=2.4.0, installed: 2.8.0]
        - six [required: >=1.5, installed: 1.12.0]
      - six [required: >=1.9.0, installed: 1.12.0]
    - cloudpickle [required: Any, installed: 1.2.2]
    - future [required: Any, installed: 0.18.1]
    - networkx [required: ==2.2, installed: 2.2]
      - decorator [required: >=4.3.0, installed: 4.4.0]
    - numpy [required: Any, installed: 1.16.4]
    - scipy [required: Any, installed: 1.3.1]
      - numpy [required: >=1.13.3, installed: 1.16.4]
    - six [required: Any, installed: 1.12.0]
    - tqdm [required: Any, installed: 4.31.1]
  - ipython [required: ==7.6.1, installed: 7.6.1]
    - appnope [required: Any, installed: 0.1.0]
    - backcall [required: Any, installed: 0.1.0]
    - decorator [required: Any, installed: 4.4.0]
    - jedi [required: >=0.10, installed: 0.15.1]
      - parso [required: >=0.5.0, installed: 0.5.1]
    - pexpect [required: Any, installed: 4.7.0]
      - ptyprocess [required: >=0.5, installed: 0.6.0]
    - pickleshare [required: Any, installed: 0.7.5]
    - prompt-toolkit [required: >=2.0.0,<2.1.0, installed: 2.0.10]
      - six [required: >=1.9.0, installed: 1.12.0]
      - wcwidth [required: Any, installed: 0.1.7]
    - pygments [required: Any, installed: 2.4.2]
    - setuptools [required: >=18.5, installed: 40.8.0]
    - traitlets [required: >=4.2, installed: 4.3.3]
      - decorator [required: Any, installed: 4.4.0]
      - ipython-genutils [required: Any, installed: 0.2.0]
      - six [required: Any, installed: 1.12.0]
  - ipython-genutils [required: ==0.2.0, installed: 0.2.0]
  - langdetect [required: Any, installed: 1.0.7]
    - six [required: Any, installed: 1.12.0]
  - matplotlib [required: >=2.2.3, installed: 3.1.1]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.12.0]
    - kiwisolver [required: >=1.0.1, installed: 1.1.0]
      - setuptools [required: Any, installed: 40.8.0]
    - numpy [required: >=1.11, installed: 1.16.4]
    - pyparsing [required: >=2.0.1,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.2]
    - python-dateutil [required: >=2.1, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
  - mpld3 [required: ==0.3, installed: 0.3]
  - pymongo [required: Any, installed: 3.9.0]
  - pytest [required: >=3.6.4, installed: 5.2.1]
    - atomicwrites [required: >=1.0, installed: 1.3.0]
    - attrs [required: >=17.4.0, installed: 19.3.0]
    - importlib-metadata [required: >=0.12, installed: 0.23]
      - zipp [required: >=0.5, installed: 0.6.0]
        - more-itertools [required: Any, installed: 7.2.0]
    - more-itertools [required: >=4.0.0, installed: 7.2.0]
    - packaging [required: Any, installed: 19.2]
      - pyparsing [required: >=2.0.2, installed: 2.4.2]
      - six [required: Any, installed: 1.12.0]
    - pluggy [required: >=0.12,<1.0, installed: 0.13.0]
      - importlib-metadata [required: >=0.12, installed: 0.23]
        - zipp [required: >=0.5, installed: 0.6.0]
          - more-itertools [required: Any, installed: 7.2.0]
    - py [required: >=1.5.0, installed: 1.8.0]
    - wcwidth [required: Any, installed: 0.1.7]
  - regex [required: Any, installed: 2019.8.19]
  - segtok [required: >=1.5.7, installed: 1.5.7]
    - regex [required: Any, installed: 2019.8.19]
  - sklearn [required: Any, installed: 0.0]
    - scikit-learn [required: Any, installed: 0.21.2]
      - joblib [required: >=0.11, installed: 0.14.0]
      - numpy [required: >=1.11.0, installed: 1.16.4]
      - scipy [required: >=0.17.0, installed: 1.3.1]
        - numpy [required: >=1.13.3, installed: 1.16.4]
  - sqlitedict [required: >=1.6.0, installed: 1.6.0]
  - tabulate [required: Any, installed: 0.8.5]
  - tiny-tokenizer [required: Any, installed: 3.0.1]
  - torch [required: >=1.1.0, installed: 1.3.0.post2]
    - numpy [required: Any, installed: 1.16.4]
  - torchvision [required: Any, installed: 0.4.1.post2]
    - numpy [required: Any, installed: 1.16.4]
    - pillow [required: >=4.1.1, installed: 6.2.1]
    - six [required: Any, installed: 1.12.0]
    - torch [required: ==1.3.0.*, installed: 1.3.0.post2]
      - numpy [required: Any, installed: 1.16.4]
  - tqdm [required: >=4.26.0, installed: 4.31.1]
  - transformers [required: >=2.0.0, installed: 2.1.1]
    - boto3 [required: Any, installed: 1.9.50]
      - botocore [required: >=1.12.50,<1.13.0, installed: 1.12.253]
        - docutils [required: >=0.10,<0.16, installed: 0.15.2]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
          - six [required: >=1.5, installed: 1.12.0]
        - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.12.253]
          - docutils [required: >=0.10,<0.16, installed: 0.15.2]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.0]
            - six [required: >=1.5, installed: 1.12.0]
          - urllib3 [required: >=1.20,<1.26, installed: 1.25.6]
    - numpy [required: Any, installed: 1.16.4]
    - regex [required: Any, installed: 2019.8.19]
    - requests [required: Any, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.9.11]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
    - sacremoses [required: Any, installed: 0.0.35]
      - click [required: Any, installed: 7.0]
      - joblib [required: Any, installed: 0.14.0]
      - six [required: Any, installed: 1.12.0]
      - tqdm [required: Any, installed: 4.31.1]
    - sentencepiece [required: Any, installed: 0.1.83]
    - tqdm [required: Any, installed: 4.31.1]
  - urllib3 [required: >=1.20,<1.25, installed: 1.25.6]

[petulla]

Same error. To fix I did:

pip uninstall pymongo
pip uninstall bson
pip uninstall flair
pip install bson
pip install pymongo
pip install flair

[dparkar]

you might want to pin the pymongo version

[alanakbik]

@holgerschmeisky thanks for reporting this. Maybe removing the dependency is the easiest fix for this? We would also like to keep the dependency tree as lightweight as possible, so we could ask users who want to use this feature to install pymongo separately.

(We do this already for the ELMoEmbeddings - users need to install allennlp in order to use them.)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alanakbik]

This was fixed a while back.