download-osm not able to download europe from geofabrik

[frodrigo]

Getting content length for https://download.geofabrik.de/europe-latest.osm.pbf
Getting md5 checksum from https://download.geofabrik.de/europe-latest.osm.pbf.md5
Unable to auto-detect the source for 'europe'

[frodrigo]

Download Europe from geofabrik still fails while it works for other continents.

docker-compose run openmaptiles-tools download-osm geofabrik europe
Unable to auto-detect the source for 'europe'

docker-compose run openmaptiles-tools download-osm geofabrik asia
Downloading https://download.geofabrik.de/asia-latest.osm.pbf (size=8,641.5 MB (9,061,298,348), md5=660b7a0732d02d9b7f959e3b16ac5945)

It is in the Geofabirk list

docker-compose run openmaptiles-tools download-osm geofabrik list
id                                                     name
-----------------------------------------------------  ---------------------------------------------------------------------------------------
central-america/nicaragua                              Central America / Nicaragua
europe                                                 Europe
europe/albania                                         Europe / Albania

[Flowkap]

Any infos? It fails on all our dev laptops but on the server (same repo same command) it fails. ... most other regions work

[TomPohys]

I test ./quickstart.sh germany in the morning, and it works. Now, 7 hour later it is not. #327

I think that is the same problem as with Europe. Manually from the Geofabrik is possible to download it and it is also in JSON list.

[TomPohys]

Germany and Europe are both working now.

[msschl]

Having the same issue... See my comment on issue 327

I ran ./quickstart.sh --empty europe/germany geofabrik and ./quickstart.sh --empty germany geofabrik

Having the same issue...

But I am unable to download europe/germany and germany. Both give the following error:

Unable to auto-detect the source for 'germany'
Makefile:312: recipe for target 'download-geofabrik' failed

[TomPohys]

Hi @msschl, could you please run make download area=germany or ./quickstart.sh germany if it works? On my machine, it works fine now. Thanks

[msschl]

I pulled the latest changes from master and tested again.

Running make download area=germany results in:

Area source has not been specified. Auto-detecting...
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 501, in _sock_connect
    sock.connect(address)
OSError: [Errno 99] Cannot assign requested address

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/app/download-osm", line 763, in <module>
    main()
  File "/usr/src/app/download-osm", line 759, in main
    exit(asyncio.run(main_async(docopt(__doc__, version=openmaptiles.__version__))))
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/src/app/download-osm", line 676, in main_async
    urls, md5, repl_url, bbox = await AreaParamParser(
  File "/usr/src/app/download-osm", line 600, in parse
    res = await site.search(self.area_id, self.is_guessing)
  File "/usr/src/app/download-osm", line 530, in search
    poly = await fetch(self.session, f"https://download.openstreetmap.fr/polygons/{area_id}.poly")
  File "/usr/src/app/download-osm", line 369, in fetch
    async with session.get(url) as resp:
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host download.openstreetmap.fr:443 ssl:default [Cannot assign requested address]
Makefile:312: recipe for target 'download' failed
make: *** [download] Error 1

Running ./quickstart.sh germany results in:

-------------------------------------------------------------------------------------
====> : Downloading germany from any source...
Downloading germany with replication support into data/germany.osm.pbf and data/germany.repl.json from any source
Creating network "openmaptiles_postgres_conn" with driver "bridge"
Area source has not been specified. Auto-detecting...
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 501, in _sock_connect
    sock.connect(address)
OSError: [Errno 99] Cannot assign requested address

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/app/download-osm", line 763, in <module>
    main()
  File "/usr/src/app/download-osm", line 759, in main
    exit(asyncio.run(main_async(docopt(__doc__, version=openmaptiles.__version__))))
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/src/app/download-osm", line 676, in main_async
    urls, md5, repl_url, bbox = await AreaParamParser(
  File "/usr/src/app/download-osm", line 600, in parse
    res = await site.search(self.area_id, self.is_guessing)
  File "/usr/src/app/download-osm", line 530, in search
    poly = await fetch(self.session, f"https://download.openstreetmap.fr/polygons/{area_id}.poly")
  File "/usr/src/app/download-osm", line 369, in fetch
    async with session.get(url) as resp:
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host download.openstreetmap.fr:443 ssl:default [Cannot assign requested address]
Makefile:312: recipe for target 'download' failed
make: *** [download] Error 1

And running ./quickstart.sh --empty europe/germany geofabrik the same:

-------------------------------------------------------------------------------------
====> : Downloading europe/germany from geofabrik...
Downloading europe/germany with replication support into data/europe/germany.osm.pbf and data/europe/germany.repl.json from geofabrik
Creating network "openmaptiles_postgres_conn" with driver "bridge"
Unable to auto-detect the source for 'europe/germany'
Makefile:312: recipe for target 'download-geofabrik' failed
make: *** [download-geofabrik] Error 1

Looks like the first two still try to download from download.openstreetmap.fr

[TomPohys]

When I write a comment it works, now on the same code it is not. I am afraid, that this does not have to be a problem in OMT-T.

[msschl]

Well at least for ./quickstart.sh --empty europe/germany geofabrik it should not try to auto-detect the source, or should it?

[TomPohys]

./quickstart.sh --empty europe/germany geofabrik at 9:30 (GMT+2) returns and start downloading Germany area

-------------------------------------------------------------------------------------
====> : Downloading europe/germany from geofabrik...
Downloading europe/germany into data/europe/germany.osm.pbf from geofabrik
Creating network "test_quickstart_postgres_conn" with driver "bridge"
Creating volume "test_quickstart_pgdata" with default driver
Downloading https://download.geofabrik.de/europe/germany-latest.osm.pbf (size=3,312.1 MB (3,472,989,647), md5=1fb2a27fc0a7ad31887d82457f4b383c)

  aria2c --checksum=md5=1fb2a27fc0a7ad31887d82457f4b383c --http-accept-gzip "--user-agent=OpenMapTiles download-osm 5.3.1 (https://github.com/openmaptiles/openmaptiles-tools)" --dir=/tileset/data/europe --out=germany.osm.pbf --auto-file-renaming=false https://download.geofabrik.de/europe/germany-latest.osm.pbf

For now, there are 4 possible ways, how to download an area. Autodetect just trying to find the working service by trying Geofabrik, then bbbike and as last openstreetmap.fr. The services can be specified and then it will not try to download from another service (as it is in your last example)

[nickpeihl]

I dug into this a bit and think I've discovered the problem. The request for the europe extract is redirected (HTTP 302). But the session.head request here does not follow redirects (aiohttp sets allow_redirects = False by default).

I added allow_redirects = True to the session.head method which sets the Content-Length header in the response.

However, now aria2c is throwing errorCode=1 total length mismatch. expected: 25527519495, actual: 29250612004. Now I'm not sure how to proceed. Is Geofabrik mis-reporting the Content-Length header?

[msschl]

Any news on this or a workaround?