Configurable tile sizes with a metatile

[zerebubuth]

The previous behaviour to generate all the tiles of different sizes within a metatile. That worked great while we were generating 512px and 256px tiles in a size 2 metatile. It kinda worked while we were generating 1024px, 512px and 256px tiles in a size 4 metatile. However, moving to a size 8 metatile means we're duplicating information across 4 different sizes of tile - two of which we don't use ever, and the last one we rarely use.

This change introduces configurable tile sizes within a metatile. For example, it's possible to specify just 256, just 512 or both. This will cut down on extra, unwanted data in metatiles.

There are some special cases to this:

At maximum zoom

For maximum nominal zoom tiles, smaller tiles are always included even if not configured. This means that if the maximum nominal zoom is 16 and the configured tile-sizes: [512], the metatile containing nominal zoom 16 (e.g: at zoom 13 for 8x8 metatiles) will contain both 512px and 256px tiles. This allows tapalcatl to treat the smaller tiles as "overzooms" for the larger tiles.

At minimum zoom

For zoom zero tiles, unused "slots" in the metatile for larger tiles will be used by lower zoom tiles. For example, if the only configured tile size is 512px and the metatile size is 8x8, then we would expect the 0/0/0 metatile to contain the 4x4 2/x/y 512px tiles only. However, this means we don't have any tiles for requests at zoom levels 0 and 1. So we also render the 512px tiles at lower nominal zooms to provide those.

Normally, a metatile would only contain tiles all at the same nominal zoom. This change breaks that, but means we don't have to add negative zoom tiles to the already complicated mental model of zooms, metatiles, nominal zooms, etc...

This last change fixes an issue with not generating zoom 0: #341.

[nvkelso]

Always two ways to skin a cat...

…

On Jul 4, 2018, at 10:45, Matt Amos ***@***.***> wrote: The previous behaviour to generate all the tiles of different sizes within a metatile. That worked great while we were generating 512px and 256px tiles in a size 2 metatile. It kinda worked while we were generating 1024px, 512px and 256px tiles in a size 4 metatile. However, moving to a size 8 metatile means we're duplicating information across 4 different sizes of tile - two of which we don't use ever, and the last one we rarely use. This change introduces configurable tile sizes within a metatile. For example, it's possible to specify just 256, just 512 or both. This will cut down on extra, unwanted data in metatiles. There are some special cases to this: At maximum zoom For maximum nominal zoom tiles, smaller tiles are always included even if not configured. This means that if the maximum nominal zoom is 16 and the configured tile-sizes: [512], the metatile containing nominal zoom 16 (e.g: at zoom 13 for 8x8 metatiles) will contain both 512px and 256px tiles. This allows tapalcatl to treat the smaller tiles as "overzooms" for the larger tiles. At minimum zoom For zoom zero tiles, unused "slots" in the metatile for larger tiles will be used by lower zoom tiles. For example, if the only configured tile size is 512px and the metatile size is 8x8, then we would expect the 0/0/0 metatile to contain the 4x4 2/x/y 512px tiles only. However, this means we don't have any tiles for requests at zoom levels 0 and 1. So we also render the 512px tiles at lower nominal zooms to provide those. Normally, a metatile would only contain tiles all at the same nominal zoom. This change breaks that, but means we don't have to add negative zoom tiles to the already complicated mental model of zooms, metatiles, nominal zooms, etc... This last change fixes an issue with not generating zoom 0: #341. You can view, comment on, or merge this pull request online at: #342 Commit Summary Refactor cut tile enumeration logic into one place. Refactor more code out into a common method. Rework so that multiple different nominal zooms can be processed at once. Refactor again to preserve exceptions. Seems simpler than catching and re-throwing without losing the stacktrace. File Changes M config.yaml.sample (33) M tests/test_config.py (4) M tests/test_process.py (286) M tilequeue/command.py (82) M tilequeue/config.py (6) M tilequeue/process.py (190) Patch Links: https://github.com/tilezen/tilequeue/pull/342.patch https://github.com/tilezen/tilequeue/pull/342.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[rmarianski]

Just a thought, we might want to consider making the api/entry point be the processor instance itself in the future that clients called. I think it would be extremely rare that in a course of a python process running we'd need to support processing across different zoom configurations and the abstraction you made here seems like a cleaner interface.

[zerebubuth]

I went backwards and forwards on the interface. Overall, I think a single function call is cleaner, but I had trouble separating out the exceptions in the fetch phase from the processing phase. That's a helpful distinction, so I thought it would be good to keep.

Anyone know how to cleanly support try:/except: changing or wrapping the exception while keeping the stack trace intact? I feel like this is something I used to know, but have forgotten 😕

[rmarianski]

Sorry, I just meant that we can keep the single function call, just have most of the "config" state in the closure/instance.