v5.0 back-ported undeclared EXPIRETILES_DIR fix & ver

Bugfix release: #282 and incorrect version number was updated to 5.0.1

Postgis 3, smarter download-osm, performance improvements

Summary

• Migrate to Postgis 3.0.1 and osml10n v2.5.8, add GZIP extension
• Improved download-osm tool
• Docs and a script to quickly set up Google Cloud testing environment
• A new profiling tool to evaluate PG function performance profile-pg-func
• SQL performance improvements
• Removed unused Natural Earth tables

download-osm Improvements

Many download-osm tool improvements (#255) - @nyurik

• Added --output <file> param to specify the location and the name of the file to save. Also allows --force to override existing file.
• allow area and URL downloading without explicitly specifying the source:

download-osm toronto      # will search Geofabrik, BBBike, and osm.fr
download-osm https://...  # will download from URL

• Auto-detection logic:

  • if starts with http:// or https:// -- try it as a URL
  • look through Geofabrik catalog, matching the last portion of the name (e.g. michigan would match us/michigan) -- this is the same logic as before
  • look through BBBike catalog (string-insensitive comparison, must be a full match)
  • check if osm.fr gives a 200 rather than 404
  • fail otherwise

• added cacheable BBBike catalog (scraped from a single HTML page), allowing non-exact searches (download-osm bbbike toronto instead of Toronto) and catalog listing (download-osm list bbbike)

• better handling of mirrors that have just the latest planet file

• added a few mirrors

• CHANGE: --make-dc will now set OSM_AREA_NAME to the full ID of the area, e.g. for Geofabrik it will be europe/monaco instead of monaco

New profile-pg-func benchmarking utility

Add profile-pg-func to evaluate PG function speed (#263) - @nyurik

profile-pg-func is a micro-benchmarking tool that runs all given functions thousands of times, in several runs, discarding slowest and fastest results. Use it to analyze relative performance between function implementations.

profile-pg-func [--file <file>]... <func>... [--raw]
                [--calls <calls>] [--runs <runs>] [--trim <trim>]
                [--verbose] [--pghost=<host>] [--pgport=<port>] [--dbname=<db>]
                [--user=<user>] [--password=<password>]

<func>        A SQL function call that should be tested, e.g. 'random()'.
              The func code can use call_idx column -- it will be set to  1..{calls}
              If --raw is used, the <func> must be a complete query.
              These magical values will be substituted in <func> before running:
              * {calls} will be replaced with --calls value
              * {run_idx}  will be replaced with the current RUN number (1..{runs})
              * {random_geopoint} will be replaced with a function call to generate
                a random geo point.

Remove unused Natural Earth tables

• Shrink the size of the import-data and postgis-preloaded images by removing all tables that OMT does not use. (#262) - @nyurik
  • ne_10m_admin_0_boundary_lines_disputed_areas
  • ne_10m_rivers_europe
  • ne_10m_rivers_north_america
  • ne_10m_roads
  • ne_10m_roads_north_america
  • ne_10m_urban_areas
  • ne_50m_admin_0_breakaway_disputed_areas_scale_rank
  • ne_50m_admin_1_states_provinces
• remove unneeded clean-natural-earth.sh from the resulting docker images

Other Improvements

• Documentation and a script to quickly set up a testing env in Google Cloud (#267) - @nyurik
• Upgrade to the official Postgis docker image (PG 9.6 + Postgis 3.0) (#268) - @nyurik
• Cleaner generate-tiles (Mapnik) output if FILTER_MAPNIK_OUTPUT is set (#256) - @zstadler
• Upgrade to osml10n v2.5.8 (#269) - @nyurik
• Create directory for state file, if needed (#249) - @zstadler
• Improve sql tests for import-borders (#258) - @nyurik
• Significant SQL performance improvements of CleanNumeric (#264) and LineLabel (#265) - @nyurik
• Migrate relevant functions from the postgis-vt-util lib into tools (#247) - @nyurik

Bug Fixes

• Download state file from geofabrik whether exact name match or not (#251) - @frodrigo

import-osm merged into tools, refactoring, profiling, new features

Documentation

• Document how to build PostgreSQL server locally and on Google Cloud (@nyurik)

Merge import-osm docker image into tools

• import-osm, update-osm, and import-diff are now executed as any other tools from the main tools docker image. (#188) (@nyurik)
• import-osm can now accept PBF file as the first parameter, rather than auto-search for the first file in the PBF_DATA_DIR (#234) (@nyurik)
• Removed pgfutter tool -- that tool's functionality was replaced with psql -c '\copy ... CSV' command (see import-data image)
• removed /tileset and /sql volume declarations (no point in forcing those dirs to be volumes by default) -- they can still be mounted as external volumes.

Dependent layer support

Added support dependent layer code via "requires" param (#205) (@nyurik)

• The code can now produce parallel or linear SQL file(s), taking into account the requires layer parameter. If layer A is required by B and C, it will produce one joined ABC file, while allowing other layers to execute in parallel.

Build & CI

• Use GitHub actions to build and publish all docker images (@nyurik)
• Revised docker publishing prorcess:
  • Auto-publish all version tags, e.g. v5.1.0 published as 5.1.0 and as minor 5.1. The minor version will always point to the latest patch version.
  • master is auto-published as latest
  • Introduce version tracking branches, e.g. v5.0 published as 5.0-dev,
    and any test branches like v5.0-pg11 will auto-publish as 5.0-pg11-dev. All *-dev docker images can be deleted at any moment.

download-osm tool

• download-osm can now generate imposm configuration (#213) (@nyurik)
• Handle several error conditions due to unexpected data (@nyurik)

mbtiles-tools tool

• meta-generate command populates mbtiles metadata table from tileset and PostgreSQL DB, with optional env var overrides. (@nyurik) (#208)
• meta-copy command copies all metadata values from one mbtile to another, optionally updating some values. (@nyurik) (#208)

Better openstreetmap.fr support

• import-borders script can optionally filter out broken refs (#216) (@zstadler & @nyurik)
• support for download-osm --imposm-cfg #238 (@zstadler)

Environment Variable Cleanup

Several env variables have been reworked (#234) (@nyurik)

• rename CONFIG_JSON ⇒ IMPOSM_CONFIG_FILE
• rename MAPPING_YAML ⇒ IMPOSM_MAPPING_FILE
• rename IMPORT_DIR ⇒ PBF_DATA_DIR
• rename DIFF_DIR ⇒ IMPOSM_DIFF_DIR
• rename TILES_DIR ⇒ EXPIRETILES_DIR

Language and utility SQL files have been combined into a single dir - /sql, which means both VT_UTIL_DIR and OMT_UTIL_DIR have been removed, and a single SQL_TOOLS_DIR location was added.

• rename OMT_UTIL_DIR and VT_UTIL_DIR ⇒ SQL_TOOLS_DIR (merged)

debug-mvt & test-perf

• debug-mvt tool support for the summary mode - runs a single MVT query instead of one-per-layer to check gzip/md5 generation. (#232) (@nyurik)
• support for layer ordering in generated mvt query (off by default) (#232) (@nyurik)
• support test-perf bbox tests (#221 & #243) (@nyurik)

Misc

• bump required python libraries to latest
• use "${VAR:?}" instead of "${VAR?}" in a few places -- ensures that VAR is set to a non-null value (as opposed to just set but could be null).
• refresh-views now accepts the --verbose flag (#228) (@zstadler)
• Remove obsolete data docker images (#225) (@nyurik)
• add osmosis to the tools image (#213) (@nyurik)

Consolidated import-data image, many tool improvements

download-osm

Many improvements and bug fixes like state file support and stats generation - @nyurik

• support for Geofabrik list and state files (#163)
• Support for OSM download metadata (#168) - Similar to docker/import-osm/download-geofabrik.sh script, uses osmconvert tool to extract file statistics, and generate a docker-compose config file with those values.
• supports --make-dc parameter to generate the dc stats file (#168)
• Support docker compose versions in download-osm (#184)
• Handle broken .md5 files in planet download (#211)
• Get proper Geofabrik catalog instead of scraping (#171)

import-wikidata

Several changes (#191) by @nyurik, @TomPohys

• BREAKING wd_names table no longer has pages column, and the id column is now a primary key
• import-wikidata will always re-create the wd_names table
• language.sql will create the wd_names table only if it's missing
• wikidata is cached in a local file

debug-mvt

• allow tile to be specified as z,x,y, handle SQL errors (#206) - @nyurik

natural-earth data

• Explicitly list natural earth tables to include (#190)

import-borders

borders (#191 by @nyurik, @TomPohys) can now be cached in a local file, generated from the downloaded PBF.

• import-borders - Parse and import first PBF file in IMPORT_DIR
• import-borders [import] planet.pbf - Parse and import planet.pbf
• import-borders parse planet.pbf - Parse planet.pbf into a CSV file, but do not import
• import-borders load borders.csv - Load borders.csv into a table

Other

• BREAKING pgfutter is no longer packaged in the tools because \copy in psql works similarly.
• A lot of new tests for SQL and Python
• mbtiles-tools bug to handle empty metadata - @nyurik

New import-data docker image

Added a new import-data image that combines import-lakelines, import-natural-earth, and import-water, which will be removed in the next release. (#187) - @nyurik

New postgis-preloaded image

This is a Docker image that has been pre-loaded with lakelines, natural-earth, and water data.
This image can be used for quicker testing, but might be less suited for a production environment.
The image contains a PostgreSQL database openmaptiles, user=openmaptiles, password=openmaptiles
(#186) - @nyurik

Environment variable changes

Minor docker var improvements (#185) - @nyurik

• lakelines, natural-earth, and water docker images now support PGCONN env var to be passed in.
• added minor postgis output when initializing databases
• when importing water, ensure PGCLIENTENCODING=UTF8 is set

Imposm tags

• Fix imposm tags handling bug (#179), allowing different layers to include additional tags (rather than overriding it, thus the last layer wins) - (#181) @frodrigo, @nyurik

PostGIS 2.4.8 / GEOS 3.6

• Upgrade to PostGIS 2.4.8/GEOS 3.6.4 to fix auto-analyze errors (#166) - @amatissart, @nyurik

etldoc cleanup

• Make graphs cleaner by removing duplicate lines between the same nodes (#164) - @nyurik

Faster Wikidata, borders, planet download. Many new features.

Big changes

• Use live Wikidata Query Service to load localized names #136 (@nyurik)
• Tileset and Layers have been rewritten to simplify definition access #143 (@nyurik)
• Import borders directly from the downloaded PBF file #142 (@nyurik)
• Postserve now uses Tileset for metadata, URL changes #145 (@francois2metz)
• Faster OSM download uses all mirrors in parallel #144 (@nyurik)
• Richer declarative SQL generation for fields #138 (@nyurik)
• Support for delayed materialized view creation #138 (@nyurik)
• Generate etl and mapping graphs for all layers in one call #147 (@nyurik)
• Use parallel-compatible OpenMapTiles fork of the postgis-vt-util, new omt_as_numeric() SQL function #157 (@jsanz)
• Many bug fixes and improvements

Cleanup

• Removed obsolete docker images import-wikidata, import-sql, generate-osmborder, import-osmborder (@nyurik)

P.S.

There has been a few bug fixes within few days after the initial release of the 4.0.0 - they have been back-ported into the same release, which is not proper, and shouldn't happen in the next one (will use -beta first). Please use docker pull if you got the initial 4.0.0 within that period.

• See: #160, #161, #162 (@nyurik)

Support for legacy postgis 2.4dev image, MVT improvements

BREAKING

• Removed --no-tile-envelope parameter from generate-sqltomvt, postserve, test-perf.
  Now ST_TileEnvelope() support is automatically determined by the PG version - @nyurik

Improvements

• Added MVT support for the legacy Postgis 2.4dev (current docker image) - @nyurik
• MVT tools now customize generated SQL based on the PostGIS version - @nyurik
• Make EXPIRETILES_ZOOM customizable in import-osm - @francois2metz
• debug-mvt now prints total MVT layer size, and can print <null> in values - @nyurik

First consolidated tools release

• Many tool repositories have been consolidated into a single tools repo - see /docker dir
• Many improvements related to generating MVT tiles using PostGIS's ST_AsMVT() function, without any extra tools like Mapnik. An MVT tile can be fully generated by PostgreSQL and sent to the client as is. See generate-sqltomvt
• New tool to test tile generation performance test-perf
• New tool to debug MVT tile content debug-mvt
• postserve and import-sql are now part of the core tools. The corresponding docker images will be removed in the next release.

Pre-release of the consolidated tools repo

Use proper shebang, e.g.  #!/usr/bin/env bash

Fix all scripts to use the same preferred bash and sh shebang

parallel Import-sql, speedy Postserve, performance testing

Many improvements with this release, mostly around MVT, SQL importing, and Performance testing

• The Postserve realtime tile server is now much more robust, supporting multiple parallel queries and request cancelations [@eyusupov, @nyurik]
• Improvements to SQL importing [@nyurik]
  • The import-sql has been merged into this repo.
  • generate-sql can now generate multiple .sql files that can be imported in parallel
  • import_sql.sh can import multiple sql files in parallel
• New test-perf tool to evaluate tile generation performance against a PostgreSQL database. [@nyurik]

This utility can test individual layers, several layers at once, as well as the whole tile. It has several pre-defined testing areas to provide cross-test consistency. Results are printed as histogram graphs, showing tile size distribution. The utility can compare current run with a previously saved one, highlighting large changes. If multiple zoom levels are tested, another histogram shows per-zoom size distribution.

• Expanded MVT generation utility with numerous new features [@nyurik]
  • support for server side gzipping,
• Support for the standard PostgreSQL environment variable naming like PGHOST, PGUSER, ... in addition to the legacy custom POSTGRES_HOST, ... [@nyurik]
• Bug fix for the mapping diagram issue, not generating images for some layers. [@nyurik]

New Postgis service serving dynamic tiles

Postserve quickstart with docker

• clone openmaptiles repo (openmaptiles-tools repo is not needed with docker)
• get a PostgreSQL server running with the openmaptiles-imported OSM data, e.g. by following quickstart guide.
• run docker pull openmaptiles/openmaptiles-tools to download the latest tools version
• from inside the openmaptiles repo dir, run this command.
  (This assumes PostgreSQL is on the localhost:5432, but if it runs inside docker, you may want to change --net=host to --net=openmaptiles_postgres_conn to match the openmaptiles quickstart, and also expose port 8090 to the host with -p 8090:8090)

docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \
    -v "${PWD}:/tileset" --net=host \
    openmaptiles/openmaptiles-tools \
    postserve openmaptiles.yaml 

Add --help to see all additional parameters.

• Run Maputnik and set its data source to http://localhost:8090