Details

Current code fails unexpectedly if the project calling decides the value is an int instead of a float, adjusted rustprooflabs/pgosm-flex#204 to work around. That should be fixed here.

Current version of osm2pgsql (1.5.1) has a minimum memory requirement published of 2GB. I don't recall seeing that previously and used to be able to run small-ish imports (e.g. Clorado, 200 MB PBF) with only 1 GB RAM or less. Attempted that today and verified that (without swap) it will fail with only 1 GB RAM.

• Update tuner to raise ValueError with hint if RAM < 2GB.
• Webapp route/template needs updating
• API - Return 400 bad request

Details

To add explanations, logic decisions need to be tracked. This is best done where the logic happens, thinking to add a dict to the class that can track logic decisions to be presented to the user.
This would make it trivial to return the details via the browser to display and via the API. Returning decision making data via the API will allow better tracking/auditing of what is being done, and why. Should help with troubleshooting in the long term.

Logic to describe

Explain why various switches and values are chosen. In no particular order:

• If append - --slim must be used, and --drop cannot be used -- Added in #7
• If non-slim cache < ram available to osm2pgsql -> don't use --slim (unless append of course)
• Calculation of max RAM to allocate -- stability and speed are both features, erring on the side of stability
• Calculation of non-slim cache required
• Use --flat-nodes=/path --cache=0 when PBF size exceeds cut-point (storage speed dependent)
• What else?

Min version supported is v1.5.0, current latest. Over time this project should allow tracking recommendations as specific new versions change performance characteristics, RAM requirements, etc.

Latest testing shows using --flat-nodes for U.S. West (2.1. GB) was slower than --slim --drop --cache=5000 w/ 8 GB RAM available. I have seen other places (NEED REFERENCE) mention 10 GB as cutoff, but my prior testing showed lower.... So much has changed w/ osm2pgsql and flex it's worth doing some testing to find the sweet spot.

rustprooflabs/pgosm-flex#129 (comment)

https://osm2pgsql-tuner.com/api/v1?system_ram_gb=8&osm_pbf_gb=2.1&append=False&pbf_filename=us-west-latest&pgosm_layer_set=run-all

Returns cmd:

osm2pgsql -d $PGOSM_CONN  --cache=0  --slim  --drop  --flat-nodes=/tmp/nodes  --output=flex --style=./run-all.lua  ~/pgosm-data/us-west-latest.osm.pbf

Right now I have no idea what the URL is w/out looking at routes or pgosm flex logs that report the URL.

Use current size (59GB) from https://planet.openstreetmap.org/pbf/

URL pattern on localhost

http://localhost:5000/recommendation?system_ram_gb=64&osm_pbf_gb=10.4&append=True&pbf_filename=north-america-latest&pgosm_layer_set=run

Returns:

osm2pgsql -d $PGOSM_CONN \
--cache=0 \
--slim \
--flat-nodes=/tmp/nodes \
--output=flex --style=./run.lua \
~/pgosm-data/north-america-latest.osm.pbf 

Printed description indicates there is sufficient RAM to not need --flat-nodes.

"A server with 64.0 GB RAM should be able to allow up to 42.2 GB RAM to osm2pgsql and still give room for Postgres. Running osm2pgsql entirely in-memory requires 27.0 GB RAM. "

Changing append to false:

http://localhost:5000/recommendation?system_ram_gb=64&osm_pbf_gb=10.4&append=False&pbf_filename=north-america-latest&pgosm_layer_set=run

No flat nodes (as well as no slim, expected w/out append)

osm2pgsql -d $PGOSM_CONN \
--output=flex --style=./run.lua \
~/pgosm-data/north-america-latest.osm.pbf 

The use of --slim with --flat-nodes seems wrong. If it is valid -- need to update printed text to explain WHY.

Details

Should make the input to get_osm2pgsql_command() more flexible to allow setting full path.

https://github.com/rustprooflabs/osm2pgsql-tuner/blob/0.0.4/osm2pgsql_tuner/tuner.py#L216

Details

Only matters in --slim mode.

When --cache is less than total available and PBF size > 2 GB.

Details

This issue was discovered while investigating rustprooflabs/pgosm-flex#275.

Example

curl "https://osm2pgsql-tuner.com/api/v1?system_ram_gb=8&osm_pbf_gb=0.203&append=True&pbf_filename=colorado-latest&pgosm_layer_set=run"

Returns the following. I expected cmd to have --append, but it does not.

{
  "osm2pgsql": {
    "cmd": "osm2pgsql -d $PGOSM_CONN  --cache=1157  --slim  --output=flex --style=./run.lua  ~/pgosm-data/colorado-latest.osm.pbf", 
    "decisions": [
      {
        "desc": "Using --append, cannot use --drop", 
        "name": "Using Append", 
        "option": "--drop"
      }, 
      {
        "desc": "No reason to use --flat-nodes", 
        "name": "Not using", 
        "option": "--flat-node"
      }, 
      {
        "desc": "Set --cache to expected requirement for given PBF size.", 
        "name": "Sufficient RAM", 
        "option": "--cache"
      }
    ], 
    "osm2pgsql_append": true, 
    "osm2pgsql_cache_max": 5.28, 
    "osm2pgsql_drop": false, 
    "osm2pgsql_flat_nodes": false, 
    "osm2pgsql_limited_ram": false, 
    "osm2pgsql_noslim_cache": 1.5075, 
    "osm2pgsql_run_in_ram": false, 
    "osm2pgsql_slim_cache": 1.130625, 
    "system_ram_gb": 8.0
  }
}