kennykb / osm-shields Goto Github PK

Describe the bug
The stored procedures are mistakenly rendering snowmobile routes as road routes.

To Reproduce
See '7I' and '7B' in https://kbk.is-a-geek.net/catskills/test4.html?la=42.6052&lo=-74.2573&z=13

Expected behavior
Shoudn't do that.

Screenshots

Environment (please complete the following information):

• PostgreSQL and PostGIS versions
  PostgreSQL 10.5 (Ubuntu 10.5-0ubuntu0.18.04) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0, 64-bit
  POSTGIS="2.4.3 r16312" PGSQL="100" GEOS="3.6.2-CAPI-1.10.2 4d2925d6" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.2.3, released 2017/11/20" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" RASTER

• Description of the OpenStreetMap export that you used
  northamerica from GeoFabrik
  sequenceNumber=1992
  timestamp=2018-09-01T20:15:02Z

• Version of Mapnik, if applicable
  3.0.19

• Tcl version if the problem involves one of the Tcl scripts
  8.6.8

'routeGraphics.tcl' and the stored procedures need to be updated to handle refs on ways. These will need to appear with a dummy 'network' (cannot be the empty string, will likely be a single underscore). They need to be parsed into possible multiple refs, separated by semicolons.

This process needs to happen only on ways that do not participate in route relations. Where there are both route relations and 'ref' tags on the way, the 'ref' on the way is highly likely to be redundant, and by convention will be a less precise identifier (e.g., ON 120 rather than CA:ON:secondary 120).

Rendering will be with customary rounded-rectangle markers, not with custom shields. A posisble enhancement would be to recognize 'I', 'US', the fifty state abbreviations, and the ten Canadian province abbreviations, and use the appropriate shields for such cases, but it is not clear that this is a good idea; it's far from clear that 'CA' or 'I' mightn't be the first part of a 'ref' in a jurisdiction outside North America.

Explicitly out of scope is attempting to handle 'SR' or 'CR' by identifying the jurisdiction in which the way appears.

Right now, Mapnik is driven with a single PNG file for each combination of network, ref and graphic size. This is rather a lot of tiny files. (This technique is inherited from the previous incarnation of osm-shields, which dates to before the time that Mapnik had an SVG interpreter.)

This task can break down into several smaller tasks.

First is to work out the plumbing so that Mapnik can render the shields directly from SVG, and switch to saving SVG instead of PNG. This will cut down on the file count tremendously, since the SVG is resolution-independent.

Second, and harder, would be to work out a way for Mapnik to render directly from template, rather than from SVG. This would mean that the 'shield_graphics' table would contain template and substitution strings, rather than naming a file containing the pre-rendered graphic.

It would be possible to go one step further and let Mapnik do it all - essentially, inventing a 'little language' so that the parsing of network and ref, and the selection of a template and forming of modifiers, would happen entirely contained in Mapnik. (I suppose that the Carto-CSS language could sort of do the job, but that doesn't feel like a very maintainable solution to me.) This last step might be a bridge to far; it's certainly possible to scale up the pipeline to frequent updates of a very large data base without taking it.

Thanks for reviving this project – I hope it succeeds where previous efforts were unsuccessful. Would it be possible to import Phil Gold’s original shield renderer’s commit history? Multiple people contributed to the SVG templates. Although that work is in the public domain and doesn’t technically need to be credit, it would be nice to give those folks credit and potentially involve them in the project going forward.

The current mode of operation of 'routeGraphics.tcl' is to run as a batch, searching the database for route relations for which graphics are available but have not yet been generated.

This takes a while - it's too heavyweight for minutely updates.

There are several possibilities for how to address this.

One possibility would be to generate graphics on demand in 'osm2pgsql' when doing the import. I don't much like this; it's piling too much responsibility into that program.

A better possibility is to trigger running 'routeGraphics' separately when inserting a route for which graphics are not yet available, or to run it as a 'cron' job. This is somewhat more attractive, except that in some systems updating a route immediately triggers re-rendering of tiles in which it appears. There would therefore be a rendered tile with the new route and no shields.

This could be mitigated by having the 'routeGraphics' program also produce a tile expiry file, like the one that comes from 'osm2pgsql', so that the presence of new graphics will redraw the routes that use them. Some sort of functionality like this is needed anyway, so that when new graphics templates become available, the tiles that would benefit from them can be expired.

Some of this would be moot if the downstream pipeline could be altered to allow Mapnik to render directly from template SVG's, but there still needs to be a way to force tile refresh when new templates are provided.

Paul Norman and Sarah Hofmann have expressed serious concern that using stored procedures and complex SQL queries will be too slow at render time. They are fairly adamant that deploying pictorial highway shields at large scale will require precomputing all the necessary data and having the material available in flat (and likely denormalized) tables.

Need to investigate feasibility of this approach, most likely in a postprocessing step after importing the data through the flex backend of osm2pgsql.

An alternative might be to carry out a detailed performance study to establish whether the existing stored procedures are or are not adequate to the task.

The current mewthod of using on the 'slim tables' of osm2pgsql to generate the 'shieldroute' and 'shieldway' tables works well for a workflow that applies daily updates from GeoFabrik (a fairly common case for those who map less than the planet). It obviously cannot scale to minutely updates of a whole planet. Moreover, the developers of 'osm2pgsql' strongly deprecate accessing the slim tables, warning that their content may change without warning.

The SQL operations to create, update, or delete route relations are pretty trivial. The obvious possibility for handling minutely updates (and improving update performance on the less-frequent ones) would be to include an option in 'osm2pgsql' to keep these two tables in sync when the database is first imported and when diffs are applied.

I went through this work at one point for the OSMAnd project; it took forever and a day to compile and assemble the result, but maybe it can help in this project.

I noticed that counties are explicitly named in the graphics grabber, but not all are. I went through the work to assemble 'a list of all counties alphabetically nationwide' since OSMAnd also does (albeit incompletely, I think) shields based on refs. Maybe it can be used as a catchall, or can be used to add to your existing lists, to help put county names into bins.

If not, please feel free to close this!

https://github.com/osmandapp/OsmAnd-tools/pull/135/files

Wisconsin has approximately 350 relations (this number continues to grow) which should all use the same shield pattern, for County Trunk Highways. Each county maintains its own system, and thus the networks are distinct, but the signs they use are consistent across the state. A general specification for the shield follows the list.

The following 72 county names are potentially used in the third field of a network tag, when prefixed with US:WI:
Adams
Ashland
Barron
Bayfield
Brown
Buffalo
Burnett
Calumet
Chippewa
Clark
Columbia
Crawford
Dane
Dodge
Door
Douglas
Dunn
Eau Claire
Florence
Fond du Lac
Forest
Grant
Green
Green Lake
Iowa
Iron
Jackson
Jefferson
Juneau
Kenosha
Kewaunee
La Crosse
Lafayette
Langlade
Lincoln
Manitowoc
Marathon
Marinette
Marquette
Menominee
Milwaukee
Monroe
Oconto
Oneida
Outagamie
Ozaukee
Pepin
Pierce
Polk
Portage
Price
Racine
Richland
Rock
Rusk
Saint Croix
Sauk
Sawyer
Shawano
Sheboygan
Taylor
Trempealeau
Vernon
Vilas
Walworth
Washburn
Washington
Waukesha
Waupaca
Waushara
Winnebago
Wood

Shields for this system uses marker M1-5A, which is outlined here: https://wisconsindot.gov/dtsdManuals/traffic-ops/manuals-and-standards/signplate/mseries/M1-5a.pdf

tclsh8.6 make-routetables.tcl --init is missing from initial-load.sh

This call to action lists several Ohio county and township route networks. Here are SVG templates at Wikimedia Commons that cover these route networks. Each SVG currently has 00 or 000 as the route number, but this text would be substituted for %num%.

In general, you can find Wikimedia Commons shields and shield templates through the modules that power road-related infoboxes at the English Wikipedia.

Previously bug 1025086 at Launchpad.

The make-routetables script reports a bunch of errors creating indexes. Should use CREATE INDEX IF NOT EXIST (or similar) to fix these.

Contributing the SVG files to handle these (with the %num%, %suf% and %mod% substituents as needed), even if no programming is done, would be very much appreciated.

US:WI:Rustic should be able to use the following base SVG as found on Wikipedia Commons under a public domain license:
https://commons.wikimedia.org/wiki/File:Rustic_road.svg

The original this is derived from is standard MR1-99, which appears here: https://wisconsindot.gov/dtsdManuals/traffic-ops/manuals-and-standards/signplate/mseries/MR1-99.pdf

The other key piece of this signage (which would appear below the above shield, see Traffic Engineering, Operations & Safety Manual page 19) is MR4-53, whose standard appears here: https://wisconsindot.gov/dtsdManuals/traffic-ops/manuals-and-standards/signplate/mseries/MR4-53.pdf

With the advent of the 'flex' backend in osm2pgsql, it now appears once more that osm2pgsql is fit for purpose to maintain the tables needed to track membership in route relations for placement of pictorial highway shields. This feature request is being created to track that development.

The idea is that the osm2pgsql style sheet can maintain these tables with additional calls to a new module, shieldtables.lua. This module will allow the following calls:

local shieldtables = require 'shieldtables'
local shieldt = shieldtables.new(prefix)

These will be called in the initialization of the style sheet, to create the object that manages the two new tables.
prefix is the table name prefix; default is planet_osm.

shieldt:process_way(object);

Must be called from osm2pgsql.process_way. In phase 2, creates any needed rows in the shieldway table for ways that bear highway numbers.

shieldt:process_relation(object)

Must be called from osm2pgsql.process_relation. Creates any needed rows in the shieldroute table to describe route relations.

return shieldt:select_relation_members(object)

Must be called from osm2pgsql.select_relation_members. Returns a table with a single key, ways, which in turn is a table of OSM ID's of ways that require phase 2 processing because they participate in route relations.

This appears to be much more in keeping with the usual workflow than would a switch to a different tool such as imposm3.
It also allows keeping a rough equivalent to the current conversion style, by adding the calls above to compatible.lua.

Much more work needs to be done, but forward progress is starting again.

Some signs are too complex and graphics-intensive to render readably at small sizes.
Erie County, NY is probably the worst offender (It's pretty unreadable in the field, as well!)

These will need a simplified representation when rendering on the map. The advantage of matching the sign precisely is lost when the result is too small to read.

Revision 185 introduced a one-off shield for each parkway in the Kentucky parkway system that depicts the signs without the Unbridled Spirit branding but with the fully spelled-out road name that ends up being unreadable at the sizes that this renderer uses. These shields would be much more discernible as the same design but with an initialism of the road name, which can be displayed in larger type. For example, the Bluegrass Parkway would be “BG”, and the Western Kentucky Parkway would be “WK”. I’ve tagged each parkway’s route relation with short_name set to the two-letter initialism that the Kentucky Transportation Cabinet uses for official purposes. This renderer could refactor the various US:KY:Parkway-*.svg files as a single file that contains a variable like the templates for other route networks.

The fundamental design of the processing for shield concurrencies requires that relation member information, at least for route relations, be available at run time.

I sketched out a design for how this might be accomplished, and asked the developers of osm2pgsql to review it before I started coding. They informed me that such a thing was not acceptable to the osm2pgsql project, and could not be made acceptable to it; their view is that no renderer will deal with relations and that all data to be rendered at run time must be on ways.

I've been unable to think of any means by which the rendering of route relation concurrencies can be addressed (particularly in the presence of minutely updates, where relation information will most assuredly be needed!) An alternative to osm2pgsql is therefore needed.

It appears that imposm3 may be suitable. Since several renderings that I already produce will need to be updated and requalified under imposm3, these will have to be converted before work on 'routeGraphics' in that environment can proceed, so this has to be a longer-term task.