hotosm / osm-analytics

OSM Analytics lets you interactively analyze how specific OpenStreetMap features are mapped in a specific region.

osm-analytics: data analysis tool frontend

OSM-Analytics lets you analyse interactively how specific OpenStreetMap features are mapped in a specific region.

Say, you'd like to know when most of a specific feature type (e.g. buildings) in a specific country or city were added. This tool lets you select the geographical region of interest and shows a graph of the mapping activity in the region. You can even select a specific time interval to get the number of touched features in that period, and the map will highlight the matching features. Alternatively, one can view the distribution of features by their mapper's user experience. The tool also gives a side by side comparison of the map state at different points in time and lets you view which HOT projects may have included the mapping of a region.

Features

• supported feature types: buildings (any closed osm way with a building tag), roads (any osm way with a highway tag), rivers (any osm way with a waterway tag)
• graphs of feature recency or mapper experience
• highlighting of features by custom date range or user experience interval
• calculated statistics: total number/length of features in selected region and date/experience range, number of contributors
• shows which hot projects influenced the mapping of the selected region
• compare map at different points in time
• data updated daily

Technical Overview & Limitations

See documentation/architecture.md for background information.

Installation and Usage

The frontend is implemented in React/Redux and based on tj/Frontend Boilerplate.

Install dependencies:

$ npm install

Run in development mode:

$ npm start

Generate static build:

$ npm run build

The deploy.sh script can be useful to publish updates on github-pages.

Embedding

This user interface supports a custom UI for embedding on 3rd party websites, using an HTML iframe. It allows generating a time comparison between two points in time for the same region.

The above visualization can be generated using a specific URL structure:

https://osm-analytics.org/#/compare/<region>/<start_year>...<end_year>/<feature_layer>/embed/<theme_name>

• iframe_base_url (http://osm-analytics.org)
• region the area of interest the embedded map is shown for. Can be a bounding box (bbox:110.28050,-7.02687,110.48513,-6.94219), an encoded polyline of a polygon (e.g. polygon:ifv%7BDndwkBx%60%40aYwQev%40sHkPuf%40ss%40%7BfA_%40uq%40xdCn%7D%40%5E)), or a hot project id (e.g. hot:4053) or a link to a github gist that contains a polygon.geojson file (e.g. gist:36ea172ef996a44d36a554383d5fb4fa).
• start_year (2016) represents the start year of an OpenDRI project
• end_year (now) represents the end year of an OpenDRI project. now can also be provided to compare with latest OSM data
• feature_layer (buildings) compare buildings, highways or waterways
• theme_name (default) use the default OSM Analytics visual style, or the opendri theme

The gap detection view can also be used as an embedded map in a very similar way:

https://osm-analytics.org/#/gaps/<region>/buildings-vs-ghs/embed/<theme_name>

The edit recency and user experience views can also be embedded like this:

https://osm-analytics.org/#/show/<region>/<feature_layer>/embed/<theme_name>/recency or https://osm-analytics.org/#/show/<region>/<feature_layer>/embed/<theme_name>/experience

Here, one can optionally supply a time or user experience selection, which triggers highlights respective features or regions on the map that fall into the given time period or user experience range. Just append the respective query parameter to the embed URL: /?timeSelection=<timestamp_from>,<timestamp_to> (timestamps are seconds since epoch) or /?experienceSelection=<experience_from>,<experience_to> (experience values as defined in the respective layer's experience field).

See Also

• lukasmartinelli/osm-activity – similar to OSM-Analytics, but with a simpler, more basic user interface
• OSMatrix by GIScience Heidelberg – "precursor" of OSM-Analytics
• joto/taginfo – aggregate statistics for all OSM tags
• tyrasd/taghistory – tagging history for all OSM tags
• Crowdlens by Sterling Quinn – prototype of interactive mapping history visualization
• mapbox/osm-analysis-collab – misc OSM contributor analyses based on osm-qa-tiles data

osm-analytics's Issues

Identify under-mapped areas: Include population analysis function

Overlaying population density maps from other sources could help to identify under-mapped areas in OSM. We had discussed about this function during initial brainstorming of OSM Analytics, but the function was not selected for implementation during the prototype phase. There are also limited population spatial datasets that are compatible with the OSM ODBL license.

@SimonbJohnson made an excellent implementation of this idea, using the WorldPop dataset and buildings from OSM:

https://medium.com/@Simon_B_Johnson/where-are-the-maps-missing-b22ceedb26f3#.7a18xtxu2

https://github.com/SimonbJohnson/missingmaps_whatsmissing

It appears that WorldPop data is licensed under CC-BY which would not make it compatible with OSM. It may be argued that the data is not used directly for tracing or importing into OSM, but only to identify under-mapped areas. Maybe @pnorman @simonpoole could comment whether this may escape the derivative definition or not ;)

Also worth mentioning that another higher resolution population dataset may be released under an open license towards the end of 2016

https://code.facebook.com/posts/1676452492623525/connecting-the-world-with-better-maps/

http://www.digitalglobeblog.com/2016/02/22/helping-facebook-connect-the-world-with-deep-learning/

Short URL, permalink and SEO

Some features that would be nice to implement for sharability:

• URL shortener (a la osm.org/go/..)
• permalinks that work with filters, to make it easier for anyone to share specific data views
• nicer URLs with semantically structured parameters (e.g. http://osm-analytics.org/#/show/ecuador/buildings/recency instead of http://osm-analytics.org/#/show/polygon:lwmnM~zbKei%7D%40uslAo%60k%40syzBi%7Bd%40fcB%7CtG%7DqrAvbfAk%7BmAcu%60Ae~%40zifBkcb%40%60naBufgAryv%40%60rf%40vrvDctb%40tu_%40uwsAvd_B%7DeSn%60Vwsl%40jndByfs%40rmnC%7DwDnhwBx%7C%7DAjesEvtcLfpO%60%7DpLksyEduF%7DkLdpsAnyv%40nrXza%40%60dwA%7Bvs%40ey%5BidjA%60~d%40ad%60%40%7B%7DHia%60Ab~~Aa_q%40zoNyghA%7DgvAej%7C%40akfF_oe%40%7C%60Mom%7B%40gjvAu%7B~EiulA/buildings/recency)

geocoding: autocomplete for osm search

Autocomplete should also work for arbitrary region search (like for the hot projects). There are a couple of osm-based search engines (photon, mapzen, mapbox) with autocomplete support, but none of these provides the full geometry of the search results (e.g. an OSM-object-id to fetch the full geometry from). This currently leaves only nominatim, which on the other hand sadly doesn't support autocomplete.

Hints/Ideas for how to tackle this are welcome! 🙇

Add option to switch between km and miles in results panel

...for our imperial unit fans :)

Maybe we can also add other visual comparison "units" like distance between Earth and the moon, etc

About Page

• inform limitations of stats (only most recent edits, reduced counts at lower zoom levels)
• more details on the project
• link github repo(s)

Add "Last Update"

Add a note, either in the results tab or directly on the map, which indicates when was the data last updated.

Add HOT projects overlay

As discussed, there should be an option to show an overlay of all HOT projects. It could be the simplified geometry version to start, then using vector tiles or some sort of heat-map rendering (at low Z levels), should be the actual geometry resolution.

The layer on/off switch could be positioned in the results bar, near the "HOT Projects" text.

Add logos of contributors and supporters

Probably in the about page somewhere:

• HOT
• OSM (for OSM community)?
• Dev Seed
• Mapbox
• American Red Cross
• GFDRR
• Knight Foundation

Anyone missing?

Front page, heat map based on world wide intensity of recent edits

this would be sweet and engaging to all sorts of people from mapathons to activation coordination.

Improve error handling/feedback when OSM boundaries for places are broken

Trying Gampaha, Batticaloa or Mnmunai North in Sri Lanka... they used to work but now they don't. This could be because of something wrong with their boundary relation/geometry in the OSM database. The search bar function should either warn of that issue or override with another more stable boundary source.

buildings highlighted are only ways, not multipolygon relations

Buildings found on the map are just the ways having building=_, while _relations* having building=* and type=multipolygon (typically buildings having an inner courtyard) are not shown (example: http://www.openstreetmap.org/relation/1120279)

Use simple basemap

the basemap shouldn't distract from the actual content on the map.

UI improvements

• hover on number of HOT projects in stats bar
• missing x in HOT projects modal
• fix map auto-centering when manually tweaking region boundary
• show currently selected date range somewhere (see mockups)
• improve color scheme, e.g. choose better highlight color (not #F00)
• …

//cc @felskia

when highlighting time-frame/user-experience, style features differently inside/outside the outline area

Custom styling of density layers

Allow the user to select number of classes, breaks and color styling for map feature density layers.

View and download user summary statistics

For any selected AOI and filters set, to have a popup window showing the top 10 (maybe 100?) mappers and their number of edits. Example:

Username	Buildings	Roads
User1	1000	50
User2	500	200
...	...	...

I feel this may be straight forward at feature zoom level, but become a bit more complicated when data is being aggregated at lower zoom levels.

geocoding improvements

• add hint text and/or tooltip that arbitrary search terms should work as well
• add visual indication (e.g. a spinner) when search is in progress
• add visual indication when search has failed
• trigger osm feature search when clicking on search icon

Error because of OSM region simplification?

Searching for San Diego throws the following error:

bundle.js:46309 error during osm region search: j…s.e…r.TopologyError {name: "TopologyError", message: "found non-noded intersection between LINESTRING ( …15 ) [ (-116.95579819155904, 33.09357074019966) ]"}

The San Diego city outline is found and created, but it looks like the simplification method creates a topological error with the boundary line self-intersecting:

That is most likely created because of the complex city boundary in that area:

Maybe it's possible to adjust the amount of simplification factor that OSMA performs when creating these regions to try to avoid these issues, which seem to be quite frequent for cities (e.g. Los Angeles and Santa Barbara have the same problem)?

Provide download option for selected data and summary statistics

This is something we originally discussed and would be great to have, allowing download in CSV or GeoJSON format:

• Full data download within selected AOI and time period (this may work with a call to the Export Tools API)
• Summary statistic for selected AOI, aggregated by year or other property (e.g. building type, etc)

normalize user experience metric of different feature types

user experience for buildings isn't the same as user experience for roads, etc.

numbers would have to be scaled/adjusted to be put on the same scale when multiple map features are active (e.g. buildings and roads).

follows #7

OSM Analytics Next Steps

A tentative summary of the Aug 18 community meeting on Gitter. Please add anything that I might have missed or any other idea in the comments. Thanks!

Priorities / Top Ideas

• Include more map tags/features for processing and display
• Improve OSMA Cruncher efficiency
• Find alternate hosting of processed tiles products
• Thematic analysis, for example to measure progress of mapping projects in achieving SDGs for specific areas (e.g. spatial distribution of schools, financial services, or drinking water points)
• Integrate @jenningsanderson's work http://mapbox.github.io/osm-analysis-collab/
• improve feature summaries for defined AOIs to be more accurate . Compare @sabas's osm-estratti vs. buildings and km of roads for the same area (Liguria) in OSM Analytics.

Tasks

• @tyrasd to host a code walk-through with interested developers and complete #44
• @rodowi to explore methods for parallelizing cruncher processing and send PR
• @jenningsanderson to work with MapBox team to integrate osm-analysis-collab
• @cgiovando draft work plan for development of thematic analysis and additional map features
• @sabas to research methods for improving summaries accuracy, send PR
• @lukasmartinelli document steps on how to submit PR for adding features to OSMA cruncher

Other Ideas

• option to switch background layer (e.g. imagery or other OSM renderings) with opacity controls
• Add functions to identify gaps in map by use of machine learning and satellite imagery. See #43
• Develop an alert mechanism that checks for any processing issues. See #49
• Integrate @lukasmartinelli's osm-activity https://github.com/lukasmartinelli/osm-activity
• Simple drop-down select tool to allow authenticated users to add data processing for any tag, similar to how Export Tools work (e.g. house numbers, road types, etc)
• Improve the visualization of shadowed edits (80% of user edits are shadowed according to @jenningsanderson). Also http://mapbox.github.io/osm-analysis-collab/shadowed-edits
• Develop documentation and demos (videos? dedicated docs? overlaid tips?)
• Develop a visual/guided query builder for the less experienced users
• Increase resolution of density grid to be more detailed a lower Z levels. See #21. Also consider @lukasmartinelli's approach of using points at low zoom levels and then switching to polygons at higher levels.
• find options to have a more detailed before/after history. Maybe having monthly snapshots in addition to yearly (as of now) to start.
• consider the idea of running on-demand crunching jobs for smaller AOIs to increase frequency or specific requests by users, without having to run the entire planet.
• Integration with other OSM tools (to-fix, MapRoulette, etc) to allow users quick editing/fixing capabilities when they notice some issues. See #38
• link OSMA and Export Tools to allow users to download data of areas that are analyzing
• improve region simplification. See #51
• Load region/AOI from URL in GeoJSON format. See #6

Add visitor analytics

Piwik?

Add beta / prototype note?

Shall we warn users that this an initial prototype release and things may not always work as expected? A "beta" note somewhere?

archived OSM TM jobs don't show up

even after a job is archived on the OSM TM, we want to get stats on it

Add Y axis legend / scale for results

Would be good to have a dynamic Y-axis legend to show the range of values in results panel.

Create statistics by an array of HOT-TM projects

For Disaster or in general mapping projects will be very useful to create queries with a lists of mapping projects, so have a fast state of elements traced.

user experience: axis labling

remove numbers, add descriptive indicators. also: fix axis extent

Add settlements (residential areas) as a feature

For lots of Missing Maps projects, we ask mappers to identify settlements first. This is to make a two/three phase approach to mapping more efficient, but it also provides NGOs with a lot of good information.

Could residential areas be included with buildings and roads in the feature options?

time bar: limit to predefined min/max values when zooming out

that should make it easier to reset the view after zooming in

add map legend

Aggregated key/tag statistics by admin areas

Originally suggested by @TylerRadford - would be a useful function to have statistics about different OSM features aggregated by admin areas (country, province, city, etc). Here's an example for miles of road per country or with distribution/density visualizations similar to how it's done in tagInfo:

https://taginfo.openstreetmap.org/keys/atm#map

BTW, that map actually already shows really well the impact of this recent HOT project in Uganda 😉

move "overlay"/variable selection to stats bar

the selection between recency of edits and editor experience should be in the stats bar instead of the current button on map

+/- zoom controls disappear in compare mode

Improve map feature color palette

@felskia wrote:

The orange stands out a bit too much though, it's so bright it feels like the color you should be paying attention to instead of the blue. My recommendation would be:

Highlighted Buildings: Light blue
Non-highlighted Buildings: Dark Blue

but:

The issue with using light/dark of the same hue is that these are already in use to indicate different levels of data density at the lower-zoom levels where features are aggregated into cells. Tricky one. Let's continue this in #21

Add POI layer

Also mentioned in #14 - to add another layer in addition to roads and buildings density. Assuming we can pull this from the same OSM QA Tile source.

As requested by @TylerRadford - this would be really useful to support some ongoing HOT projects.

@tyrasd - let us know if this is something quick that you can add for the beta launch this week.

Thanks!

add info button to stats bar -> human friendly explaination of stats

Visual diagram of OSM Analytics architechture

Add a simple diagram to the readme describing the overall app architecture and interactions with external tools/APIs.

Something similar to hotosm/osm-analytics-cruncher#5 for the backend cruncher workflow.

Update script stopped running?

Hallo! I just wanted to see how mapping for Sri Lanka was going, but saw that neither of projects 1914/1915 were listed in the search dropdown (when I search by name or ID), and manual URL hacking didn't reveal them either:
http://osm-analytics.org/#/show/hot:1914/buildings/recency

Most likely they're not yet indexed. Is there a delay for new projects? They've both been launched in the last 2-3 days.

(I love OSM Analytics -- a great project.)

make date selection slider more obvious

couple of options:

• show date slider all the time and make date bars drap-droppable instead of current click-drag
  • like google finance
  • could be more touch-friendly
  • what should the initial state of the time selection be? last month/year?
  • check if technically doable with current graphing library (vega)

or just

• add visual hints and/or help text to explain that the time bar is selectable and zoomable

//cc @felskia

Monitoring and alerts

Add a monitoring function that routinely checks for data sources and tile processing status and sends out sysadmin email alerts in case something is not working as expected. This could further be enhanced by automatic fall over options to backup data sources.

make bars in stats graph hoverable

…display exact date and number of edits at the specific day. Similarly also for the experience graph.

load region from URL in geojson format

add geojson:… url parameter mode for easy linking from external tools

GeoJSON AOI input

Allow ability to input custom area of interest that may not coincide with an existing administrative area

add an "x" close button to the bottom bar

It would be good to add an "x" close button to the bottom bar (used for comparison). I discovered that the only way to close that bar and come back to the map is to click on "Analysis Map" on the top bar, which is not intuitive at all.

add more map feature types

• waterways
• POIs in general -> #30
• custom specific POIs (maybe dynamically loaded from Overpass or something)?

low zoom levels: use aggregated data view

like in the first prototype or devseed/worldpop for example

use opacity to represent feature density

for feature highlighting: use quantiles (e.g. 20%-80%) to determine date range an aggregated tile should be highlighted in

Ecuador search doesn't work

Not sure if a problem with nominatim or actual boundary map feature.... Earlier today searching "Ecuador" would find it correctly (as for any other country), but right now it does not work.

before/after view

invert region-selection polygon

-> shade outside regions to give focus to the actual region of interest

also, workaround for #2

OSM Analytics not launching

spinning!

52.207.244.74 is unreachable, at least one problem

@dodobas @tyrasd