osmlab / to-fix

A task manager for OpenStreetMap

 ____  __       ____  __  _  _
(_  _)/  \  ___(  __)(  )( \/ )
  )( (  O )(___)) _)  )(  )  (
 (__) \__/     (__)  (__)(_/\_)

A task manager for OpenStreetMap.

Coordinate with other users and work down a queue of tasks without getting in each other’s way.

Related tools

• A JOSM plugin is available at tofix-plugin.
• A CLI for creating and updating tasks is available at tofix-cli.

Workflow

This is the frontend component to to-fix. The server component is at to-fix-backend.

This is a React + Redux application bootstrapped with create-react-app.

To start development, you will require node.js to be installed.

npm install && npm start

To build and publish to gh-pages, run

npm deploy

You can configure some details at src/config.

Wiki

• How to create a new to-fix task
• How to work on an existing to-fix tasks
• List of open tasks

to-fix's Issues

"Mispelled" is misspelled

In the sidebar "Mispelled tags" is misspelled.

Request to localhost:8111 generate errors and some edit requests go to the wrong place.

On the main instance of the application (http://osmlab.github.io/to-fix) with Firefox 34 and Firefox 37.
On the task "unconnected_major1"

When a task is loaded, I always got the error “not well-formed” in the console.

When clicking "edit" I always have a "Cross-Origin Request" error

Une demande multi-origines (Cross-Origin Request) a été bloquée :
la politique « Same Origin » ne permet pas de consulter la ressource distante située sur 
http://localhost:8111/load_and_zoom?left=137.083662&right=137.0903798&top=35.1245148&bottom=35.118097&select=node610332909.
Ceci peut être corrigé en déplaçant la ressource sur le même domaine ou en activant CORS.

My browser is in french and the error too but anyway the problem is clearly a request to localhost:8111

After fixing the task when clicking "fixed" I always have the error displayed in the page " error fetching key value 0002-51e49101d93de03af7480bdf33c1288d " (value not the same each time)
And the console displays:
"aucun élément trouvé" which means "no element found"
"syntax error"

Clicking "skip" does not generate this key value error.

Sometimes the circle signaling the error on the way is missing (only the way is shown) when this happen (1 times out of 10) clicking "edit" always send me to this location:
http://openstreetmap.us/iD/release/#background=Bing&map=20.00/-77.02271/38.90085
(no particular error in the console except "not well-formed" as always when loading task)

Tell me if you need more tests!

Persist fixed/* in S3

General sketch:

• Create local fixed directory by running fixed.js
• Get fixed.zip from s3
• Concat the files from both directories that match and correspond to particular errors
• Uniq all the files
• Zip up the combined files
• Upload them to s3
• Version responsibly

Prefill iD changeset comment

https://github.com/openstreetmap/iD/blob/master/API.md#url-parameters

• ?comment

Catch JOSM remote control failures

Need to show some kind of error message and link to http://josm.openstreetmap.de/wiki/Help/Preferences/RemoteControl

Autofix connectivity

The Unconnected major class of errors definitely has some subset that's automatically fixable. We should figure out 'automatic fix' recommendations and present them as options alongside iD + JOSM editing.

Error in Tag: highway=*_link

I'm working with One-way without exit, the roads are correct, and they don't have oneway tag, but the program give me as error. in all tags highway=primary_link and others highway=*_link

https://www.openstreetmap.org/way/250705314/history

Sharing an error

Is it possible to get a permalink for the error to share an issue with others?
Or even a link to view the area on osm.org would help.

New data source for inconsistent street names

A common problem with OSM data for routing is, that street names are not consistent. This is a major problem for generating routing instructions.

There are two major error cases of this sort:

• Small unnamed street segment between street segments of the same name
• Inconsistently tagged segments, alternating between having only name, only ref or name and ref tags.

What would be a good data format to feed into to-fix?

Would a CSV file with triplets of osm way ids work, or should I also include geometry information?

Generating this data should not be too hard, I could give it a try on the weekend.

Streams don't need exits

Upstream, but I don't know which source

missing error type ldb errors out of script

Check for existence of leveldb database; if not present, probably best to 404.

Increase the bbox on download in JOSM

is possible to increase the bbox on download in JOSM, now is download exactly to way, could be good if we add few meters the bbox like this.

Skip 410 Gone ids

Most of our errors reference an OSM id, we take the id, make a request against the OSM api and get the actual geometry. Sometimes the id that we had has since been deleted. Right now the map just shows nothing in those cases but we should automatically skip to the next error in those cases.

Should also keep in mind that most items have multiple geometries, we only need to skip if both are missing and fallback to an alternate (#37) if only one is missing.

Handle missing data

OSMI is down right now, which would be bad to update with.

So:

• be able to handle empty tables
• only sync if there is content

Basically, don't wipe out good data.

Datastore notes

This has been a mess, some notes:

• Initial idea: everything in S3 with a micro server. Server requests map to simple s3cmd commands
  • Importing to S3 turned out to be very slow for ~500k objects, each only a few bytes. Bad idea; too much hammer not enough nail.
• Parse.com, import a csv, grab a random object.
  • No random query, would love to use Parse or Firebase to host this though
• Current idea: each user gets assigned to a namespace and only grabs the next object from their namespace, hopefully greatly reducing the chance of clashing (ie multiple users editing the same thing at the same time which would happen with a single namespace).
  • What happens when one namespace runs out of errors? or when they all start running low and everyone is clashing? Assign multiple namespaces (5) to a user and cycle through them? How do we ensure even distribution?
  • Do we need a checkout system again? Very much want to avoid.
  • Maybe rotate through namespaces by considering userid + time? Each request from each user would hopefully hit a different namespace every time. Not sure what happens when we approach zero.
• I also did a redis thing; generate the csv, import it to redis, deal with it there, but I don't want to maintain an instance and it feels limited.
• Fallback: RDS and a micro instance.

Stopgap for today, run directly off the already running Postgres. I hate it, but I need to get away from this for now. I'll figure out namespacing later.

Don't use port 3000

Yay, at Peace Corp HQ. Ohh, can't demo to-fix because that port is blocked. :(

Datastore issues

Just a bit of a brain dump on what I've been having trouble with.

Redis worked well:

• Fast
• simple hashing
• good tools/libraries

but:

• no simple mass deletion, literally have to loop through your db and delete, redis crashed on me, I lost everything and had to rebuild
• ran out of memory, could do a larger instance but it's only a matter of time
• I could store values in something like leveldb, but this is adding complexity quickly, I don't want to maintain those things, should go with a hosted solution to make development simpler as well

This weekend:

• Tried mongodb
  • it's ok, not totally thrilled about it, I could get it working, but I'd have to maintain it, etc...
• Tried dynamodb
  • would work but it's complicated and hacky, not using as intended
  • have to worry about iops, especially on initial import

This is one of those things I think a relational db is better at. Going to try and get something up quickly on RDS and see what happens, if not, dynamodb it is. Added advantage of RDS, importing CSVs, which would be simple enough.

Edit specific area

Do you think the ability to choose one's editing area in to-fix would be within scope or beneficial?

Define differences from MapRoulette

MapRoulette is an established project with an interface for adding challenges, also hosted on osmlab. The readme for to-fix should state how it differs from maproulette.

Better yet would be to add any missing functionality into the existing project.

List available tasks

Just need a simple endpoint for listing available stuff. This will be used for listing stuff on the front end, manually. Getting this list manually is tedious.

Unique id

Currently we don't unique identify anything, just grab a random thing and have no way of tracking it. There's value in being able to identify each error uniquely, mostly that we could track specific items that are problematic and need reviewing.

Right now we key things for leveldb here:

to-fix/import-csv.js

Lines 32 to 35 in dd61094

	db.put(count, JSON.stringify(data), function (err) {
	if (err) console.log('-- error --', err);
	});
	count++;

It's just an incrementing integer. We should replace it with an md5 of the value string. Simultaneously this will necessitate changes in index.js here:

to-fix/index.js

Line 78 in dd61094

var newKey = (+new Date() + lockperiod).toString() + Math.random().toString().slice(1, 4);

cc @sbma44

Wrong mixedlayer on covered=colonnade + tunnel=building_passage

This is wrong "fix": http://www.openstreetmap.org/changeset/27025584
Pathways are on the same layer (photo proof: http://mosday.ru/photos/61_184.jpg ), so they should not be disconnected.
Such cases should not be fixed automatically.

Script updating database

Just had to do this manually, not a big deal but it should be scripted.

Take load.sh, rm existing directories and the exact ldb that is being loaded. Hopefully quick enough to avoid downtime. Not going to prioritize a seamless rotation right now.

Ability to logout

Right now someone has to manually remove localstorage entries.

Server/client/data split

Right now this repo contains:

• Data: Scripts for setting up databases to ingest keepright, osmi, missing tiger data and export a csv
• Server: A node server backed by redis for accepting csvs and serving tasks
• Client: A UI for presenting tasks

These things should be split up somehow, especially the data side. It's confusing to think about "the server" or "the database" because there are multiple possibilities.

This isn't a priority but something I'm keeping in mind to make the entire thing simpler and easier to understand.

add tests

It sucks that I'm pestering @aaronlidman every time I want to commit code. let's create some simple tests for data import and prioritization.

Documentation

Needs docs for

• Keyboard shortcuts for to-fix
• Loading tasks
• Writing loaders

Loader

Each error type will have a loader which sets up the map and populates the UI. Only requirement is a unique name, unique id will be provided.

Fallback to iD

Idea from Eric: fallback to opening in iD if JOSM fails. This avoids the alert, presents clear actions, keeps the UI minimal (no second button).

Goal for usability in adding a new error source

• Provide a csv of bugs/errors
• Write a loader
  • To parse the csv row and put stuff on the map and UI

I like csv for now, we'll see if it doesn't work out, maybe json.

Show description of issue type

A little note (even in the title attribute) would make things a little more user friendly:

Script redis setup

I've been doing this manually and just backgrounding it, which is wrong.

Ref: http://redis.io/topics/quickstart "Installing Redis more properly"

Just had my first crash because I tried to mass delete 15M keys in a really hacky way, lost the database, able to recreate in a couple minutes but redis should be run as a service.

Add sentry error tracking

Specify loader by querystring

Tasks are hardcoded to their loaders, let's make it a little more flexible with an optional querystring. Hardcoding is only needed for the UI.

It used to work like this, not sure why I removed it, probably to keep the url simple.

Todo

• better JOSM bounds
  • nodes: too small, use map bounds when it's a node
  • ways: buffer a little, give some context
• OSM tilelayer
• Satellite (Bing & Mapbox) tilelayers
• Figure out datastore
• User tracking
• iD integration
  • iframe? fork (iD minus)? link?
• Automate source updates
  • is the implied tracking idea possible?

Encourage small changeset area and meaningful changeset names

Here's an example "to-fix" changeset:

http://www.openstreetmap.org/changeset/24436746

The description, "http://osmlab.github.io/to-fix/?error=keepright/nonclosedways" is not especially meaningful. What can be done to encourage mappers to use meaningful changeset names?

Similarly, the area of that is essentially worldwide - it's not easy to see what's changed and where.

Link to this repo for bug reports

I spent too much time (from my view that is) to find this github repo to post a bug report. Could you please add a link to it or did I just not find it?

Display remaining errors to fix in current category

That could discourage the user if the number is too high.

Maybe remember number of errors when the error category was added and make a progress bar ?

Error sources

source	# of errors	notes
Keepright	26,672,285	only a fraction of these are useful, maybe 3M, a breakdown
OSM Inspector	200k - 1M	very useful, downloading requires work, some are too large and time out
OSM Notes	192,811	not in yet

General info display

Have a variable that will set the content of a basic info div. The immediate use case is the TIGER stuff, with #name, but there will be more. Something very basic, like:

{
    name: 'Some street name',
    description: 'do this and that and this',
    number: 'some number of significance'
}

This will then make a basic table or something with all that info displayed in a pretty way. Like how osmly did tags but it doesn't need to be editable.

Sidebar doesn't scroll on small screens

Tricky height stuff with the different sidebar sections

Walkthrough

Like iD and Mapbox Studio, highlighting features, how to use the thing, quick overview in 4 or 5 pages, animated gifs if needed, etc...

what is this for?

Watching this new repo with some interest. What is your objective with it?

Tracking dbs aren't created

Line 105

Templatify the UI

Logic and UI are all intermingled, UI should be in simple templates.

Use some lib EJS, Mustache, Handlebars.

Doesn't work in Chrome or Firefox with HTTPS

You're loading unsecure content. Use protocol agnostic references in your HTML.

Keyboard shortcuts

Let user choose the editor

It would be nice to be able to choose the editor in which tasks are edited. JOSM, iD, other.

I had some trouble understanding what's happening when the edit button didn't work. It turned out the tasks were opening in JOSM which I had left open, and not in iD which I had used previously.

Add minimap

Bottom right corner

Fallback to way id if node id is gone

Nodes seem to go missing more often than ways, so the ref we have is outdated. This means iD can't open the right thing and instead shows a blank spot in DC. We should fallback to way ids in those cases, highlighting the slightly wrong thing is better than nothing.