osmlab / maproulette

This is a very old version of MapRoulette. Find the current version at osmlab/maproulette3 and the backend at maproulette/maproulette2

Welcome to MapRoulette 1, the powerful & popular bug fixing tool (or is it a game?) for OpenStreetMap.

This a very old version of Maproulette. Modern MapRoulette is now separated into a React based front end application at https://github.com/osmlab/maproulette3 and a Scala based backend at http://github.com/maproulette/maproulette2

This README deals with development related topics only. If you are interested in contributing to OpenStreetMap by fixing some bugs through MapRoulette, just head over to the MapRoulette web site and get started - it should be pretty self explanatory.

That said, read on if you want to contribute to MapRoulette development and are ready to deploy your local instance.

Contributing

Please fork the project and submit pull requests. But before you do, unless your fix is something really minor, please get in touch so we can discuss.

See the code style conventions at https://github.com/osmlab/maproulette/wiki/Code-style-conventions needs updating

Frameworks used

MapRoulette relies heavily on the lightweight Flask web application framework, and some of its extensions, notably Flask-OAuth, Flask-RESTful, Flask-Script, Flask-Runner and Flask-SQLAlchemy. For working with geospatial data, MapRoulette relies on GeoAlchemy2 and Shapely.

Deploying MapRoulette

If you want to deploy your own live MapRoulette server, see the documentation here.

If you want to set up MapRoulette locally for testing and development, and you want to benefit from the built-in debugging features of Flask, follow these steps.

MapRoulette requires Python 3.5 or higher.

Creating new Challenges

Ceating and maintaining a good challenge is a little more complex than just pushing a button. It's not too difficult though. For a gentle introduction, please see this workshop at SOTM EU and these slides to get started.

Contact

Bug and feature requests are best left as an issue right here on Github. For other things, contact [email protected]

MapRoulette now also has its very own Slack community. Invite yourself here!

maproulette's Issues

Write a README

user level for challenge owners

the challenge API should require different userlevel than the other API calls. Challenge contributors should receive a challenge API key or have a flag in their user account that provides them with access.

Create endpoints for reading / writing prefs through OSM API

Move to osmlab

Suggest to move this project over to the OSMLab GitHub org. https://github.com/osmlab/

Rename the repository

maproulette-front suggests there is a separate 'back' but there isn't.

no name residential ways

There are over 3 million no-name residential ways in the (US) data. A big chunk of these are either service roads, they don't exist at all, or they could perhaps be named using TIGER 2012 overlay tiles.

This could be a MR challenge of insane proportions. Could there be ways (hehe) to divide this into smaller challenges? By way length perhaps?

Circular dependency in DB table setup

https://gist.github.com/tmcw/5541688

Create a production rollout fabric script.

Will make it easier to test the code if a new system can be rolled out automatically.

Have support for empty challenges in client

The client needs to support a response by the server that there are no further tasks for it.

Create a stats page

Create unit tests

Create project Wiki

The project does not seem to have the wiki enabled. @emacsen can you enable it please so I can copy over the setup guide to here? There will be more wiki pages in the future, on challenge creation for example.

OAuth authentication

Implement OAuth authentication against OSM

cfg file should not be in git

The cfg file is still in git. It can't stay in the main because it has secrets.

Please remove it and make a .gitignore file so it doesn't come back. I'm looking at you @mvexel :)

We can (and probably should) have a separate repo for this and the fabric configs.

Use db sessions to allow models to create new instances

To make programming easier, it would seem that the best way to support actions like "assigning" a task would be something like

task = Task(...)
osmuser = OSMUser(...)
task.assign(osmuser)

But this requires the Task be able to instantiate an Action object in the models.

It appears this doesn't work because the models don't have access to a db session, which is required for instantiation. (I may be wrong about this, and if so, this issue is invalid).

If I'm right and this is the problem, then we need to modify the models.py such that we pull in the db engine, and then we use that to string our models from, such as in the example:

http://pythonhosted.org/Flask-SQLAlchemy/models.html

utf8/oauth crash

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1701, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1689, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1687, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1344, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Library/Python/2.7/site-packages/flask_oauth.py", line 433, in decorated
    return f(*((data,) + args), **kwargs)
  File "/Users/tmcw/src/maproulette/maproulette/__init__.py", line 155, in oauth_authorized
    data = osm.get('user/details').data
  File "/Library/Python/2.7/site-packages/flask_oauth.py", line 211, in get
    return self.request(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/flask_oauth.py", line 286, in request
    headers=headers))
  File "/Library/Python/2.7/site-packages/flask_oauth.py", line 84, in __init__
    self.data = parse_response(resp, content, strict=True)
  File "/Library/Python/2.7/site-packages/flask_oauth.py", line 48, in parse_response
    return get_etree().fromstring(content.decode(charset))
  File "lxml.etree.pyx", line 2993, in lxml.etree.fromstring (src/lxml/lxml.etree.c:62980)

  File "parser.pxi", line 1607, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:92730)

ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

Implement python-level logging

Rewrite code to get challenge then task

The code current calls getInitialTask, which gets a task, then a challenge.

That call pattern is backwards, and was done mainly for compatibility.

Replace it with a clean call pattern of getting a challenge, then a task.

Create UI branch

@emacsen can you create a UI branch so @tmcw can hack on that bit?

Allow additional files for challenges

Currently, a challenge's help text is set as a text field in the database, but a lot of help text will require some additional images or videos to aid the documentation process.

It would seem reasonable that a challenge author would be able to add these files to their documentation pages.

Design challenge selection interface

What do we do when there are 5, 10, 50 challenges?

Finalize the client routes

API routes are not done:

Challenge calls:
GET:
/api/challenges?difficulty=#&near=##.##,##.##
/api/challenges/<challenge_id>
/api/challenges/<challenge_id>/meta
/api/challenges/<challenge_id>/stats
/api/challenges/<challenge_id>/tasks

(where challenge_id is the slug, not the internal ID)

Task calls:
GET
/api/challenges/<challenge_id>/tasks/<task_id>
/api/challenges/<challenge_id>/tasks?near=...&num=###

POST
/api/challenges/<challenge_id>/tasks/<task_id>

The code in the client needs to be synced up to the api properly as well.

Geocode function has problems with some towns

Some towns show up as "Unidentified". This needs to be identified and fixed.

Badges

Once MR is collecting statistics on a user, it would be easy to create "achievement badges".

Users could select whether or not they want badges.

License

Agree on a license for this project.

Add internationalization/localization

MapRoulette should handle Internationalization and Localization. Here's a brief roadmap of how I imagine this working.

The first step is to translate all the user elements.

This has two parts:

1. Translating all the template data
2. Translating all the user elements

For this, we'd use Flask-Babel (http://pythonhosted.org/Flask-Babel/). Then for the template, we'd use the template translations in Jinja2 (http://jinja.pocoo.org/docs/templates/#i18n-in-templates)

Then we need to translate the other user elements, such as the buttons in the dialog box.

The buttons are controlled in the challenge template in the buttons property (or will, as soon as it's coded up).

The buttons can be translated with the gettext method

The second issue with translations is the issue of how to handle challenges themselves.

This also has a two components.

1. All the challenge data (title, description, help text) need translation.
2. Any per-task instructions.

In my mind, there are a few ways we can handle this, but here's a simple way and a complicated way:

The simple way (probably my preferred option):

We can treat each translation of a challenge as a separate challenge. We'd simply add a field to a challenge with some localization information (such as the language) and then when a user has the selection process, they'd be given a challenge that matched their browser or chosen preferences.

If each challenge is separate, then there's no issue around translation issues.

The complicated way:

We can create a mechanism in the database to handle all these translations. Then we choose the right language at run time for challenge metadata, and per-task instructions (if there are any instructions).

The problem with the complicated way is that it would require a significant change in the database, and a lot of new logic to handle the internationalization handling.

The problem with the simple way is we are going to end up creating duplicates of tasks, N per challenge. This is bad, because in addition to simply producing more work, it also means that tasks that have been set to ignore (ie skipped too many times, or set to cantfix) won't be picked up across translation.

I've been thinking about this problem and we could do something to handle this, where we create a new field in the challenge table which contains a string field, which contains the slug of another challenge. Then the database lookups for tasks uses the specified challenge slug instead of the current one. This would be useful for Review tasks as well as translations, but unfortunately this technique only works if the tasks themselves don't contain textual instructions.

Version control the API

Discussions are leading to some changes in the api which will be necessary for translations.

It probably makes sense to move the relevant API calls to some version controlled method (ie (/api/foo -> /api/1.0/foo) or to somehow specify that the API will not be ensured to be 100% backward compatible between versions.

-front?

A little confused by this being maproulette-front - is this intended to be just a frontend? It has quite a bit of server code.

Blog

Agree on some kind of outlet for news and technical updates for both users and developers. A blog was suggested.

Switch to Javascript

If this is agreed-upon and nobody has large amounts of unpushed changes (/cc @emacsen ) I'll volunteer to do this switch.

Make init support /api/task

/api/task gets an appropriate task and returns it.

The init function should be able to call this function, retrieve the task, as well as set up the sidebar and map appropriately.

move views to separate file

Power lines that intersect with ways

For reasons best known to them, some mappers make power lines intersect with ways where they cross. If this were real, the results would be disastrous. In any case it seems like a good project for MapRoulette.
The is a corollary to ways that intersect with land-use areas, with administrative boundaries, etc., etc.

Port 6000

On macs/chrome this is an unsafe port. 3000 is a more kosher option.

Move session handling to database

Currently session data is stored on the filesystem. This is fine for a single system implementation, but should be moved to a database.

Some help on implementation is available here:

http://flask.pocoo.org/docs/patterns/sqlalchemy/

Though it's incomplete.

Make creating/supporting a new challenge easier

Right now, making a challenge is quite complex.

The challenge author must create a title, description, blurb, help text, introduction text, select a tile server, and even a dialog box, with proscripted actions.

And those actions must match the logic used to determine when a task is done and when it's available.

There should be a way to make this easier for the user, by creating sensible defaults.

And similarly there should be a way to allow the logic behind available and done to change with the type of challenge it is.

Clean way to handle review challenges

Review challenges are going to be a little tricky.

I see two ways of handling them.

The first way is that we specify that a review task is a special kind of challenge, with special rules, such as the ability to inspect another challenge's table.

The second way is that we have duplicate entries for review challenges. The way this would work is we'd need to add some additional API parameters for challenge authors to see what tasks are in what state, download all "complete" tasks, and then use that data to supply the review challenge.

Any task that's then set as "not complete" would need to get re-fed back to the original challenge.

The difference between these two methods of handling review tasks is that in the first method, the logic lives in MR, where in the second, the logic lives outside MR, and there's duplicate challenge data.

My personal view is that it's far easier to keep the challenge logic simple, and coherent if we keep review tasks as entirely separate, and then provide a reference implementation for how we handle them for other challenge authors to copy.

But I'm open to hearing alternative ideas on this.

Next task by challenge should use the current location

Route user to Authentication when performing any action

Users should be authenticated against OSM so we can tightly integrate with OSM preferences through the API and collect home location and preferred editor. Buttons 'skip', 'edit in..', 'not an error' etc should route to authentication if user is not already authenticated.

Localization

osmlab

Want to move this to osmlab, where we're going to keep a few osm-related projects?

Finish call for next task by challenge

Create 'Log in with you OSM account' UI element.

element will call http://client/oauth/authorize

Create scaffolding

i.e. a mock challenge with tasks for unit testing

Prepare for production deployment

decide on a production deployment strategy, see http://flask.pocoo.org/docs/deploying/ for options.

Make challenge selection smarter

The current implementation of challenge selection uses a very simple algorithm to find users a new task to work on.

In psueocode, it's something like

GET A TASK WHICH MATCHES THE USER DIFFICULTY AND THE USER'S LOCATION IS WITHIN.

This is good, but we could improve this a few different ways:

We could request a task count within a certain radius.

We ideally want not only challenges which are in the right place, but which need work in that place. A simple way to determine this would be to select the challenge based on how many tasks are nearby.

We could set a task priority system

Right now, all challenges are equal, but it doesn't have to be that way. Tasks which are of importance to the community might be weighted more heavily.

We could weigh a factor such as how complete/incomplete/active a challenge is.

Ideally we'd want to either support challenges which are gaining momentum, or else balance out tasks which are not getting enough help

Create a geoJSON clear function

Currently, Leaflet does not support a way to clear the geoJSON layer (see Leaflet/Leaflet#1416) so a new function should be created to clear the map in between runs.

Service roads reclassification

The US suburbs are generally filled with access=private highway=service roads that are imported from TIGER. These are prominent on low zooms ( http://www.openstreetmap.org/?lat=38.68482&lon=-77.45203&zoom=17&layers=M ) and need to be retagged for them to be really useful for routing. One task would be to identify this case (short service roads with this tagging scheme, unedited since tiger) and have users retag them into something useful.

Encode resolve button sequence into challenge metadata

The button sequence that appears when the user returns from editing will vary based on challenge content and needs to be encoded into the challenge metadata.

Markdown converter

A lot of text is stored in the database.

That text is stored as Markdown.

Previously, the text was pre-rendered before sending it to the client, but it makes a lot more sense to convert it on the client itself.

Integrate a Markdown converter into the client.