streetcomplete / sc-statistics-service

Statistics generation service for StreetComplete

StreetComplete is an easy to use editor of OpenStreetMap data available for Android. It can be used without any OpenStreetMap-specific knowledge. It asks simple questions, with answers directly used to edit and improve OpenStreetMap data. The app is aimed at users who do not know anything about OSM tagging schemes but still want to contribute to OpenStreetMap.

StreetComplete automatically looks for nearby places where a survey is needed and shows them as quest markers on its map. Each of these quests can then be solved on site by answering a simple question. For example, tapping on a marker may show the question "What is the name of this road?", with a text field to answer it. More examples are shown in the screenshots below.

The user's answer is automatically processed and uploaded directly into the OSM database. Edits are done in meaningful changesets using the user's OSM account. Since the app is meant to be used on a survey, it can be used offline and is economic with data usage.

To make the app easy to use, quests are limited to those answerable by asking simple questions.

• See the latest release notes.

Screenshots

Download

Quests

There are quite a few different quest types now and more will be added over time. You can see a community-managed list of all quests in the OSM wiki.

FAQ

You can find a list of frequently asked questions in the wiki.

Contributing

This is an active open-source project, so you can get involved in it easily! You can do so without any programming or OpenStreetMap knowledge! Just choose a task that you like.

Here are a few things you can do:

• 🐛 Test and report issues
• 📃 Translate the app into your language
• 🕵️ Solve notes left by StreetComplete users
• 💡 Suggest new quests, or, even better, implement them.
• ➕ and more…

Also, if you like StreetComplete, spread the word! ❤️

License

This software is released under the terms of the GNU General Public License.

Sponsors

Many users are currently supporting this app through GitHub sponsors, Liberapay and Patreon. If you like the app, you can join them ☺️ to support the continued development and maintenance of the app.

Within the frame of Prototype Fund round 15 (March 2024 to August 2024), the German Federal Ministry of Education and Research currently sponsors Tobias Zwick to work on StreetComplete for iOS.
Development on this app was also sponsored in round 8 (September 2020 to February 2021) of the Prototype fund, with focus on collecting more data points and on general improvements of this app.

Since mid 2020, JawgMaps provides their vector map tiles service to StreetComplete for free, i.e. the background map displayed in the app.

Past Sponsors

The NLnet foundation sponsored development on this app in three individual grants with funds from the European Commission:
Two grants given to Mateusz Konieczny in 2019 and 2021 enabled him to work on StreetComplete for about one year in total, with a focus on clearer UI and improvements on data collection. Furthermore, yet another grant enabled Tobias Zwick to work about five months in 2021/2022 on - most notably - the overlays functionality and measuring with AR.

In August 2020, the OpenStreetMap foundation funded the development of Map Maintenance with StreetComplete within the frame of the microgrants program.

sc-statistics-service's Issues

Rate Limit request for new Changesets from the OSM API

I would add a check for the last update in classes/ChangesetsWalker.class.php Line 51 that the last update for the user is at least some time ago. I think something between 30 seconds and an hour would be good.

This would require to also set the last_update timestamp in the changesetsWalkerStateDao on an update with no new Changesets in Line 58 (maybe generally a good Idea to see then the last check was)

update_users.php crashes when executed for user which become deleted

See for example

php update_users.php 4909211

I plan to add some more graceful handling of that problem, if possible:

• error with an explanation
• specific error code

Unless there is some good reason to avoid doing this.

Possible XML External Entity Injection

In the following location, untrusted XML is loaded via simplexml_load_string:

https://github.com/westnordost/sc-statistics-service/blob/85d3ad2f585123dc554923928df58f0f15951e35/ChangesetsParser.class.php#L10

Processing untrusted XML could possibly introduce a XML External Entity Injection vulnerability where files from the server could be exfiltrated. In theory, PHP even allows executing code this way.

While it is possible that the default configuration on your system is safe, it is not guaranteed. Hence it is recommended to explicitly set libxml_disable_entity_loader(true); beforehand.

Uncaught Error: Class 'ChangesetModifiedElementsParser' not found

When fetching http://localhost:8080/get_statistics.php?user_id=<USER_ID> I'm receiving the following error:

Fatal error: Uncaught Error: Class 'ChangesetModifiedElementsParser' not found in /workspace/classes/ChangesetElementIdsFetcher.class.php:19
Stack trace:
#0 /workspace/classes/ChangesetAnalyzer.class.php(17): ChangesetElementIdsFetcher->__construct(NULL, NULL)
#1 /workspace/classes/ChangesetsWalker.class.php(36): ChangesetAnalyzer->__construct(Object(mysqli), 'mariadb', NULL, NULL)
#2 /workspace/get_statistics.php(31): ChangesetsWalker->__construct(Object(mysqli), 'mariadb', NULL, NULL) #3 {main} thrown in /workspace/classes/ChangesetElementIdsFetcher.class.php on line 19

I then replaced ChangesetModifiedElementsParser with ChangesetElementIdsParser, but received the next error:

Fatal error: Uncaught TypeError: Argument 1 passed to ChangesetElementIdsParser::parseElements() must be of the type array, object given, called in /workspace/classes/ChangesetElementIdsParser.class.php on line 16 and defined in /workspace/classes/ChangesetElementIdsParser.class.php:23

The environment I used is outlined in #7.

I suspect that the files in this repository are not up to date with what is actually deployed.

Unclear "PHP Fatal error: Uncaught Exception: OSM API returned error code 0"

I have not yet researched this, I am planning to do this. But if someone knows what is going on here then - please let me know.

https://github.com/streetcomplete/sc-statistics-service/blob/master/classes/ChangesetModifiedElementsFetcher.class.php#L30

It could be related to not providing OSM user login

    const OSM_API_USER = null;
    const OSM_API_PASS = null;

which may result in rate limiting or blocking access (but at least initially is not)

mateusz@mateusz-virtualbox:~/Desktop/sc_statistics_service_development/sc-statistics-service$ php update_users.php 1722488
analyzing...
1722488
PHP Fatal error:  Uncaught Exception: OSM API returned error code 0 in /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetModifiedElementsFetcher.class.php:30
Stack trace:
#0 /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetAnalyzer.class.php(45): ChangesetModifiedElementsFetcher->fetch()
#1 /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetAnalyzer.class.php(23): ChangesetAnalyzer->getSolvedQuestsCountOfChangeset()
#2 /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetsWalker.class.php(115): ChangesetAnalyzer->analyze()
#3 /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetsWalker.class.php(75): ChangesetsWalker->analyzeChangesets()
#4 /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/update_users.php(30): ChangesetsWalker->analyzeUser()
#5 {main}
  thrown in /home/mateusz/Desktop/sc_statistics_service_development/sc-statistics-service/classes/ChangesetModifiedElementsFetcher.class.php on line 30

EDIT: trying https://stackoverflow.com/a/10228081/4130619

Concurent Requests for same User

At the moment it is possible that the code is concurrently updating the same user in parallel in response to either multiple requests with the same user or by running a request and the cron update at the same time.

I think nothing bad happens from it beside unnecessary requests to the OSM API.

It could be prevented by usage of transactions over the whole process for each user and SELECT FOR UPDATE on the changesets_walker_state entry for the user.

Consider editing repository setings to remove "Packages" section

"Packages No packages published" is displayed right now, fortunately this pointless section can be removed.

Edit repo page config to remove it (cog next to the description).

I am not making a PR as it is defined in proprietary github settings, not in a git repository - and I have no rights to modify repo settings.

Stats are incorrect

I'm pretty sure having 10 000 pts should place me higher than last place

Versions affected
Android 12 w/ StreetComplete 42.0

.htaccess file mentioned in README missing

README.md says:

Perform appropriate file protection measures if included .htaccess file is not used

I do not see such a .htaccess file. Don't know if you want to add one or remove that statement from README.md. In the latter case, protecting files should be handled differently.

Besides, and perhaps related:

I'm not that happy with

https://github.com/westnordost/sc-statistics-service/blob/85d3ad2f585123dc554923928df58f0f15951e35/delete_data_of_deleted_users.php#L22-L25

where you download a foreign file into your webroot. It has the .txt extension, which is good so far, .php would obviously be bad. However, having foreign content in your webroot is certainly not the best thing you can do. In theory a webserver could be misconfigured to interpret PHP code from .txt files.

So I propose blocking this explicity in .htaccess or better, download it into a temporary directory outside of the webroot.

mappy days within recent history

Right now API serves total count of days with at least one edit.

I want to add also statistics about how many days with active mapping happened in the last history.

Initial idea (1)

"recentDaysActive": 19

but that requires hardcoding how many days count as recent in sc-statistics-service and StreetComplete and potential changes need to be coordinated.

Next idea (2) - return also info how many days were taken into account. This way it is defined in one place and just modifying statistics service will result in changes to what SC displays.

{
"recentDaysActive": 19,
"recentDaysActiveRange": 31
}

Next idea (3) - show data day by day

recentDaysActive: {
"2021-12-29": false,
"2021-12-30": false,
"2021-12-31": true,
"2022-01-01": true,
"2022-01-02": true
}

or just list of recent active days

recentDaysActive: [
"2021-12-31",
"2022-01-01",
"2022-01-02"
]

While it would transfer more data and requite nastier SQL to construct it has some benefits.

It would allow to immediately after an edit to update profile statistics. Instead of waiting for statistics service to update.

It would be possible to show per/day statistics like this one (with binary yes/no coloring)

Missing Database Index

Entries from the table changesets are mostly selected by user_id, so there should be an index on user_id to speed up the queries.

Leaderboards by country and by quest

A possible extension to this service would be to store leaderboards by country and by quest.

These leaderboards should not be shown directly in the app but shown for example like this:

• in the profile screen where the total star count is shown, a string like "#75 worldwide, #2 in Denmark" could be shown
• in the quest type details dialog where the star count for that quest type is shown, the same could be shown for that quest type

So, you can see how you are doing yourself, but there is no direct comparison to other users.

Taken into account for these leaderboards would only the users currently in the database.
Depending on the runtime of the SQL queries to ascertain the rank of a user worldwide, in his country, and the same for each quest type, it will most probably be necessary to add additional tables in which to store the leaderboards and update these tables in a scheduled task.

Implementation-wise, there should be DAO(s) for these table(s), a script that updates these tables and probably the whole JSON-generation for the API should move into an own class instead of being in get_statistics.php.