gravitystorm / snapshot-server Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 2.0 599 KB

Serving OSM map requests from an osmosis pgsnapshot schema

Ruby 79.58% JavaScript 5.01% CSS 6.84% HTML 8.56%

snapshot-server's People

Contributors

Stargazers

Watchers

Forkers

pnorman cyclestreets

snapshot-server's Issues

Load XML files directly

In addition to the fast-as-lightening approach of using osmosis from the command line to load the data, it would be good to have a web interface to add data to projects. Ideally, this would queue uploads and process them asynchronously.

Overall assessment status page

We need better ways to motivate completion of assessment. An overall status (and a list of projects with completion status) is probably the best way to do it.

missing gem install

After running sudo bundle install then rake db:create an error is returned:

pnorman@merry:~/osm/snapshot-server/config$ rake db:create
(in /home/pnorman/osm/snapshot-server)
git://github.com/ctran/annotate_models.git (at master) is not checked out. Please run `bundle install`

Project.transfer could be optimised

Project.transfer is pretty much the least efficient way possible to complete the intended task. Before refactoring it though, it would be good to have some tests.

Osmosis errors when following readme

When following the readme and doing osmosis --rx --wp I get the following error

SEVERE: Thread for task 1-read-xml failed
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT version FROM schema_info]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "schema_info" does not exist
  Position: 21
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:233)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:406)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:463)
        at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:471)
        at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:476)
        at org.springframework.jdbc.core.JdbcTemplate.queryForInt(JdbcTemplate.java:485)
        at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.queryForInt(SimpleJdbcTemplate.java:113)
        at org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:64)
        at org.openstreetmap.osmosis.pgsnapshot.common.SchemaVersionValidator.validateVersion(SchemaVersionValidator.java:47)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.CopyFilesetLoader.run(CopyFilesetLoader.java:78)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlCopyWriter.complete(PostgreSqlCopyWriter.java:108)
        at org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.complete(EntityProgressLogger.java:61)
        at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:110)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "schema_info" does not exist
  Position: 21
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)
        at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
        at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:440)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:395)
        ... 13 more

postgis 2 support (function prefixes)

The current code lack st_ prefixes on some function calls, including

setsrid, in models/project_node.rb
y and x in models/project.rb

and maybe others too?

Missing translation

en.projects.tagged_ways.title - check for nodes and relations too.

Tests for osm file upload

There's no tests for osm file uploads.

Request tests for the form appearing at the right places
Test that only admins can upload files
Unit tests for lib/osm_reader.rb

Where there are no tagged_entities, state this

... rather than just showing a blank table.

move default status to be a per-project configuration

At the moment the status is a global configuration, but it's perhaps more appropriate to be able to control this per-project.

It's not a huge priority until someone starts using other values in the status field than those hardcoded into the potlatch2 snapshotserver vector layer.

Show where the snapshot data is on a map

For sparse datasets, it can be hard (impossible?) to find where the data actually is, and a pain to pan around while mapping. A slippy map on the server with a vector layer (using map calls?) would be useful.

P2 code should have bounds

The potlatch 2 vector background layers supports bounds, such as

This avoids querying when the map is panned out of the area. The snippets generated on project#show should calculate appropriate bounds for the project.

Decouple the file upload from the processing

Importing a non-trivial OSM file will take some time. As a guide, a 1Mb .osm file takes about 30s on my laptop.

It would be good to return a "processing..." page to the client as soon as the file is received, and process the file in a background task.

Tilejson-style endpoints for layers

Each layer comes with an extent and title, policyfile and url endpoint. There's already automatically generated code for use with potlatch's vectors.xml file.

However, copying and pasting a bunch of XML isn't particularly user friendly, and the problem gets worse if we add more details to the datasets, like a description or a license or whatever.

Tilejson is a nice idea for wrapping up all the details about a tile layer into one document. We should do the same for the vector layers.

This would allow mappers to copy+paste a simple url into their editor, in order to use custom vector background layers. Without this, they need a sysadmin to configure vectors.xml, or else hunt around for a correctly configured instance.

hstore if not exists

If attempting to do rake db:migrate on a DB with hstore installed the following error is returned

pnorman@merry:~/osm/snapshot-server$ rake db:migrate
==  SetupHstore: migrating ====================================================
-- execute("CREATE EXTENSION hstore")
rake aborted!
An error has occurred, this and all later migrations canceled:

PGError: ERROR:  extension "hstore" already exists
: CREATE EXTENSION hstore

Tasks: TOP => db:migrate
(See full trace by running task with --trace)

CREATE EXTENSION IF NOT EXISTS would fix this. I intend to prepare a patch.

Fix voting buttons on issue pages

They look like they are supposed to go somewhere else - probably beside the description. At the moment they get their own section and the whitespace looks wrong.

Missing column

Following the instructions in the readme I ran rake db:migrate and got an error:

pnorman@merry:~/osm/snapshot-server/config$ rake db:migrate
(in /home/pnorman/osm/snapshot-server)
rake aborted!
PGError: ERROR:  relation "geometry_columns" does not exist
LINE 1: SELECT * FROM geometry_columns WHERE f_table_name='schema_mi...
                      ^
: SELECT * FROM geometry_columns WHERE f_table_name='schema_migrations'

Tasks: TOP => db:migrate
(See full trace by running task with --trace)

I do not know if this is a code error or a documentation error.

Get spork working

I got spork working for cyclescape - do the same for this project too.

Map calls are full of lazy-loading badness

You can get 60k individual node select statements, if large ways extend beyond the bounding box. Needs investigating (with bullet) and optimising.