apache / incubator-baremaps Goto Github PK
View Code? Open in Web Editor NEWCreate custom vector tiles from OpenStreetMap and other data sources with Postgis and Java.
Home Page: baremaps.apache.org
License: Apache License 2.0
Create custom vector tiles from OpenStreetMap and other data sources with Postgis and Java.
Home Page: baremaps.apache.org
License: Apache License 2.0
I am working with Windows 10 and JDK 14. When using the LocalFileSystem
in the application I run into the following exception when parsing a Windows File path passed from the CLI to LocalFileSystem
:
java.nio.file.InvalidPathException: Illegal char <:> at index 2:
It happens when Path path = Paths.get(uri.getPath());
on Windows is called. I saw several similar bugs in the internet, and some fixed this by resolving the path like this: Paths.get(new File(uri).getAbsolutePath())
, but I am unsure if it also works for Mac OS and Linux`.
That's why I am opening an issue instead of directly making a PR.
The LocalFileSystem
then looks like this:
import com.baremaps.util.vfs.FileSystem;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
public class LocalFileSystem extends FileSystem {
@Override
public boolean accept(URI uri) {
return uri.getHost() == null && uri.getPath() != null;
}
@Override
public InputStream read(URI uri) throws IOException {
Path path = getPathFromUri(uri);
return Files.newInputStream(path);
}
@Override
public byte[] readByteArray(URI uri) throws IOException {
Path path = getPathFromUri(uri);
return Files.readAllBytes(path);
}
@Override
public OutputStream write(URI uri) throws IOException {
Path path = getPathFromUri(uri);
if (!Files.exists(path.getParent())) {
Files.createDirectories(path.getParent());
}
return Files.newOutputStream(path);
}
@Override
public void writeByteArray(URI uri, byte[] bytes) throws IOException {
Path path = getPathFromUri(uri);
if (!Files.exists(path.getParent())) {
Files.createDirectories(path.getParent());
}
Files.write(path, bytes);
}
@Override
public void delete(URI uri) throws IOException {
Path path = getPathFromUri(uri);
Files.deleteIfExists(path);
}
private Path getPathFromUri(URI uri) {
return Paths.get(new File(uri).getAbsolutePath());
}
}
I faced this error after starting the server with a custom layer:
[ERROR] 2021-02-21 14:46:02.017 [main] Preview - Unable to read the configuration file.
com.fasterxml.jackson.databind.JsonMappingException: /config/custom/skating_mushrooms/layer.yaml (through reference chain: com.baremaps.config.Config["layers"]->java.util.ArrayList[0])
...
java.lang.NullPointerException
...
but my layer is actually named skating_mushrooms/layer.yml
, therefore I suspect that the file extension is hard codded.
From here, there are probably two solutions:
.yml
extensions, e.g. within a regexp,Hi,
Thanks for this very good and interresting project.
I want import data in shapefile format from IGN DB Topo (https://geoservices.ign.fr/documentation/diffusion/telechargement-donnees-libres.html#bd-topo).
Since 1 January 2021, all public data (vector and ortho-images) from the National Institute of Geographic and Forest Information (IGN) are free and accessible free of charge (https://www.actuia.com/english/the-igns-public-data-relating-to-topography-relief-and-visualization-of-the-territory-are-free-and-open-to-the-public/)
Cdt
Youcef HILEM
The link goes to https://www.camptocamp.com/en/solutions/geospatial-solutions/, but should (probably) go to https://www.camptocamp.com/geospatial_solutions
Behind a proxy, the tile source and the html pages may have different origin.
MBTiles is a specification for storing arbitrary tiled map data in SQLite databases for immediate usage and for efficient transfer.
Hello,
I am trying to import planet.pbf (55GB) using the OSMVecto example. According to log output, the import is at over 1800%:
[INFO ] 2020-12-17 08:51:28.203 [main] StreamProgress - Progress: 1850.24%
I'm running out of disk space since the import seems to require a lot (currently ~1.3TB and growing) and it's difficult to estimate the disk requirements for the full import with >100% progress indicator.
Empty tiles raises errors in the browser:
evented.js:109 Error: Error: http status 200 returned without content.
We should find a way to handle this case more properly
I'm not sure to fully understand the what the export
command need as arguments to properly work:
# baremaps export \
--database 'jdbc:postgresql://'${HOST}':'${PORT}'/'${POSTGRES_DB}'?user='${POSTGRES_USER}'&password='${POSTGRES_PASSWORD} \
--config "${config_file}" \
--log-level=DEBUG \
--repository=./data/tiles
[INFO ] 2021-03-09 08:22:58.654 [main] Export - 56 processors available
[INFO ] 2021-03-09 08:22:58.681 [main] Export - Reading configuration
[DEBUG] 2021-03-09 08:22:58.795 [main] FileBlobStore - Read
java.io.IOException: Is a directory
...
Here is the help
:
Usage: baremaps export [--enable-s3] [--mbtiles] [--batch-array-index=READER]
[--batch-array-size=BATCH_ARRAY_SIZE] --config=YAML
--database=DATABASE [--log-level=LOG_LEVEL]
--repository=URL [--tiles=TILES]
Export vector tiles from the database.
--batch-array-index=READER The index of the batch in the array.
--batch-array-size=BATCH_ARRAY_SIZE
The size of the batch array.
--config=YAML The YAML source configuration file.
--database=DATABASE The JDBC url of the Postgres database.
--enable-s3 Enable Amazon S3 integration.
--log-level=LOG_LEVEL The log level.
--mbtiles The repository is in the MBTiles format.
--repository=URL The tile repository URL.
--tiles=TILES The tiles to export.
Why is the --repository
expecting a URL? Can't we just simply save the tiles to the file system? How do we achieve that then?
And what exactly are the --tiles
and --mbtiles
flags doing? Maybe I must use one of those instead, but the --repository
flag seems mandatory and if I set a path to the -mbtiles
one I got:
Invalid value for option '--mbtiles': './data/tiles' is not a boolean
which seems to suffer a tiny offset in what I understand from the documentation.
Maybe it could be nice to also print out the expected values for each args, as many bash command help pages are actually doing.
Thanks for having a look!
The current approach for importing data drops the tables, creates a new one, fills them, creates the indexes, etc. This means that the database is in an unusable state for some time. It would be a good thing to perform these operations within a transaction to ensure that the database remains in a usable state.
This would ensure that baremaps works on every major operating systems (see #84).
Investigate the use of curl and of the maven plugin.
https://sdkman.io/vendors
https://github.com/sdkman/sdkman-vendor-maven-plugin
The description of this issue comes from openstreetmap-vecto:
baremaps/openstreetmap-vecto#24
Right now, the command uses createTempDirectory
. This is a good default value, but we shall be able to specify the location of the LMDB cache, especially if the host has several hard drives with different performance characteristics (HDD, SSD, see #97). Additionally, we should ensure that the cache is deleted when the execution terminates.
serve
process expects to have an AWS S3 configuration on the hosting machine, while it should run even though you don't have S3 configured.
The CLI should be able to display the version number set in the pom.xml file.
I faced this error after starting the server with a custom layer having non standard SQL characters (must be double quoted):
[WARN ] 2021-02-21 14:54:26.213 [armeria-common-worker-epoll-2-5] HttpResponseSubscriber - [id: 0x1dea6d9c, L:/172.28.0.3:9000 - R:/10.193.21.89:42482] Unexpected exception from a service or a response publisher: com.baremaps.server.TileService@238a9e9c
java.lang.IllegalArgumentException: The SQL query malformed
at com.baremaps.tile.postgres.PostgisQueryParser.parse(PostgisQueryParser.java:52) ~[baremaps-tile-postgres-0.4.0.jar:?]
at com.baremaps.tile.postgres.PostgisTileStore.lambda$null$0(PostgisTileStore.java:137) ~[baremaps-tile-postgres-0.4.0.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at com.baremaps.tile.postgres.PostgisTileStore.query(PostgisTileStore.java:138) ~[baremaps-tile-postgres-0.4.0.jar:?]
at com.baremaps.tile.postgres.PostgisTileStore.read(PostgisTileStore.java:110) ~[baremaps-tile-postgres-0.4.0.jar:?]
at com.baremaps.server.TileService.lambda$doGet$0(TileService.java:58) ~[baremaps-server-0.4.0.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:522) ~[armeria-1.4.0.jar:?]
at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) ~[micrometer-core-1.6.3.jar:1.6.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
I suspect the PostgreSQL layer name MUST be escaped in the SQL query defined in the ./config/layer.yaml
file, especially when they contain non standard SQL characters such as capital letters or digits, because when I escape the layer name using double quotes, it doesn't print this error at startup.
...
FROM 1_differential_rabbiHousing${zoom}
---
-> goes bad...
...
FROM "1_differential_rabbiHousing"${zoom}
...
-> goes better...
Would it be possible to quote (/check for) double quote around the PostgreSQL table names in the backend?
Here's the config.yaml
:
id: OpenStreetMapVecto
server:
host: localhost
port: 9000
center:
lon: 6.5743
lat: 46.5189
zoom: 14
bounds:
minLon: 5.8358
minLat: 45.6591
maxLon: 10.9794
maxLat: 47.8700
minZoom: 6
maxZoom: 14
layers:
- config/rabbits/1_differential_rabbiHousing/layer.yaml
stylesheets:
- config/rabbits/1_differential_rabbiHousing/stylesheet.yaml
/config/rabbits/1_differential_rabbiHousing/layer.yaml
:
id: 1_differential_rabbiHousing
type: point
queries:
- minZoom: 13
maxZoom: 22
sql: |
SELECT id, rabbit_id, name, familiy, has_carot, speed, geometry
FROM "1_differential_rabbiHousing"${zoom}
WHERE geometrytype(geometry) LIKE '%POINT%'
/config/rabbits/1_differential_rabbiHousing/stylesheet.yaml
:
id: 1_differential_rabbiHousing
styles:
- id: ogc_fid
layer: 1_differential_rabbiHousing
type: fill
layout:
visibility: visible
paint:
fill-color: rgb(0, 0, 228)
fill-antialias: true
I'm not sure whereas the layer
and id
in the stylesheet has to represent the same object and what they do refer to?
I was able to eventually get through the steps outlined in https://www.baremaps.com/examples/openstreetmap/ to show a map of Liechtenstein with the Blueprint style.
I've been trying to get this to work with Switzerland and the Openstreetmap-vecto style. When I import the switzerland-latest.osm.pbf file, I encountered exceptions such as:
[WARN ] 2021-01-28 11:20:12.881 [ForkJoinPool.commonPool-worker-83] GeometryHandler - Unable to build the geometry for relation 1713904
java.lang.NullPointerException: null
at org.locationtech.jts.algorithm.locate.IndexedPointInAreaLocator.locate(IndexedPointInAreaLocator.java:88) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.isInRing(EdgeRing.java:261) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.findEdgeRingContaining(EdgeRing.java:87) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.findShellContaining(HoleAssigner.java:106) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHoleToShell(HoleAssigner.java:78) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:72) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:40) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.polygonize(Polygonizer.java:255) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.getGeometry(Polygonizer.java:188) ~[jts-core-1.17.1.jar:?]
at com.baremaps.importer.geometry.GeometryHandler.handle(GeometryHandler.java:104) ~[baremaps-importer-0.3.4.jar:?]
at com.baremaps.osm.domain.Relation.accept(Relation.java:47) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:20) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:15) ~[baremaps-osm-0.3.4.jar:?]
at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) [?:?]
at com.baremaps.osm.BlockEntityHandler.handle(BlockEntityHandler.java:27) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.pbf.DataBlock.handle(DataBlock.java:42) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:15) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:9) [baremaps-osm-0.3.4.jar:?]
and then error messages such as:
[WARN ] 2021-01-28 11:20:32.414 [ForkJoinPool.commonPool-worker-123] GeometryHandler - Unable to build the geometry for relation 7521607
java.lang.NullPointerException: null
[WARN ] 2021-01-28 11:20:37.100 [ForkJoinPool.commonPool-worker-15] GeometryHandler - Unable to build the geometry for relation 5503310
java.lang.NullPointerException: null
It's unclear to me whether the import job finished despite the errors, and perhaps gave up after encountering the errors. Either way, when I try to run the server, I get exceptions such as:
[ERROR] 2021-01-28 11:47:05.277 [armeria-common-worker-epoll-2-6] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_ways_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-5] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_relations_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-4] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_water_polygons" does not exist
Position: 335
I believe something similar happened when I tried to import us-west. My larger goal is to import the OSM data for the world.
On a tangential note, I have used imposm3 with some success before; could Baremaps' server be made to work with imposm imports? (e.g. specifying an appropriate mapping file)
Thank you.
As a data integrator
I want to import only the data that my style requires
Because I want to reduce the costs of storage and the import time
use case
I want to create a style for a basemap, that does not contain the buildings, but on a large dataset (planet OSM). I would like then to add a blacklist (in the config file?) where I list the tags related to buildings, so that in the end they are not imported in the PG database.
The /config/style.json
file is used to render the vector tiles in mapbox. It can easily be edited with maputnik. As for now, the style provided does not support all the zoom levels. The idea would be improve this style, taking inspiration from what OpenStreetMap does for raster tiles.
It feels quite empty on the Docker Hub, e.g. here https://hub.docker.com/r/baremaps/baremaps
compared to, e.g. https://hub.docker.com/r/osgeo/gdal
where one can spot the github page hosting the Dockerfile which has been used to build''n'push the image to the hub:
it's a "nice to have" feature, but an immensely useful one, especially for newcomers imho.
Hi there,
I'm following the OSM tutorial, and at this stage:
$ baremaps import \
> --database 'jdbc:postgresql://localhost:5432/baremaps?&user=baremaps&password=baremaps' \
> --file 'https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf'
I faced these warnings:
[INFO ] 2021-02-05 09:31:14.555 [main] Import - 56 processors available
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.lmdbjava.ByteBufferProxy$AbstractByteBufferProxy (file:/opt/baremaps/lib/lmdbjava-0.8.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.lmdbjava.ByteBufferProxy$AbstractByteBufferProxy
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2021-02-05 09:31:14.777 [main] HttpBlobStore - Read https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
Just wondering if it's normal?
Environment:
Ubuntu 18.04.5 LTS 5.4.0-65-generic x86_64 GNU/Linux
Release used: https://github.com/baremaps/baremaps/releases/tag/v0.3.4
[INFO ] 2020-12-07 16:10:20.906 [main] Import - 16 processors available
java : Exception in thread "main" java.lang.UnsatisfiedLinkError: unknown
At line:14 char:1
+ java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps import --datab ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Exception in th...kError: unknown:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:93)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:71)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:50)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:60)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:138)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:86)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:392)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:371)
at org.lmdbjava.Library.<clinit>(Library.java:125)
at org.lmdbjava.Env$Builder.open(Env.java:486)
at org.lmdbjava.Env$Builder.open(Env.java:512)
at com.baremaps.cli.Import.call(Import.java:116)
at com.baremaps.cli.Import.call(Import.java:45)
at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
at picocli.CommandLine.access$1200(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
at picocli.CommandLine.execute(CommandLine.java:2058)
at com.baremaps.cli.Baremaps.main(Baremaps.java:43)'
It appears I miss some native library, but I do do not know which one. I dropped https://jdbc.postgresql.org/download.html to the lib
directory, just in case (didn't help).
(...and remove circleci builds)
Hi!
I noticed a peculiar thing with coordinates when running baremaps to PostgreSQL database and thought to ask about the purpose or if there's something strange here.
The setup is:
Not sure if a language issue. If I change language settings, Baremaps stops working (LMDB errors with code 112, the ava wrapper does not understand it, Baremaps stops). Maybe I check a bit later.
As for an example for https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
SELECT * FROM public.osm_ways WHERE id = 294942376;
SELECT * FROM public.osm_nodes WHERE id IN (2985838396, 2985838357, 2985838346, 2985838390, 2985838517, 2985838501, 2985838396);
SELECT ST_AsText(geom) FROM public.osm_nodes WHERE id = 2985838396;
SELECT ST_SRID(geom) FROM public.osm_nodes WHERE id = 2985838396;
SELECT ST_AsText(geom) FROM public.osm_ways WHERE id = 294942376;
to a database that is generated from https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf
. It's that the coordinates look very large. Here are the results from the SQL queries
id | version | uid | timestamp | changeset | tags | nodes | geom |
---|---|---|---|---|---|---|---|
294942376 | 1 | 0 | 2014-07-29T11:56:56 | 0 | "building"=>"yes" | [2985838396, 2985838357, 2985838346, 2985838390, 2985838517, 2985838501, 2985838396] | 0103000020110F000001000000070000001C92315647273041BF750E0517C45641EA7078054B27304113BCC09010C45641148CE30540273041FE3D2B2C10C456417DDC054F3E273041ACE9432E13C456411B446521382730418707BCF612C45641E414FC2836273041B0C2E46716C456411C92315647273041BF750E0517C45641 |
id | version | uid | timestamp | changeset | tags | lon | lat | geom |
---|---|---|---|---|---|---|---|---|
2985838346 | 1 | 0 | 2014-07-29T11:56:50 | 0 | NULL | 9.509781400000001 | 47.1582123 | 0101000020110F0000148CE30540273041FE3D2B2C10C45641 |
2985838357 | 1 | 0 | 2014-07-29T11:56:50 | 0 | NULL | 9.509880200000001 | 47.1582219 | 0101000020110F0000EA7078054B27304113BCC09010C45641 |
2985838390 | 1 | 0 | 2014-07-29T11:56:51 | 0 | NULL | 9.509766 | 47.1582858 | 0101000020110F00007DDC054F3E273041ACE9432E13C45641 |
2985838396 | 1 | 0 | 2014-07-29T11:56:51 | 0 | NULL | 9.5098471 | 47.1583796 | 0101000020110F00001C92315647273041BF750E0517C45641 |
2985838501 | 1 | 0 | 2014-07-29T11:56:52 | 0 | NULL | 9.5096928 | 47.158364600000006 | 0101000020110F0000E414FC2836273041B0C2E46716C45641 |
2985838517 | 1 | 0 | 2014-07-29T11:56:53 | 0 | NULL | 9.5097105 | 47.158280500000004 | 0101000020110F00001B446521382730418707BCF612C45641 |
st_astext POINT(1058631.33669389 5967964.07900757)
st_srid 3857
POLYGON((1058631.33669389 5967964.07900757,1058635.02136903 5967938.26176359,1058624.02300334 5967936.69013929,1058622.30868319 5967948.72289507,1058616.13045145 5967947.8552264,1058614.16009646 5967961.62333743,1058631.33669389 5967964.07900757))
For instance, taking one point from there should be at https://www.openstreetmap.org/node/2985838396 and the geometry it produces probbly should be POLYGON((9.5098471 47.1583796,9.5098802 47.1582219,9.5097814 47.1582123,9.509766 47.1582858,9.5097105 47.1582805,9.5096928 47.1583646,9.5098471 47.1583796))
instead of that with the odd coorginates. At least this is how it looks to me when I plug the polygon, say, to http://arthur-e.github.io/Wicket/sandbox-gmaps3.html.
Hello, my question is pretty simple :
Is there a way to know or estimate the export time ?
❗ I'm currently exporting since 7 hours now, for a DB based on the openstreemap-vecto project, for a ~120MB osm.pbf file on a 2vcore VPS.
❓ I know it's a light machine and that I must except a looong export time, but I'm curious about what influence the export time and if there is some known numbers (export benchmarks ?).
By the way, I'm doing all of that in a Docker container which mirror the tiles/ (export) folder onto the host machine, if that makes a difference (even if I think it's more of a CPU bottleneck than a disk one but anyway).
Thanks by advance for you help. 👍
https://www.jooq.org/doc/latest/manual/sql-building/sql-parser/
https://www.jooq.org/doc/latest/manual/sql-building/dsl-context/custom-settings/settings-parser/
Set Settings.parseUnknownFunctions to IGNORE
AWS Batch allows to distribute the computation of embarrassingly parallel problems. The creation of vector tiles is embarrassingly parallel. Adding command line arguments for the number of batches and the batch to be processed would allow baremaps to be deployed on AWS batch as a docker image.
During some imports, I got many of these warnings: Unable to build the geometry for relation XXXXXX
Here's a log dump when such even occurs:
(...)
Import openstreetmap
--2021-02-18 15:18:36-- https://download.geofabrik.de/europe/switzerland-latest.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 116.202.112.212, 88.99.142.44
Connecting to download.geofabrik.de (download.geofabrik.de)|116.202.112.212|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘switzerland-latest.osm.pbf’ not modified on server. Omitting download.
[INFO ] 2021-02-18 15:18:37.186 [main] Execute - 56 processors available
[INFO ] 2021-02-18 15:18:37.205 [main] Execute - ../queries/osm_create_extensions.sql
[INFO ] 2021-02-18 15:18:37.302 [main] Execute - ../queries/osm_drop_views.sql
[INFO ] 2021-02-18 15:18:37.319 [main] Execute - ../queries/osm_drop_tables.sql
[INFO ] 2021-02-18 15:18:37.321 [main] Execute - ../queries/osm_create_tables.sql
[INFO ] 2021-02-18 15:18:40.281 [main] Import - 56 processors available
[INFO ] 2021-02-18 15:18:40.558 [main] ImportTask - Creating cache
[INFO ] 2021-02-18 15:18:45.616 [ForkJoinPool.commonPool-worker-60] ProgressLogger - 3.2%
[INFO ] 2021-02-18 15:18:50.679 [ForkJoinPool.commonPool-worker-52] ProgressLogger - 5.19%
(...)
[INFO ] 2021-02-18 15:22:46.344 [ForkJoinPool.commonPool-worker-63] ProgressLogger - 91.96%
[INFO ] 2021-02-18 15:22:54.597 [main] ImportTask - Importing data
[INFO ] 2021-02-18 15:23:03.576 [ForkJoinPool.commonPool-worker-27] ProgressLogger - 6.52%
[INFO ] 2021-02-18 15:23:10.033 [ForkJoinPool.commonPool-worker-43] ProgressLogger - 10.67%
(...)
[INFO ] 2021-02-18 15:25:31.238 [ForkJoinPool.commonPool-worker-12] ProgressLogger - 94.32%
[INFO ] 2021-02-18 15:25:39.018 [ForkJoinPool.commonPool-worker-38] ProgressLogger - 95.6%
[WARN ] 2021-02-18 15:25:47.889 [ForkJoinPool.commonPool-worker-48] GeometryHandler - Unable to build the geometry for relation 3362179
java.lang.NullPointerException: null
at org.locationtech.jts.algorithm.locate.IndexedPointInAreaLocator.locate(IndexedPointInAreaLocator.java:88) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.isInRing(EdgeRing.java:261) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.findEdgeRingContaining(EdgeRing.java:87) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.findShellContaining(HoleAssigner.java:106) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHoleToShell(HoleAssigner.java:78) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:72) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:40) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.polygonize(Polygonizer.java:255) ~[jts-core-1.18.0.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.getGeometry(Polygonizer.java:188) ~[jts-core-1.18.0.jar:?]
at com.baremaps.osm.geometry.GeometryHandler.handle(GeometryHandler.java:104) ~[baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.domain.Relation.accept(Relation.java:47) ~[baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.handler.EntityHandler.accept(EntityHandler.java:20) ~[baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.handler.EntityHandler.accept(EntityHandler.java:15) ~[baremaps-osm-0.4.0.jar:?]
at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65) ~[?:1.8.0_275]
at java.util.ArrayList.forEach(ArrayList.java:1259) [?:1.8.0_275]
at com.baremaps.osm.handler.BlockEntityHandler.handle(BlockEntityHandler.java:27) [baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.domain.DataBlock.handle(DataBlock.java:39) [baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.handler.BlockHandler.accept(BlockHandler.java:15) [baremaps-osm-0.4.0.jar:?]
at com.baremaps.osm.handler.BlockHandler.accept(BlockHandler.java:9) [baremaps-osm-0.4.0.jar:?]
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:439) [?:1.8.0_275]
at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:901) [?:1.8.0_275]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) [?:1.8.0_275]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) [?:1.8.0_275]
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) [?:1.8.0_275]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) [?:1.8.0_275]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_275]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_275]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_275]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_275]
(...)
Not quite sure where to search from here...
Typical examples are:
https://www.openstreetmap.org/relation/2585841
https://www.openstreetmap.org/relation/11312837
https://www.openstreetmap.org/relation/10032656
https://www.openstreetmap.org/relation/11485685
The docker images are created and published manually. This may be automated with the (jib-maven-plugin)[https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#quickstart] and with github actions.
The page https://www.camptocamp.com/en/solutions/geospatial-solutions/ is dead.
Referenced in README.md and https://www.baremaps.com/.
If the consuming code of the stream is doing lots of small reads, using a BufferedInputStream might improve performance here.
The current distribution contains a bash script. The equivalent for windows systems should be created. Probably something like a baremaps.bat file containing:
@echo off
set dir=%~dp0%
set LIB="%dir%..\lib\*"
java -cp "$DIR/../lib/*" com.baremaps.cli.Baremaps %*
hi,sir, i download the latest code of the baremaps,but in the com.baremaps.osm.pbf.FileBlock,it looks like missing com.baremaps.osm.binary.*
For instance, a subset of the layers could be returned by passing parameters in the URL or in the query parameters.
/z/x/y.pbf?layers=layer1+layer2+layer3
If no filter is specified, all the layers are returned in the vector tile.
I just made a few tests on my laptop to check how many tiles/s baremaps can handle in serve
mode. I used the lichtenstein example "as is". In my tests baremaps cannot handle more than 10rps (requests per second). If used with more (say 12rps) the success rate drastically drops to 20-25 % and we get 75% of failed connections.
This behaviour is consistent with the cpu which was completely used, and probably it was no more possible to create new connections with the server.
up to 10 rps we get:
['baremap-10rps.log']
response time [ms] statistics for test : '10rps'
mean : 1389.6041366666666
median : 388.16700000000003
min-max-range : 9150.050000000001
percentage-of-http-200 : 100.0
percentage-of-http-errors : 0.0
percentage-of-network-errors : 0.0
percentile90 : 4911.853500000011
percentile95 : 6935.2443
std : 2162.730614611565
and with 12rps:
['baremap-12rps.log']
response time [ms] statistics for test : '12rps'
mean : 2418.637625
median : 2029.3229999999999
min-max-range : 9834.434
percentage-of-http-200 : 24.444444444444443
percentage-of-http-errors : 0.0
percentage-of-network-errors : 75.55555555555556
percentile90 : 5562.04
percentile95 : 6381.597750000001
std : 2206.189541154367
the test setup (not optimal) is my laptop pc:
to test I used yandex-tank with the input files provided as attachement yandex-config.zip. To run yandex, just do
docker run \
-v $(pwd):/var/loadtest \
--net host \
-it \
--entrypoint /bin/bash \
direvius/yandex-tank
and in the container:
yandex-tank -c shoot-baremaps.ini baremaps.ammo
While I am experimenting with Windows, I have noticed, that the osm_relations
table is always empty after the import. I think the problem is due to the PostgisReferenceCache
implementation I am using.
In PostgisReferenceCache#get
an IllegalArgumentException
is thrown when the Reference does not exist:
I think this will cause the whole stream to cancel when it is thrown in the RelationBuilder#build
:
Can't the Cache return null
, when the Element is not available? Or should the call in the stream be wrapped with Try.of(...)
?
The config/config.yaml
file is used to generate vector tiles. It contains SQL queries for each zoom levels. As for now, the example provided only specifies queries for the zoom level 14. The idea would be to add queries for zoom levels 0 to 13, taking inspiration from what OpenStreetMap does for raster tiles.
https://www.openstreetmap.ch/
https://github.com/gravitystorm/openstreetmap-carto
https://wiki.openstreetmap.org/wiki/Features
To deploy baremaps-server in an environment with complex http routing, It would be probably beneficial to be able to parametrize the service route instead of having it hardcoded as /tiles
.
See this point: https://github.com/baremaps/baremaps/blob/master/baremaps-server/src/main/java/com/baremaps/server/StyleBuilder.java#L54
As tiles are a form of space partitionning, SP-GIST are probably a better fit than GIST for indexing geometries. The idea would be to investigate how an SP-GIST can be configured to match tile boundaries.
As a developer,
I'd like to be able to see the rendered SQL query on stdout,
so that I can copy-paste it in my pg admin tool to debug it.
For the moment, I'm unable to spot what's wrong from such error messages (printed to stdout on server's side) when browsing the app:
[WARN ] 2021-02-21 15:06:32.103 [armeria-common-worker-epoll-2-7] HttpResponseSubscriber - [id: 0x8de6f642, L:/172.28.0.3:9000 - R:/10.193.21.89:42688] Unexpected exception from a service or a response publisher: com.baremaps.server.TileService@258a2a21
java.lang.IllegalArgumentException: The SQL query malformed
at com.baremaps.tile.postgres.PostgisQueryParser.parse(PostgisQueryParser.java:52) ~[baremaps-tile-postgres-0.4.0.jar:?]
at com.baremaps.tile.postgres.PostgisTileStore.lambda$null$0(PostgisTileStore.java:137) ~[baremaps-tile-postgres-0.4.0.jar:?]
The SQL query is apparently still missing when --log-level=DEBUG is enabled:
[DEBUG] 2021-02-22 11:02:58.713 [pool-2-thread-10] FileBlobStore - Read config-rabbits.yml
[DEBUG] 2021-02-22 11:02:58.713 [pool-2-thread-6] FileBlobStore - Read config-rabbits.yml
[DEBUG] 2021-02-22 11:02:58.718 [pool-2-thread-6] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/layer.yml
[DEBUG] 2021-02-22 11:02:58.718 [pool-2-thread-10] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/layer.yml
[DEBUG] 2021-02-22 11:02:58.720 [pool-2-thread-6] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/stylesheet.yml
[DEBUG] 2021-02-22 11:02:58.720 [pool-2-thread-10] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/stylesheet.yml
As new services will be added, the command line interface needs to be reorganized. The current idea is to have something similar to what aws-cli does, i.e., the sub command correspond to the context. In baremaps it could translate as follow:
baremaps [context] [action]
baremaps osm import
baremaps osm update
baremaps tiles export
baremaps tiles serve
baremaps tiles preview
baremaps location serve
baremaps search serve
baremaps route serve
...
@nicolas-heigvd do not hesitate to list here all the ideas you may have regarding docker's best practices. For instance, a possibility may be to use the kubernetes-maven-plugin to create docker images and k8s configurations.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.