cartodb / windshaft-cartodb

Windshaft tailored for CARTO

Windshaft-CartoDB

The CARTO Maps API tiler. It extends Windshaft and exposes a web service with extra functionality:

• Instantiate Anonymous Maps through CARTO's map configuration (MapConfig).
• Create Named Maps based on customizable templates.
• Get map previews through Static Maps API.
• Render maps with a large amount of data faster using Tile Aggregation.
• Build advanced maps with enriched data through Analyses Extension.
• Fetch tabular data from analysis nodes with Dataviews

Build

Requirements:

• Node 12.x
• PostgreSQL >= 11.0
• PostGIS >= 2.4
• CARTO Postgres Extension >= 0.24.1
• Redis >= 4
• libcairo2-dev, libpango1.0-dev, libjpeg8-dev and libgif-dev for server side canvas support
• C++11 to build internal dependencies. When there's no pre-built binaries for your OS/architecture distribution.

Optional:

• Varnish
• Statsd

PostGIS setup

A template_postgis database is expected. One can be set up with

$ createdb --owner postgres --template template0 template_postgis
$ psql -d template_postgis -c 'CREATE EXTENSION postgis;'

Install

To fetch and build all node-based dependencies, run:

$ npm install

Run

You can inject the configuration through environment variables at run time. Check the file ./config/environments/config.js to see the ones you have available.

While the migration to the new environment based configuration, you can still use the old method of copying a config file. To enabled the one with environment variables you need to pass CARTO_WINDSHAFT_ENV_BASED_CONF=true. You can use the docker image to run it.

Old way:

$ node app.js <env>

Where <env> is the name of a configuration file under ./config/environments/.

Test

You can easily run the tests against the dependencies from the dev-env. To do so, you need to build the test docker image:

$ docker-compose build

Then you can run the tests like:

$ docker-compose run windshaft-tests

It will mount your code inside a volume. In case you want to play and run npm test or something else you can do:

$ docker-compose run --entrypoint bash windshaft-tests

So you will have a bash shell inside the test container, with the code from your host.

Coverage

$ npm run cover

Open ./coverage/lcov-report/index.html.

Docker support

We provide docker images just for testing and continuous integration purposes:

• nodejs-xenial-pg1121
• nodejs-xenial-pg101

You can find instructions to install Docker, download, and update images here.

Useful npm scripts

Run test in a docker image with a specific Node.js version:

$ DOCKER_IMAGE=<docker-image-tag> NODE_VERSION=<nodejs-version> npm run test:docker

Where:

• <docker-image-tag>: the tag of required docker image, e.g. carto/nodejs-xenial-pg1121:latest
• <nodejs-version>: the Node.js version, e.g. 10.15.1

In case you need to debug:

$ DOCKER_IMAGE=<docker-image-tag> npm run docker:bash

Documentation

You can find an overview, guides, full reference, and support in CARTO's developer center. The docs directory contains different documentation resources, from a higher level to more detailed ones.

Contributing

• The issue tracker: Github.
• We love Pull Requests from everyone, see contributing to Open Source on GitHub.
• You'll need to sign a Contributor License Agreement (CLA) before submitting a Pull Request. Learn more here.

Developing with a custom Windshaft version

If you plan or want to use a custom / not released yet version of windshaft (or any other dependency), the best option is to use npm link. You can read more about it at npm-link: symlink a package folder.

$ cd /path/to/Windshaft
$ npm install
$ npm link
$ cd /path/to/Windshaft-cartodb
$ npm link windshaft

Versioning

We follow SemVer for versioning. For available versions, see the tags on this repository.

License

This project is licensed under the BSD 3-clause "New" or "Revised" License. See the LICENSE file for details.

windshaft-cartodb's Issues

[REQUEST] Request several columns data via 'interactivity' parameter

Separated by commas maybe?

test environment settings not making trough all testing

serverOptions:
{ base_url: '/tiles/:table',
  grainstore:
   { datasource:
      { user: 'tileuser',
        host: '127.0.0.1',
        port: 5432,
        srid: 4326,
        extent: '-20005048.4188,-20005048.4188,20005048.4188,20005048.4188',
        simplify: true } },
  redis:
   { host: '127.0.0.1',
     port: 6333,
     idleTimeoutMillis: 1,
     reapIntervalMillis: 1 },
  enable_cors: true,
  varnish_host: '',
  varnish_port: null,
  cache_enabled: false,
  log_format: '[:date] :req[X-Real-IP] \u001b[90m:method\u001b[0m \u001b[36m:req[Host]:url\u001b[0m \u001b[90m:status :response-time ms -> :res[Content-Type]\u
001b[0m',
  req2params: [Function],
  getInfowindow: [Function],
  getMapMetadata: [Function],
  flushCache: [Function] }

  ......[Mon, 09 Jul 2012 16:49:30 GMT] - GET 127.0.0.1:5555/ 200 3 ms -> text/html; charset=utf-8
.
....
....

  ✖ 14 of 15 tests failed:

  1) server get'ing blank style returns default style:
     Error: Redis connection to 127.0.0.1:6379 failed - ECONNREFUSED, Connection refused

Where does :6379 comes from if the server was configured with redis port 6333 ?

Database connection settings not honored

Windshaft is trying to connect to the wrong postgresql server (wrong port).

No matter what I write in all the environment configuration files it still tries 5432.
Doesn't honor PGPORT environment variable either.

Send CORS headers for all requests (when enabled)

When running main app and tiler on different TCP ports there are cross-domain issues (you get no response when using ajax).
This can be worked around by setting a configuration variable (enable_cors).

But such configuration is not used by all request types.
The ones served by windshaft have been fixed here: CartoDB/Windshaft#39

This ticket is to fix the ones served by Windshaft-cartodb (metadata, in particular)

@javisantana please scream here too if you see reasons not to do it.

Add authentication test

The acceptance test has no test for unauthorized use

Enable acceptance tests on "make check"

Only unit tests are currently run by "make check", as I couldn't successfully run the acceptance tests.
We want all tests to be run instead.

@tokumine : can you take care of this ?

Deleting "map_style" from redis doesn't trigger an error

While working on #18 I've had troubles with weird behaviors observed on redis key cleaning.
In particular the error found in Windshaft-cartodb log channel says:

FATAL:  database "cartodb_dev_user_2_db" does not exist (encountered during parsing of layer 'gshhs_c_l1' in map '/home/src/cartodb/')

While the database is clearly there (on the right port).

It basically looks like when an MML is missing from redis, something is re-creating one with default values.
And these defaults do not match the configured parameters for postgresql connection.

It is very confusing. I'd like much better an error than a failing attempt to workaround a trouble.
Especially when a default MML may give more authorization than a properly constructed one.

fe_sendauth: no password supplied

This is what I get from Windshaft.
I see no place in the config files to specify a password...

The tiler should use publicuser when requesting unauthenticated tiles

There is a public user already for the SQL API. The tiler right now uses a particular user which has access to all tables, even private. When unathenticated it should use the publicuser for connecting and respect permissions.

Replace SISMEMBER for HMGET

As @Ferdev pointed, this needs to be changed

https://github.com/Vizzuality/Windshaft-cartodb/blob/develop/lib/cartodb/carto_data.js#L189

The details are on the email.

Only dump message on production error (no stack)

http://andrew.cartodb.com/tiles/neemo/1/1/2.png8?cache_buster=0&map_key=c1b5e152490185cb43b39382abe50b5b5ec2f284

Gives:

{"stack":"Error: Coordinates out of range: %d/%d/%d\n at MapnikSource.getTile (/home/ubuntu/www/Windshaft-cartodb/node_modules/windshaft/node_modules/tilelive-mapnik/lib/mapnik.js:306:25)\n at Function. (/home/ubuntu/www/Windshaft-cartodb/node_modules/windshaft/lib/windshaft/server.js:139:35)\n at next (/home/ubuntu/www/Windshaft-cartodb/node_modules/step/lib/step.js:51:23)\n at Object.getRenderer (/home/ubuntu/www/Windshaft-cartodb/node_modules/windshaft/lib/windshaft/render_cache.js:80:13)\n at Function. (/home/ubuntu/www/Windshaft-cartodb/node_modules/windshaft/lib/windshaft/server.js:134:30)\n at next (/home/ubuntu/www/Windshaft-cartodb/node_modules/step/lib/step.js:51:23)\n at Function.finishSetup (/home/ubuntu/www/Windshaft-cartodb/lib/cartodb/server_options.js:58:13)\n at next (/home/ubuntu/www/Windshaft-cartodb/node_modules/step/lib/step.js:51:23)\n at Function. (/home/ubuntu/www/Windshaft-cartodb/lib/cartodb/carto_data.js:131:17)\n at next (/home/ubuntu/www/Windshaft-cartodb/node_modules/step/lib/step.js:51:23)","message":"Coordinates out of range: %d/%d/%d"}

add some sql parsing to know what tables are involved in the query

This will be useful for:

• fine tuning on cache purging
• security on joins

Add test for access to db when NO map key is known

One test in the req2param suite exposed a bug in "master" branch about accesses to a database for which there is no map_key generated being considered as authenticated (ie: "undefined" map_key specified, "undefined" map_key expected then all fine, log in as superuser).

I belive the bug is now fixed in "master" (and wasn't exposed in "develop"), but I think it'd make sense to have a focused test for this in the acceptance suite.

Potential rendering issue: markers truncated on tile boundary

When supplying very large point sizes and allow-overlap=false, they seem not to be rendered on neighboring tiles. I imagine this is similar to the issue that was solved with the CDB_ gridding methods,

https://examples.cartodb.com/tables/carto_tests/embed_map?sql=SELECT%20st_transform(st_setsrid(st_makepoint(10*x,10*x),4326),3857)%20as%20the_geom_webmercator%20FROM%20generate_series(-6,6)%20x

#carto_tests {
   marker-fill: #FF6600;
   marker-opacity: 1;
   marker-width: 28;
   marker-height: 38;
   marker-line-color: white;
   marker-line-width: 3;
   marker-line-opacity: 0.9;
   marker-placement: point;
   marker-type: ellipse;
   marker-allow-overlap: false;
   marker-spacing: 15;
   marker-max-error: 0.4;
}

Allow rendering from WGS84 geometry column

It is currently possible to use a WGS84 geometry column but reprojection performed by mapnik fails for geometries that goes outside of the valid webmercator extent.

protect public tables from having their styles set from external

Basically need another session key :|

Error: Postgis Plugin: bad connection

I was seeing complete tiles missing from the display of a map and when looking at the windshaft log file I found a bunch of:

[TILE RENDER ERROR]
Error: Postgis Plugin: bad connection

For other tiles it works fine so dunno what to look at.
Maybe a limit on number of client connections ?

use last-modified headers

to improve cache performance

test data needed

the acceptance test refers to tables that are not in the .sql provided so nobody can run the tests

it would be good to have the data in the repo :)

cc @tokumine

db_user Parameter in mapnik styles

I noticed there's a in the XML generated by grainstore.
I think it has to do with a parameter added in the configuration file for Windshaft-cartodb, which gets passed verbatim as the datasource of the XML.

The parameter was introduced when moving to PostgreSQL driven authentication, but doesn't need to be proxied to mapnik (it's for internal use only).

I don't like config.postgres to be fully proxied, btw. Anyway the fix here is likely to use a different setting for the db_user, rather than config.postgres

Thanks @lbosque for spotting this, btw

Add tags for releases

There are currently no tags ...

Expose Varnish cache TTL in configuration

Currently hard-coded in lib/cartodb/cartodb_windshaft.js:

            res.header('Cache-Control', 'no-cache,max-age=86400,must-revalidate, public');

Would be better exposed in the configuration

Posting a style doesn't update the style of maps served to unauthenticated requests

I belive unauthenticated requests will always be served tiles with the default style, no matter what you set when you POST a style. This is because when you POST you must be authenticated and being authenticated changes a different redis key than the one used by non authenticated requests.

This issue was introduced with the recent security fix preventing unautenticated access to private tiles.

This is an important bug, most likely affecting both the "develop" and the "master" branches.

It's critical to add a testcase for this in the server acceptance suite.

cartodb-importer: found = in conditional, should be ==

/home/src/cartodb/cartodb/lib/importer/lib/cartodb-importer/decompressors/tar.rb:15: warning: found = in conditional, should be ==

remove configuration files from repository

The config/environments/*.js files are configuration files, shouldn't be in the repository.
I think they should be either manually edited or created by a ./configure step.
There's an "config_off_repo" branch doing it.

configure logging format

TypeError: Cannot read property 'redis' of undefined

It happens when trying to start the application with node app.js production or node app.js development

This sounds like the change I made in 15162e8
which was basically to make Windshaft-cartodb actually use the
configuration parameters which were otherwise not used (specifically
the redis ones, for RedisPool).

Evidently I the application initializes "global.environment.redis" after
requiring that piece of code and thus the problem.

I'll see how to reorder inclusions for a quick fix, altought a better long
term fix would likely be turning CartoData into a real class...

Cleanup testing data after run

We currently create a database and a couple of users in the PostgreSQL cluster while running tests.
It would be wise to choose pseudo-random names for both db and users, and to remove them after the test has been run.

Timeout on "make check"

I'm getting many timeouts which are fixed by disconnecting my machine from the net.
Usually this is a sign of waiting for DNS lookup responses...

This ticket is to modify the tests environement to avoid DNS lookups.

npm install failing on mapnik and tilelive-mapnik

After running npm install it builds most dependencies and bombs out once it reaches [email protected]:

Waf: Leaving directory `/home/linuxadmin/dev/Windshaft-cartodb/node_modules/windshaft/node_modules/tilelive-mapnik-cartodb/node_modules/mapnik/build'
Build failed:  -> task failed (err #1): 
    {task: cxx node_mapnik.cpp -> node_mapnik_1.o}
npm ERR! error installing [email protected] Error: [email protected] install: `node-waf -v configure build`
npm ERR! error installing [email protected] `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! error installing [email protected]     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing [email protected]     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing [email protected]     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! error installing [email protected] Error: [email protected] install: `node-waf -v configure build`
npm ERR! error installing [email protected] `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! error installing [email protected]     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing [email protected]     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing [email protected]     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! error installing [email protected] Error: [email protected] install: `node-waf -v configure build`
npm ERR! error installing [email protected] `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! error installing [email protected]     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing [email protected]     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing [email protected]     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! [email protected] install: `node-waf -v configure build`
npm ERR! `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the mapnik package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-waf -v configure build
npm ERR! You can get their info via:
npm ERR!     npm owner ls mapnik
npm ERR! There is likely additional logging output above.
npm ERR! 
npm ERR! System Linux 3.0.0-12-generic-pae
npm ERR! command "node" "/usr/local/bin/npm" "install"
npm ERR! cwd /home/linuxadmin/dev/Windshaft-cartodb
npm ERR! node -v v0.4.9
npm ERR! npm -v 1.0.106
npm ERR! code ELIFECYCLE
g++ -o obj/jsgc.o -c -Wall -Wno-format -MMD -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DHAVE_LOCALTIME_R -DLINUX -DX86_LINUX -DAVMPLUS_IA32 -DAVMPLUS_UNIX -DAVMPLUS_LINUX  -UDEBUG -DNDEBUG -UDEBUG_linuxadmin -DEDITLINE -I. -I./obj  -Os -fstrict-aliasing -fno-exceptions -fno-rtti -Wstrict-aliasing=2 jsgc.cpp
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/linuxadmin/dev/Windshaft-cartodb/npm-debug.log
npm not ok

I have a standard node v0.4.9 install at /usr/local/bin and all the dependencies installed according to the docs. Any ideas?

Add --with-geomfield configure switch

It may be useful to have switches to specify which geometry field is expected to contain the geometry to be rendered.
In addition to the name, we'll also need to know the SRID and, I think, the extent.

Fix global leaks

The acceptance tests reveal a few global "leaks". It takes removing the --ignore-leaks flag in mocha call in run_tests.sh to see it

Unauthenticated request can change a map's style

This bug seemed to be fixed but wasn't really ... it's just that a request results in a 500 response code, but the style is still changed !

make default border 0.3

Add mapkey auth to the get/set styling

Unauthenticated requests can get style of private tables

You do can get the style of a private table with an unauthenticated request

FAIL: server get'ing completed infowindow with callback returns information with callback

  ✖ 1 of 22 tests failed:

  1) server get'ing completed infowindow with callback returns information with callback:
     AssertionError: undefined. Invalid response body.
    Expected: simon({"infowindow":"this, that, the other"});
    Got: simon({"infowindow":null});

This is "develop" branch with node-0.6

Run tests with "make check"

Make running tests as easy as possible... "make check" follows GNU standards

OSX 10.7 (Lion) requires hiredis 0.1.14

In order to run npm install on OSX 10.7 Lion (Xcode 4.3), you need to be installing hiredis 0.1.14 (not 0.1.12 as in the package.json file).

See redis/hiredis-node#9 for detils.

cache_validator test fails with node-0.6 (in develop branch)

  ✖ 1 of 1 test failed:

  1) cache_validator should call purge on varnish when invalidate database:

  AssertionError: "purge obj.http.X-Cache-Channel == test_db:test_cache\n" == {"0":112,"1":117,"2":114,"3":103,"4":101,"5":32,"6":111,"7":98,"8":106,"9":46,"10":104,"11":116,"12":116,"13":112,"14":46,"15":8
      at /home/src/cartodb/Windshaft-cartodb/test/acceptance/cache_validator.js:36:20

When a tile request fail should return a 400 status code

If user requests a tile that doesn't exist Windshaft answers with a 500 status code, but it should return a 400 status code, like the SQL api when fails.

Authenticated tiles request param name change

Right now to get authenticated tiles we use the param name map_key. On the other hand in the SQL API we use api_key. This difference produces a lot of problems for developers.

I propose we accept map_key and api_key. We will change eventually to api_key to be consistent, but for the time being having both would be great.

Points appear incomplete with some styles

Sometimes when a styled point exceeds the edges of a tile it appears cut.
It doesn't happen always. Not sure which situations cause this problem.
I can't see any error in windshaft log either.

I have a example in staging.cartodb.com:
https://staging.cartodb.com/tables/untitled_table_expor/embed_map

The applied style is:

untitled_table_expor {

marker-fill: #F1600F;
marker-opacity: 0.2;
marker-width: 78;
marker-line-color: white;
marker-line-width: 3;
marker-line-opacity: 0.9;
marker-placement: point;
marker-type: ellipse;
marker-allow-overlap: true;
}

Here there are two screenshots:
https://dl.dropbox.com/s/5e8mhfn7byfnip4/cut_points.png
https://dl.dropbox.com/s/y544d3z9zz0qxkv/cut_points2.png

Be able to reference external assets from a URL

There is a package in carto to do something like this. It is more about enabling it inside cartodb. @tokumine can probably explain better.

Send PostgreSQL errors in the response body

It seems to me that when PostgreSQL errors occur they are not transported to the client, but just happen as logs on the server side. This is from "not authorized" accesses to database tables to SQL syntax errors.

The client only receives a 500 status code and an empty JSON.

Port tests to mocha

expresso testing tool lifetime ended, we want tests to run with mocha

Integrate README with requirements

 node app.js development 

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: libmapnik2.so.2.0: cannot open shared object file: No such file or directory

The README says nothing about libmapnik ?

upgrade cluster implementation to use cluster2

cluster module doesn't work with node-0.6 and built-in node-0.6 cluster isn't good enough (not sure why).

This ticket was moved here from CartoDB/Windshaft#18

What's missing from the core cluster @lbosque @tokumine ?
I've a working cluster2 implementation but has no logs

Add tags for releases

I see there are no tags, nor releases on the npm registry.
Would be nice to start that (and a NEWS file)

Make test for unauthenticated style setting predictable

The current test for POST'ing good style from unauthenticated request is suffering from a race condition as it assumes the mocha tests will run one after another. This is not guaranteed!