cartodb / cartoframes

CARTO Python package for data scientists

CARTOframes

A Python package for integrating CARTO maps, analysis, and data services into data science workflows.

Python data analysis workflows often rely on the de facto standards pandas and Jupyter notebooks. Integrating CARTO into this workflow saves data scientists time and energy by not having to export datasets as files or retain multiple copies of the data. Instead, CARTOframes give the ability to communicate reproducible analysis while providing the ability to gain from CARTO's services like hosted, dynamic or static maps and Data Observatory augmentation.

Try it Out

• Stable (1.2.4):
• Latest (develop branch):

If you do not have an API key, you can still use cartoframes for creating maps locally.

The example context only provides read access, so not all cartoframes features are available. For full access, Start a free trial or get free access with a GitHub Student Developer Pack.

Features

• Create interactive maps from pandas DataFrames (CARTO account not required)
• Publish interactive maps to CARTO's platform
• Write and read pandas DataFrames to/from CARTO tables and queries
• Create customizable, interactive CARTO maps in a Jupyter notebook using DataFrames or hosted data
• Augment your data with CARTO's Data Observatory
• Use CARTO for cloud-based analysis

Common Uses

• Visualize spatial data programmatically as matplotlib images, as notebook-embedded interactive maps, or published map visualizations
• Perform cloud-based spatial data processing using CARTO's analysis tools
• Extract, transform, and Load (ETL) data using the Python ecosystem for getting data into and out of CARTO
• Data Services integrations using CARTO's Location Data Streams

More info

• Complete documentation: https://carto.com/developers/cartoframes/

• Source code: https://github.com/CartoDB/cartoframes

• Bug tracker / feature requests: https://github.com/CartoDB/cartoframes/issues

  cartoframes users must have a CARTO API key for most cartoframes functionality. For example, writing DataFrames to an account, reading from private tables, and visualizing data on maps all require an API key. CARTO provides API keys for education and nonprofit uses, among others. Request access at [email protected]. API key access is also given through GitHub's Student Developer Pack.

cartoframes's Issues

smarter batching

Aim for a 1 MB batch size (1e6 characters).

Creation of carto-frame from csv,postges,hdfs,geojson etc

Might want to be able to specify geom col / conversions

Ability to specify widgets

Look at both deep-insights.js for now and constructing a .carto with a viz file in it for the builder.

propagating pandas metadata

Notes on propagating pandas metadata:

• pandas-dev/pandas#2485
• geopandas does a good job of propagating metadata, seems to be by subclassing Dataframes similar to what we tried in cartopandas.py.
  • A geodataframe stores it's own metadata: https://github.com/geopandas/geopandas/blob/v0.2.1/geopandas/geodataframe.py#L47

optionally exclude geoms using the carto python sdk

Batch Update Values for Speed

Explore both Update ; method and the weird version here: http://stackoverflow.com/questions/18797608/update-multiple-rows-in-same-query-using-postgresql

create styling that is responsive to basemap

cc @makella

abstract data type mapping to a separate class

These functions have some redundancies, and could be made more general/maintainable:

cartoframes/cartoframes/utils.py

Lines 53 to 121 in 21263b2

	# TODO: combine this with other datatype maps
	def map_dtypes(pgtype):
	"""
	Map PostgreSQL data types (key) to NumPy/pandas dtypes (value)
	:param pgtype: string PostgreSQL/CARTO datatype to map to pandas data type
	Output
	:param : string data type used in pandas
	"""
	# may not be a complete list, could not find CARTO SQL API documentation
	# about data types
	dtypes = {'number': 'float64',
	'date': 'datetime64',
	'string': 'object',
	'geometry': 'object',
	'boolean': 'bool'}
	try:
	return dtypes[pgtype]
	except KeyError:
	# make it a string if not in dict above
	return 'object'
	def dtype_to_pgtype(dtype, colname):
	"""
	Map dataframe types to carto postgres types
	"""
	if colname in ('the_geom', 'the_geom_webmercator'):
	return 'geometry'
	else:
	mapping = {'float64': 'numeric',
	'int64': 'int',
	'datetime64': 'date',
	'object': 'text',
	'bool': 'boolean'}
	try:
	return mapping[dtype]
	except KeyError:
	return 'text'
	def map_numpy_to_postgres(item):
	"""
	Map NumPy values to PostgreSQL values
	"""
	import math
	if isinstance(item, str):
	if "'" in item:
	return "'{}'".format(item.replace("'", "\'\'"))
	return "'{}'".format(item)
	elif isinstance(item, float):
	if math.isnan(item):
	return 'null'
	return str(item)
	elif item is None:
	return 'null'
	return str(item)
	def datatype_map(dtype):
	"""
	map NumPy types to PostgreSQL types
	"""
	# TODO: add datetype conversion
	if 'float' in dtype:
	return 'numeric'
	elif 'int' in dtype:
	return 'int'
	elif 'bool' in dtype:
	return 'boolean'
	else:
	return 'text'

Documentation

ALL OF IT!

Ability to generate static maps

Add progress bar to carto_sync()

Borrow whatever pymc3 does

port over stuart's DO code to cartoframes framework

carto_map options

arguments :

Basic usage

• stylecol
• css (optional)
• color_ramp (optional)
• quantification_method (optional)
• linear, log (optional)

Advanced

• css (just whatever)

Static

Multimap!!!!!

• stylecols = [ ]
• synced = True

add an option for labels on top

Creation of carto-frame from dataframe

Allow Basemap specification in df.carto_map

`CartoContext.write(df, <tablename>)` does not work if carto dataset is syncing with an external source

create a python module for cartocolors

investigate if leveraging the ipynb widget framework makes sense for maps

e.g., this project https://github.com/ellisonbg/ipyleaflet

cc @stuartlynn

mimic interface for `pandas.DataFrame.to_sql`

For consistency across the API, makes sense to mimic the interfaces for other, similar methods:

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html

cc @stuartlynn

add continuous integration

with travis https://docs.travis-ci.com/user/languages/python/

Week 1 Sprint

Argumentation of data frame by DO

make sure cartodb index is type int in dataframe

Method to make df.carto_map permanent

Probably actually a argument to df.carto_map( persist=True)

create a basemaps class

Extract existing code to make it more modular.

Let CartoCSS class inherit properties from Basemap instances.

optionally add zoom and center information on interactive maps

Will help users set the static view of a map

ability to make a cartocontext.map() without specifying datalayers

Sometimes a data scientist just needs to show a map!

methods for getting to carto data and maps pages

something like

• df.get_datapage()
  • prints out the url where the data is displayed (publicly or through auth)
• df.get_mappage()
  • prints out the url where the map is displayed (publicly or through auth)

Reformat in to a proper pip module

Multimap!!!!!

options:

stylecols = [ ]
synced = True

Ref #19

Look at migrating updates from the SQL api to Batch API when over a certain limit

add obs_data_table and obs_meta_table to user accounts

For the data observatory augmentation to work, users will need obs_data_table and obs_meta_table PL/pgSQL functions in their accounts.

cc @talos

Make cartoframes.html template work with onprem

https://github.com/CartoDB/carto-workshop/blob/master/05-apis/exercises/maps_api_guide.md#custom-api-endpoints

create styling responsive to point density

cc @makella

Test to make sure metadata persists properly through various workflows

Scikit-learn
Pysal
Pandas operations

ensure module is compatible with python 2 and 3

collect all allowable names for cartocolor and colorbrewer for ramp-checking

This will be used to add a check on whether an entered ramp is valid.

Examples notebooks

Notebook showing the different ways to use the module

• for each method/function
  • read_carto
  • sync from previous cartoframe
  • sync from new dataframe->cartoframe
  • carto_map
  • data observatory
• data sci workflows (involving common operations, external libraries)

modify interactive map template to have error fallback

Most importantly for development, it shows the types of errors for the grey screens. For users, it might be something useful.

Sync table chunks denominator needs 1 added

Guessing this starts at zero and the counter isn't applied to all.

maps from private data

Creation of Data From from DO

add proportional symbol styles

cc @makella

Scope out how to run analysis

For now just use SQL calls

add iframe fall back that's a static image and link to interactive map

Interactive maps depend on iframe embeds to show their content. Iframes in ipynbs do not show in places like GitHub, so having a fallback that is a static image with a link to the interactive map would be a good substitute.

Allow custom css in df.carto_map

Scope out default cartography

add confirmation with summary of changes before syncing

align function/method names