sethoscope / heatmap

Python script for generating high quality heatmaps based on any coordinate data (GPS tracks, eye tracking, etc).

heatmap

why?

There are a few kinds of heat maps. This program is for when you have data points, each with a pair of orthogonal coordinates (X/Y, lat-lon) and you want to plot them on a map such that they blob together a bit to indicate density.

So, it's good for things like:

• eye tracking data
• lat/lon geocoded data points
• GPS tracks

why not?

It's not good for:

• showing results in realtime (because it's too slow)
• running in a browser (because it's in Python)
• automatically layering on proprietary map systems

So... why use a slow data visualizer that doesn't run in a browser? Because the output looks better.

There's another kind of heatmap, also called a choropleth map, in which you divide your map into regions and color each region to indicate something. This tool is not for that.

A more thorough description and examples are posted at http://sethoscope.net/heatmap/

change to command line

As of version 1.13 (April 2017), the method of specifying input files has changed. The old way will continue to be supported for at least one year, but the new way is nicer. Read this tutorial for examples and help on switching to the new way.

Tests

heatmap's Issues

.csv file is parsed differently than a plain file

When using a .csv file as input, only Lat/Lon is interpreted. When using a plain file, a third value is used as a weight for that coordinate.

It is desirable for .csv and plain files to be processed the same.

Result image wrong resolution

heatmap.py -o photos.png --width 1000 --osm -B 0.8 --osm_base http://b.tile.stamen.com/toner photo-points

When you call this the result resolution is 915 x 357.

1. How can i get the exact resolution? 1000 x 380(approximately) with dynamic padding(only map around without data).
2. if both --width and --height set how can i get exactly that resolution with dynamic padding?

Adding axis

Could there be an option to add an axis to the pictures? This would be especially helpful when making lat/long plots, to help pinpoint the geographical position of the plot.

PyPi

Package and distribute via PyPi (so people can just pip install ...).

We'd need:

• setup.py, init.py
• a name available in their namespace (not heatmap, pyheatmap, or heatmappy)
• ongoing version number upkeep
• occasional follow-up releases

Fix for using pillow instead of PIL

Hi - lovely tool! I have 1 small change to suggest, in heatmap/heatmap.py if you change lines 25 and 26 from:
import Image
import ImageColor
to:
from PIL import Image
from PIL import ImageColor
then you can use https://github.com/python-imaging/Pillow (far more cross-platform) rather than just PIL (no other change required - this lets me use pillow with OpenStreetMap data). Cheers, Ian.

Suggestion to reduce blockly output

Thanks for your code. I do have one small suggestion. Perhaps inserting so code to skip zero valued data (at line 615):
for (coord, val) in matrix.items():
if val == 0.0:
continue
x = int(coord.x - extent.min.x)

I found that my output was very "blocky" without this because where the data is zero valued, a color is still assigned. This is probably due to the color scheme I used.

test coverage

Test coverage dropped significantly.
https://coveralls.io/builds/10860249

The arguments are order dependant

Just a minor issue I noticed. The --osm argument works if placed at the end of the arguments, but not at the start.

argument 2 to map() must support iteration error

i've got this error (on line 739 pb in lambda expression for cropping in osm mode) when i'm running this command line on W vista with python 2.6 and also ubuntu 12.04 with python 2.3.

my command line :

python heatmap.py -p points.txt -o /home/me/sortie.png --zoom 6 --osm --debug

it works fine without osm option (it's normal i think after viewing the code)

the points.txt file contains data like this :

48.8121419271 2.471333313
48.8483642578 2.2652498881
48.8584716797 2.3121388753
48.8147460938 2.4018333435
48.8720540365 2.295249939
48.9736409505 2.4612223307
48.8883056641 2.2926389058
48.8649739583 2.2992777507
48.8420491536 2.36881663

appromatively 65000 lines

i doesn't understand what it means...

Thx for your help !

ZeroDivisionError when input point(s) refer to a single location

In fact, I was trying to make a heat map of points where one point is highlighted/marked with a different color as reference to all others. The idea was to create a map with the other points and another with just one location with a different color (to compose them later). It was when this error happened:
...
File "heatmap.py", in AutoSetScale
pixels_per_degree = float(options.width - padding) / bounding_box_xy.SizeX() * SCALE_FACTOR
ZeroDivisionError: float division by zero.

ps.: BTW, heatmap has been very useful for me. Thanks!

Openstreetmap function broken; easy fix

The --osm flag results in an error. This is because the osmviz library has deprecated OSMManager.createOSMImage in favor of OSMManager.create_osm_image.

Change line 702 to:
image, bounds = osm.create_osm_image((c1.lat, c2.lat, c1.lon, c2.lon),

and it is fixed.

TypeError when loading processed data file

I get an error when trying to load a processed data file. I'm using ActivePython 2.7.2.5 on Windows XP.

No errors saving the file:

C:\>heatmap.py --verbose --width 100 --csv test.csv --save test.data -o test.png
    1108 ms  // reading csv
    1187 ms  // read 10 points
    1187 ms  // Determining scale and scope
    1203 ms  // Scale: 167.319570
    1203 ms  // Map extent: 60.1568688447,24.9011476857,60.2091844949,25.0514534
    1203 ms  // creating an appending matrix
    1203 ms  // processing data
    1453 ms  // combining coincident points
    1625 ms  // saving image (100 x 71)
    2046 ms  // saving data
    2171 ms  // end

But I get this error when loading back:

C:\>heatmap.py --verbose --width 100 --load test.data -o test2.png
    1750 ms  // loading data
Traceback (most recent call last):
  File "M:\bin\heatmap.py", line 1039, in <module>
    main()
  File "M:\bin\heatmap.py", line 938, in main
    matrix = pickle.load(open(options.load))
TypeError: ('__init__() takes exactly 1 argument (2 given)', <class '__main__.Matrix'>, (<type 'float'>,))

I get the same error when changing pickle.dump() and pickle.load() to use 'wb' and 'rb', and/or mode 0 instead of 2.

A quick, simple IDLE test works:

>>> a = [1,2,3]
>>> import cPickle as pickle
>>> pickle.dump(a, open("test", 'w'), 2)
>>> b = pickle.load(open("test"))
>>> a == b
True
>>> a
[1, 2, 3]
>>> b
[1, 2, 3]

KeyError: 'lat' for trkpt with no lat/lon

I have a GPX from my Strava archive that looks like this (I've snipped away most of the track points for brevity; I can share the full file if needed), where the first <trkpt> has no lon or lat attributes:

<?xml version="1.0" encoding="UTF-8"?>
<gpx xmlns="http://www.topografix.com/GPX/1/1" xmlns:gpxdata="http://www.cluetrust.com/XML/GPXDATA/1/0" creator="Wahoo ELEMNT BOLT" version="5.8">
  <metadata>
    <link href="https://gotoes.org/strava/Combine_GPX_TCX_FIT_Files.php">
      <text>GOTOES STRAVA TOOLS</text>
    </link>
    <time>2019-08-31T09:02:36Z</time>
  </metadata>
  <trk>
    <type>Other</type>
    <trkseg>
      <trkpt>
        <ele>31.2</ele>
        <time>2019-08-31T09:02:36Z</time>
        <extensions>
          <gpxdata:temp>0</gpxdata:temp>
        </extensions>
      </trkpt>
      <trkpt lon="24.93841" lat="60.16992">
        <ele>31.2</ele>
        <time>2019-08-31T09:02:37Z</time>
        <extensions>
          <gpxdata:temp>29</gpxdata:temp>
        </extensions>
      </trkpt>
      <trkpt lon="25.34622" lat="60.86568">
        <ele>117.4</ele>
        <time>2019-08-31T13:00:17Z</time>
        <extensions>
          <gpxdata:temp>25</gpxdata:temp>
        </extensions>
      </trkpt>
    </trkseg>
  </trk>
</gpx>

Running heatmap.py on it gives this error:

$ python3 heatmap.py --width 1280 /tmp/test/test.gpx -o out.png
Traceback (most recent call last):
  File "heatmap.py", line 1272, in <module>
    main()
  File "heatmap.py", line 1250, in main
    config.fill_missing()
  File "heatmap.py", line 1199, in fill_missing
    self.shapes = list(self.shapes)
  File "heatmap.py", line 813, in read_file
    for trkseg in track.segments():
  File "heatmap.py", line 115, in _parse
    point = TrackLog.Trkpt(elem.attrib['lat'], elem.attrib['lon'])
KeyError: 'lat'

It probably happens because the first track point doesn't have latitude or longitude attributes (see GPX above).

Can such track points be skipped?

About the data: the problematic file was generated by combining two files using this tool, which is recommended by Strava so will be quite widely used. I don't have the original pre-merge files handy to check if the same issue occurs there or if it was introduced by the merge tool.

Problem using osm

Hello,

I am using your heatmap code and I would like to use, like you did, the osm option. So I tried first with the same as you, the map of Orkney islands (I downloaded some other .gpx file or orkney islands). But I keep getting the following error :

Traceback (most recent call last):
File "heatmap.py", line 1262, in
main()
File "heatmap.py", line 1241, in main
config.fill_missing()
File "heatmap.py", line 1195, in fill_missing
get_osm_background(self, padding)
File "heatmap.py", line 754, in get_osm_background
image, img_bbox_ll = _get_osm_image(bbox_ll, zoom, config.osm_base)
File "heatmap.py", line 691, in _get_osm_image
from osmviz.manager import PILImageManager, OSMManager
File "/home/FUNDP/mcohilis/.local/lib/python3.4/site-packages/osmviz/manager.py", line 60
raise Exception, "UNIMPLEMENTED"
^
SyntaxError: invalid syntax

Here is my command line :
python3 heatmap.py -o orkney.png --extent=58.7,-2.5,59.8,-2 --height 800 --osm orkney.gpx

So it seems like there is a syntax error in osmviz package? Seems weird. I also tried with your other example (photo-points) and I get the same error.
Do you know something about it?

Thanks a lot in advance,

Marie

Is there a possibility to pass dir as input?

Hi!
Thanks for cool script!
I have a question: i have milions of lines with coordinates in different csv files, so I can't merge them into one csv file. Is there a possibility to give dir name as input, but not exact csv file?

Thanks,
Alex.

make memory use independent of input size when possible

At present, all the points or segments are created first, before any processing is done. If instead they were created one at a time and discarded, and when decay is set to 0 or 1, memory use would be independent of the size of the data set and we could handle far more data (bounded perhaps only by floating point rounding problems when adding up heat in the matrix).

Unless the user supplies enough of the scaling, extent, and output size parameters, we'll need to pass through the data once just to compute that, then again for the main processing. Another option would be to scale down the matrix as we expand the bounding box with new data. The output scale would often not be ideal, but for large data sets it might be a better choice.

ZeroDivisionError and IndexError on certain input files

I get these errors with certain degenerate input files using Anaconda Python 2.7.12:

negative-values.txt

C:\>python heatmap.py -o negative-values.png --width 100 negative-values.txt
Traceback (most recent call last):
  File "heatmap.py", line 1263, in <module>
    main()
  File "heatmap.py", line 1250, in main
    image = ImageMaker(config).make_image(matrix)
  File "heatmap.py", line 627, in make_image
    color = self.config.colormap.get(val / maxval)
ZeroDivisionError: float division by zero

negative-and-positive-values.txt

C:\>python heatmap.py -o negative-and-positive-values.png --width 100 negative-and-positive-values.txt
Traceback (most recent call last):
  File "heatmap.py", line 1263, in <module>
    main()
  File "heatmap.py", line 1250, in main
    image = ImageMaker(config).make_image(matrix)
  File "heatmap.py", line 627, in make_image
    color = self.config.colormap.get(val / maxval)
  File "heatmap.py", line 578, in get
    return self.values[int(floatval * (len(self.values) - 1))]
IndexError: list index out of range

python 3 test is failing

myint's pull request triggered a build that passed, but when I merged it in, the new build failed.

Traceback (most recent call last):
File "/home/travis/builds/sethoscope/heatmap/heatmap.py", line 1027, in main()
File "/home/travis/builds/sethoscope/heatmap/heatmap.py", line 1016, in main bounding_box_xy)
File "/home/travis/builds/sethoscope/heatmap/heatmap.py", line 611, in SavePNG img.save(filename)
File "/home/travis/virtualenv/python3.2/lib/python3.2/site-packages/PIL/Image.py", line 1399, in save raise KeyError(ext) # unknown extension
KeyError: '.ppm'

animation

Hi !

I am using this code to make an animation.
It works well but I would like to know if it is possible to make the points ephemeral (because it is temporal) ?

Here is my command line :
python3 heatmap.py -b black -r 4 -W 1200 -d 0.1 -I osmt.png --extent=49.7,3.6,50.8,6.0 -G Blue.png -a -o Namur100-5.mpg -P mercator Namur100_copie.txt.
Is it possible to change something in this line or should I change something in the code ?

Thank you in advance,
Kind regards

Add required dependencies to README

I recently came across heatmap and have had a bit of trouble getting the correct dependencies installed. Once I find out what is required to work with the code, I'll send a PR.

not useful as a library

It's useful as a script, but not as a library. It should be refactored so the following is possible:

• Operate exactly as it does, with the options passed in code, rather than via command line flags.
• Input data provided via code, rather than an external file.
• Return image data, rather than writing it to a file.
• Support incremental rendering controlled by caller. Rewrite animation support using that interface.

And it should be generally tidy, which it currently is not.