Coder Social home page Coder Social logo

topojson-server's Introduction

TopoJSON

TopoJSON is an extension of GeoJSON that encodes topology. Rather than representing geometries discretely, geometries in TopoJSON files are stitched together from shared line segments called arcs. This technique is similar to Matt Bloch’s MapShaper and the Arc/Info Export format, .e00.

TopoJSON eliminates redundancy, allowing related geometries to be stored efficiently in the same file. For example, the shared boundary between California and Nevada is represented only once, rather than being duplicated for both states. A single TopoJSON file can contain multiple feature collections without duplication, such as states and counties. Or, a TopoJSON file can efficiently represent both polygons (for fill) and boundaries (for stroke) as two feature collections that share the same arc mesh. See How To Infer Topology for a visual explanation of how TopoJSON works. See Command-Line Cartography for an introduction to TopoJSON and related tools. See TopoJSON Format Specification for the format specification.

To further reduce file size, TopoJSON can use quantized delta-encoding for integer coordinates. This is similar to rounding coordinate values (e.g., LilJSON), but with greater efficiency and control over loss of information. And like GeoJSON, TopoJSON files are easily modified in a text editor and amenable to gzip compression.

As a result, TopoJSON is substantially more compact than GeoJSON, frequently offering a reduction of 80% or more even without simplification. Yet encoding topology also has numerous useful applications for maps and visualization above! It allows topology-preserving shape simplification, which ensures that adjacent features remain connected after simplification; this applies even across feature collections, such as simultaneous consistent simplification of state and county boundaries. Topology can also be used for Dorling or hexagonal cartograms, as well as other techniques that need shared boundary information such as automatic map coloring.

Installing

If you use NPM, npm install topojson. Otherwise, download the latest release. You can also load directly from d3js.org as a standalone library. AMD, CommonJS, and vanilla environments are supported. In vanilla, a topojson global is exported:

<script src="https://unpkg.com/topojson@3"></script>
<script>

var topology = topojson.topology({foo: geojson});

</script>

Try topojson in your browser.

API Reference

topojson-server's People

Contributors

fil avatar mbostock avatar mfogel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

topojson-server's Issues

Solution for ERR_STRING_TOO_LONG?

I'm trying to run geo2topo on the GADM dataset: 6 layers, 7.3 GB in total. My problem is that even with raising the memory by node --max-old-space-size=20000, I cannot convert multiple layers of that dataset.

The process always exits with the following message:

buffer.js:763
    return this.utf8Slice(0, this.length);
                ^

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.toString (buffer.js:763:17)
    at JSON.parse (<anonymous>)
    at ReadStream.<anonymous> (.../country-levels/node_modules/topojson-server/bin/geo2topo:107:46)
    at ReadStream.emit (events.js:323:22)
    at endReadableNT (_stream_readable.js:1204:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_STRING_TOO_LONG'

The server has 64 GB of RAM, so it should be able to handle this in memory.

invalid array length Allocation failed - JavaScript heap out of memory

I'm using geo2topo with 2-3 GB ndjson, but it results in an exception.

invalid array length Allocation failed - JavaScript heap out of memory

Using --max-old-space-size=40000 on a 64 GB server. Memory usage tops at around 13 GB.

I tried node versions 8, 10, 12, 13, same results. I tried vm.max_map_count=655300 based on this SO answer: https://stackoverflow.com/a/59923848/518169, it didn't make a difference.

Is there any way to fix this bug? (note: I'm not asking a question which could be answered on StackOverflow, I'm reporting that this script is terminating in an error even though there is plenty of available memory).

geo2topo fails on large files

I'm trying to produce a topojson file of all ZCTAs in the US. I converted the Census TIGER data to geojson (1.3 GB), and then when I try to convert it to topojson, I get the error below.

$ geo2topo zcta5.geo.json >  zcta5.topo.json 
buffer.js:644
    return this.utf8Slice(0, this.length);
                ^

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.toString (buffer.js:644:17)
    at JSON.parse (<anonymous>)
    at ReadStream.<anonymous> (/Users/germuska/.nvm/versions/node/v10.4.1/lib/node_modules/topojson/node_modules/topojson-server/bin/geo2topo:107:46)
    at ReadStream.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1081:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

As my next step, I extracted roughly half the data (based on ZCTA ids beginning with 0-4). This produced a 609M geojson file. When executing geo2topo on that file, I got a different error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory from the v8 JsonParser

I tried again, splitting into 5 groups, and got the same error on a geojson file of around 300MB.

When I split into 10 geojson files, they all make it through correctly, but now I realize that I don't know a simple way to put them all back together again!

Can anything be done to produce topojson from large geodata files?

customized top level properties are thrown out

notes

  1. working without pipes but file based - input is the resulting output file from the previous converting step
  2. issue is even valid for geostitch

tooling and versions
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected]
$ node --version: v6.11.2

reproduce

  1. using "ne_10m_admin_0_countries.shp" from unzipped natural earth
  2. $ shp2json -n "./ne_10m_admin_0_countries.shp" > "ne_10m_admin_0_countries-ndjson.geojson"
  3. $ ndjson-map 'i = d.properties.ISO_A3, d.iso = i === "-99" ? undefined : i, i = d.properties.NAME_LONG, d.name = i === "-99" ? undefined : i, delete d.properties, d' < "ne_10m_admin_0_countries-ndjson.geojson" > "ne_10m_admin_0_countries_country_codes.json"
  4. $ geo2topo -q 1e5 -n countries="./ne_10m_admin_0_countries_country_codes.json" > "./ne_10m_admin_0_countries.topojson"

if you then look into "ne_10m_admin_0_countries.topojson", you should see the properties "iso" and "name" are gone, against many tutorials on the internet like this: medium, while they are in "ne_10m_admin_0_countries_country_codes.json"

could anybody confirm and maybe explain why this is happening?

here is a rather old sof discussion

Reconverting topojson that was outputted by topojson-client

Hi,

I have a geojson object that was converted from a topjson using the topojson-client's feature method. When I try to reconvert the same object using the topology method of the topojson-server, I get all objects as undefined.

How can I reconvert a geojson that was outputted by the feature method back to topojson? I want to do so to use the merge method to merge certain paths from the topojson-client.

Geo2topo without files

would there be a way to use geo2topo using string input rather than files?

geo2topo "some geojson" and output result in the console?

topology generated incorrectly for shapes with holes that contain loops

In processing and simplifying census shapes, we ended up creating shapes that have a specific characteristic that appears to be handled incorrectly by the code that converts shapes to topology. The attached file demonstrates the issue (.txt ending for github).

Essentially, one shape contains a hole that ends up touching itself at a single point - it is essentially two holes that share a single point but described with a single ring in the polygon structure. All other processing is fine with this structure. The other shape is a multi-polygon that contains two separate polygons that fill the holes.

When converting these two shapes to a topology, the arcs that are part of the holes are treated as "lines" during the dedup process and therefore do not end up matching the "rings" that describe the polygons. The result is that there are two separate arcs describing the same series of points.

During merge process (in topojson-client), the result of the merge is a multi-polygon that includes the holes and the polygons that fill them because the arcs are not recognized as shared.

We have a forked version (in @dra2020/topojson-server) that implements a fix that works for us. Essentially at the point where dedupLine is going to be called, we first check if the start and end points of the line are the same and instead call dedupRing.

twoshapes.geojson.txt

should linestrings which fold back on themselves be cut?

In this sample GeoJSON a LineString goes from A to B to C to B to A.

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "type": "LineString",
        "coordinates": [[0, 0], [1, 0], [2, 0], [1, 0], [0, 0]]
      }
    }
  ]
}

I was expecting that encoded as TopoJSON the arc A to B to C would be saved as the arc A-B-C and the LineString saved as that arc + reverse of the arc, but instead it's just one arc.

{"type":"Topology","objects":{"abcba":{"type":"GeometryCollection","geometries":[{"type":"LineString","arcs":[0]}]}},"arcs":[[[0,0],[1,0],[2,0],[1,0],[0,0]]],"bbox":[0,0,2,0]}

Is this the intended output or is it a bug?

Documentation refers to a mystery us-states.json GeoJSON file

The README refers to a us-states.json file, but there does not seem to be any such file in the repo, nor any link to where it is (if it is intended to be from some other location).

As such, it is harder than ideal for a new user to figure out how the library works and be sure what kind of expected data is required by way of example.

Migrate topojson.filter with spherical coordinate-system from previous API

Hi,

I need some help with moving to the new API v3 of Topojson.

In my working code I have the following line:

topojson.filter(topology, { "coordinate-system": "spherical" } );

How can I do the same in the new API v3?
By looking at the documentation I understand that the new topojson.filter(topology[, filter]) accepts a filter function rather than an object with a "coordinate-system"?

Thanks for any help or advice.

ReferenceError: Promise is not defined

I get (using node on Ubuntu 14.04):

$ geo2topo us-states.js > us.topo

/usr/local/lib/node_modules/topojson-server/bin/geo2topo:25
Promise.all(commander.args.map(read)).then(write).catch(abort);
^
ReferenceError: Promise is not defined
    at Object.<anonymous> (/usr/local/lib/node_modules/topojson-server/bin/geo2topo:25:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

https://stackoverflow.com/a/35640553
implies that there could be a missing dependency, perhaps, but I'm not a node expert.

I have:

$ /usr/bin/node -v
v0.10.25

$ npm -g ls
/usr/local/lib
├── [email protected]
└─┬ [email protected]
  └── [email protected]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.