Interactive Map with d3.js

DECEMBER 07, 2013

This is a step by step tutorial of how to make responsive, interactive and zoomable map with d3.js. Converting Shapefile into GeoJSON then Topojson, you can build web/mobile-ready map application.


This is an interactive map. Please click United States, Japan or their states/prefectures to see zoom animation.


1 Gather Geo Data

d3.js doesn't come with geo data, so let's get country, state/province and city data first. Natural Earth is one of the best sources for Earth data. When you go to the download page, there are three data scales: 1:10m, 1:50m and 1:110m. 1:10m has the most detailed data but is the largest file size, which is not usually suitable for web application data. However, 1:50m or 1:110m currently doesn't contain state level data for any countries, except for US and Canada. In order to get all states and provinces for all countries, we have no choice but to use 1:10m data. Let's download the 1:10m files:

1:10m Cultural Vectors Admin 0 - Countries Download without boundary lakes Version 3.0.0
Admin 1 - States, Provinces Download without large lakes Version 3.0.0
Populated Places Download populated places Version 3.0.0


2 Shapefile    GeoJSON    Topojson

Once you download and unzip them, you can find a bunch of files with different file extensions. We'll use a file that ends with '.shp', called shapefile. In order to use the data on web application, we need to convert a shapefile into JSON. First, we transform shapefile into GeoJSON. Then, convert GeoJSON into Topojson. I'm not explaining details for each file, but basically, we are trying to reduce data size while maintaining certain topology quality. Here is the file size comparison for the country data:


Install GDAL

To convert a shapefile into GeoJSON, we'll use a command line tool, ogr2ogr in Geospatial Data Abstraction Library - GDAL. If you are OS X user and have brew, install gdal like this:

> brew install gdal


Install Topojson

Another CLI tool that we use will use is topojson, which converts GeoJSON to Topojson. Topojson CLI requires node.js, so install it first. Then install topojson by typing:

> npm install -g topojson


3 Convert Files

Let's start with the countries shapefile. Go to ne_10m_admin_0_countries_lakes directory and run the following:

> ogr2ogr -f GeoJSON countries.json ne_10m_admin_0_countries_lakes.shp


This will create GeoJSON file called "countries.json" from the shapefile. When you open the file, you can see lots of array of numbers along with attribute names, like ADMIN, SU_A3, NAME... And you can find all countries including Antarctica there.


Exclude Antarctica

For this demo, we don't really need Antarctica (sorry, no offense). As an exercise, let's exclude Antarctica from the map. In ogr2ogr, you can use SQL-like conditional statement to query certain features. To exclude just Antarctica:

> ogr2ogr -f GeoJSON -where "SU_A3 <> 'ATA'" countries.json ne_10m_admin_0_countries_lakes.shp


Pick attributes

One of the reasons why GeoJSON file is so big is that it contains lots of attributes. We don't need everything, so let's pick only necessary ones, as we convert it to Topojson. We just picked SU_A3 and NAME:

> topojson --id-property SU_A3 -p name=NAME -p name -o countries.topo.json countries.json


The first -p converts attribute name from uppercase to lowercase, and the second -p picks up 'name' as an attribute. Topojson has less than 10% of the GeoJSON file size, but it still has 2.5MB, which is heavy for web applications. We need to reduce the file size further.


4 Simplify Files

We removed lots of attributes from GeoJSON, but it still has lots of vertices. We need to remove some vertices from each country feature, while not degrading topology. How to do that?


A. Use ogr2ogr -simplify

ogr2ogr CLI has an option called -simplify, which smooth polygons and reduce the file size significantly. Let's try this and followed by topojson command:

> ogr2ogr -f GeoJSON -simplify 0.2 -where "SU_A3 <> 'ATA'" countries.json ne_10m_admin_0_countries_lakes.shp
> topojson --id-property SU_A3 -p name=NAME -p name -o countries.topo.json countries.json


Wow, the Topojson file got shrunk from 2.5MB to 307KB! However, if we visualize the polygon data, we can see the problem. Do you see boundaries are not lined up? The tool successfully simplify topology for each feature, but couldn't preserve layer topology. Lines should be aligned between countries.


B. Need sophisticated algorithm: mapshaper.org

Fortunately, there is a great online tool, mapshaper.org. It supports Douglas-Peucker and Visvalingam algorithms and preserves layer topology. In addition, it supports shapefile, GeoJSON and Topojson format as import as well as export.


We can upload a shapefile, simplify it online and export as Topojson. However, there is one problem: it loses attributes. We want to maintain ids and names. So, let's do this: convert Shapefile to GeoJSON with ogr2ogr tool. Upload a GeoJSON file and simplify it at mapshaper.org. Export it as GeoJSON again. Then, convert GeoJSON to Topojson with topojson tool. I used 3% for simplification.


The layer topology problem was resolved while preserving attributes, and it looks much better (below). In addition, the file size was further shrunk to 173KB. We can use this data for web applications.


5 Create State Topojson

Exactly the same way we did for countries, let's create states and provinces Topojson file. For the demo purpose, we only support two countries here, but you can extend this approach to all countries. The state shapfile has all states/provinces data.


I picked two countries: USA and Japan. Since the state data will be lazy loaded, keep two countries' states in separate files.

> ogr2ogr -f GeoJSON -where "gu_a3 = 'USA'" states.json ne_10m_admin_1_states_provinces_lakes.shp
> mv states.json states_usa.json
> ogr2ogr -f GeoJSON -where "gu_a3 = 'JPN'" states.json ne_10m_admin_1_states_provinces_lakes.shp
> mv states.json states_jpn.json


The reason why we use the output file name 'states.json' for both countries is that the name is also used as object in the data. It's easier if object names are same across all countries in visualization.


Upload these two GeoJSON files to mapshaper.org again, simplify them, and export them back as GeoJSON. Then convert them to Topojson:

> topojson --id-property adm1_cod_1 -p name -o states_usa.topo.json states_usa.json
> topojson --id-property adm1_cod_1 -p name -o states_jpn.topo.json states_jpn.json


Two Topojson outputs are 21KB and 11KB respectively. Fantastic!


6 Create City Topojson

City Shapefile doesn't contain polygon but has city coordinates. We'll extract major cities in USA and Japan. City popularity is scaled in scalerank. This is the scale for map zoom level. One means it's visible at the most zoomed out position. You cannot see scalerank = 10, unless you zoom in 10 steps.

When you extract cities, make sure to get state as well. This is used for filtering later on. These are ogr2ogr and topojson commands:

> ogr2ogr -f GeoJSON -where "ADM0_A3 = 'USA' and SCALERANK <= 4" cities.json ne_10m_populated_places.shp
> topojson -p name=NAME -p state=ADM1NAME -p name -p state -o cities_usa.topo.json cities.json
...
> ogr2ogr -f GeoJSON -where "ADM0_A3 = 'JPN' and SCALERANK <= 6" cities.json ne_10m_populated_places.shp
> topojson -p name=NAME -p state=ADM1NAME -p name -p state -o cities_jpn.topo.json cities.json


7 Visualize with d3 Geo

Now that we got all data we need, let's make an interactive map with d3.js. Here is the entire code. There are many examples and documentations out there, explaining how to use d3.js, so I don't go through it, but this mainly uses d3's geo library. It's written quick and dirty, but I hope you'll get a gist of it.

<script src="//d3js.org/d3.v3.min.js"></script>
<script src="//d3js.org/topojson.v1.min.js"></script>
<script>
var m_width = $("#map").width(),
    width = 938,
    height = 500,
    country,
    state;

var projection = d3.geo.mercator()
    .scale(150)
    .translate([width / 2, height / 1.5]);

var path = d3.geo.path()
    .projection(projection);

var svg = d3.select("#map").append("svg")
    .attr("preserveAspectRatio", "xMidYMid")
    .attr("viewBox", "0 0 " + width + " " + height)
    .attr("width", m_width)
    .attr("height", m_width * height / width);

svg.append("rect")
    .attr("class", "background")
    .attr("width", width)
    .attr("height", height)
    .on("click", country_clicked);

var g = svg.append("g");

d3.json("/json/countries.topo.json", function(error, us) {
  g.append("g")
    .attr("id", "countries")
    .selectAll("path")
    .data(topojson.feature(us, us.objects.countries).features)
    .enter()
    .append("path")
    .attr("id", function(d) { return d.id; })
    .attr("d", path)
    .on("click", country_clicked);
});

function zoom(xyz) {
  g.transition()
    .duration(750)
    .attr("transform", "translate(" + projection.translate() + ")scale(" + xyz[2] + ")translate(-" + xyz[0] + ",-" + xyz[1] + ")")
    .selectAll(["#countries", "#states", "#cities"])
    .style("stroke-width", 1.0 / xyz[2] + "px")
    .selectAll(".city")
    .attr("d", path.pointRadius(20.0 / xyz[2]));
}

function get_xyz(d) {
  var bounds = path.bounds(d);
  var w_scale = (bounds[1][0] - bounds[0][0]) / width;
  var h_scale = (bounds[1][1] - bounds[0][1]) / height;
  var z = .96 / Math.max(w_scale, h_scale);
  var x = (bounds[1][0] + bounds[0][0]) / 2;
  var y = (bounds[1][1] + bounds[0][1]) / 2 + (height / z / 6);
  return [x, y, z];
}

function country_clicked(d) {
  g.selectAll(["#states", "#cities"]).remove();
  state = null;

  if (country) {
    g.selectAll("#" + country.id).style('display', null);
  }

  if (d && country !== d) {
    var xyz = get_xyz(d);
    country = d;

    if (d.id  == 'USA' || d.id == 'JPN') {
      d3.json("/json/states_" + d.id.toLowerCase() + ".topo.json", function(error, us) {
        g.append("g")
          .attr("id", "states")
          .selectAll("path")
          .data(topojson.feature(us, us.objects.states).features)
          .enter()
          .append("path")
          .attr("id", function(d) { return d.id; })
          .attr("class", "active")
          .attr("d", path)
          .on("click", state_clicked);

        zoom(xyz);
        g.selectAll("#" + d.id).style('display', 'none');
      });      
    } else {
      zoom(xyz);
    }
  } else {
    var xyz = [width / 2, height / 1.5, 1];
    country = null;
    zoom(xyz);
  }
}

function state_clicked(d) {
  g.selectAll("#cities").remove();

  if (d && state !== d) {
    var xyz = get_xyz(d);
    state = d;

    country_code = state.id.substring(0, 3).toLowerCase();
    state_name = state.properties.name;

    d3.json("/json/cities_" + country_code + ".topo.json", function(error, us) {
      g.append("g")
        .attr("id", "cities")
        .selectAll("path")
        .data(topojson.feature(us, us.objects.cities).features.filter(function(d) { return state_name == d.properties.state; }))
        .enter()
        .append("path")
        .attr("id", function(d) { return d.properties.name; })
        .attr("class", "city")
        .attr("d", path.pointRadius(20 / xyz[2]));

      zoom(xyz);
    });      
  } else {
    state = null;
    country_clicked(country);
  }
}

$(window).resize(function() {
  var w = $("#map").width();
  svg.attr("width", w);
  svg.attr("height", w * height / width);
});
</script>


You can download Javascript source and TopoJson data files in github here.


Related Posts

Flight Animation with d3.js

This is a step by step tutorial of how to create flight animation on the map, using d3.js Geo and Transition. Everything is rendered in SVG, and you can learn from creating a world map, plotting airports to animating flying planes.

Responsive d3.js

It's increasingly important to make d3.js charts responsive for touch-based devices such as mobile and tablet. While this approach doesn't solve everything, it's one way of making SVG based charts dynamic and responsive.

Prettyprint in Bootstrap 3

Here is how to pretty print code, using google code prettify and Bootstrap 3.