Skip to content

Layergroup Workflow

Jump to bottom
strk edited this page Nov 21, 2013 · 9 revisions

Layergroup workflow

this document explains how CartoDB tiles are served using Multilayer API (also called layergroup API) https://github.com/CartoDB/Windshaft/wiki/Multilayer-API.

To explain it an example scenario is provided where two clients want to fetch tiles for the same layergroup, that's it, they want to render a map of a table called european_countries where the countries are rendered in white;

The layergroup as explained in Multilayer API document should be like:

{
  "version": "1.0.1",
  "layers": [{
    "type": "cartodb",
    "options": {
      "cartocss_version": "2.1.1", 
      "cartocss": "#layer { polygon-fill: #FFF; }",
      "sql": "select * from european_countries"
    }
  }]
}

We take into account that is the first time that map is requested and all the caches are empty.

workflow diagram

workflow image

NOTE: the image is a fireworks png.

workflow steps description

  1. Client 1 builds the json document for the layergroup (described before) using cartodb.js. A POST to the tiler is done. Important things here:

2.- windshaft reads the json document and goes to the SQL API to get two things:

  1. which tables are involved in the queries inside layergroup document
  2. what is the last time (last_updated) those tables where modified

With the last_updated and a hash(SQL + CartoCSS) a layergroup_id is generated:

{ layergroup_id: 8d6096d7ee7352bbae40074ca8dee352:12345678}

first part 8d6096d7ee7352bbae40074ca8dee352 is the hash, second part, after the ':' is the ``last_update`

The tiler code extracting "last_updated" is here: http://github.com/CartoDB/Windshaft-cartodb/blob/1.5.0/lib/cartodb/server_options.js#L295

NOTE: actually windshaft should request SQL API though varnish but is disabled due a bug.

Here's the code extracting https://github.com/CartoDB/Windshaft-cartodb/blob/1.5.0/lib/cartodb/server_options.js#L96

3.- windshaft returns a json with what we call layergroup_id and the last_updated (not used currently but useful for debugging pourposes)

4.- client 1 builds the tiles url using the CDN url provided in vizjson or by the user. If there is no CDN url, default CDN url is used (https://github.com/CartoDB/cartodb.js/blob/develop/src/cartodb.js#L18). If no_cdn option is used with cartodb.js the url generated is direct to the tiler like:

http://user.cartodb.com/tiles/layergroup/:z/:x/:y.png

and request the urls. There is a cache miss since the tiles are not cached: - browser cache - CDN - varnish

5.- windshaft fetch the data for that layergroup using the layergroup_id (from redis right now), calls mapnik with some XML (generated from SQL and CartoCSS in the layergroup), mapnik fetch the data from postgres and returns the png and windshaft serves the tile.

important things:

  • windshaft set headers to cache the tile forever, there are the interesting headers:

    Cache-Control: public,max-age=31536000
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT

  • the tile is cached in Varnish and the CDN

  • important note: between CDN and varnish there is a nginx proxy (doc link needed) to route request from CDN to the correct user server

6.- client 2 request the layergroup (see 1)

7.- see 2

8.- see 3. Notice the layergroup is the same since the neither SQL or CartoCSS have changed. We supose the tables weren't changed (not inserts, updates, deletes, privacy changes, schema changes...)

9.- client 2 builds the url for the tiles, the same urls than client 1 fetches them

10.- the tiles are returned by CDN because they were previously cached in 5

11.- the client 1 does an insert in the table (european_countries) so the last_updated for that table is changed

12.- the same than 1

13.- same than 2 but in this case last_update changes since the table was updated in 11

14.- same than 3 but the layergroup_id is different, the last_updated part has changed. Notice the first part (the hash) didn't because the layergroup is the same

and at this point the workflow returns to 4

Clone this wiki locally