tilegarden
🗺️
🌻
Contents
About
Tilegarden is a serverless tile-rendering tool using Mapnik, built for AWS Lambda. Serve custom-generated map tiles without having to worry about server maintenance, scaling, or paying for resources that aren't being accessed.
Usage
Deployment to AWS
A deployed instance of Tilegarden consists of several AWS resources:
- An AWS Lambda function which handles the tiling logic.
- An API Gateway API that acts as an endpoint and routes HTTP requests to the lambda function.
- A CloudFront distribution that acts as a proxy in order to circumvent some issues that API Gateway has serving image content.
- Specify your map in a map configuration file and place it in
src/tiler/src/config/(where you can find pre-existing examples). A full specification for Carto's .mml format can be found here. - Specify your production credentials and lambda function settings in
.env. The following variables are required:AWS_PROFILE: the name of the AWS user profile you want to deploy your project as. You may have this set already in your environment, otherwise you'll want to set it to one of the names of the sets of credentials specified in~/.aws/credentials. Defaults to “default”.PROJECT_NAME: the name of your project. This must be unique among the functions you currently have deployed to AWS Lambda.PROD_TILEGARDEN_*: the credentials necessary to connect to your production database.LAMBDA_REGION: the AWS region to which you want your lambda functions deployed.LAMBDA_ROLE: the name/ARN of the IAM role to give your lambda. Leave this blank to have one created for you automatically.LAMBDA_SUBNETSandLAMBDA_SECURITY_GROUPS: only required if you need to connect to other AWS resources (such as an RDS instance), in which case these should match the values that those resources have.- See below for more options.
- See below to make sure your AWS profile has the requisite permissions for automated deployment.
- Run
./scripts/deploy --newto create new instances of the necessary AWS resources.- Updates to an existing project can be deployed by running
./scripts/deploy(no “--new”). Note that only certain function settings (LAMBDA_TIMEOUTandLAMBDA_MEMORY) are updated by this command, changes to IAM Role/subnets/security groups/etc. must be made manually on the AWS dashboard.
- Updates to an existing project can be deployed by running
- All deployed resources can be removed by running
./scripts/destroy.
Additional Configuration Options
USER: Left unspecified so that the code knows who the local user is from inside docker containers. Gets tacked on to the end of the project name at deployment.LAMBDA_TIMEOUT: Time (in seconds) before your lambda function is automatically cancelled (maximum 300). You'll probably need to modify this value as tile rendering can take longer than AWS's default timeout, depending on the complexity of the data you're rendering.LAMBDA_MEMORY: Amount of memory (in MB) to allocate per function. Must be a multiple of 64, minimum 128, maximum 3008. As withLAMBDA_TIMEOUT, more complex data requires more allocated memory, but 128MB should be enough in most scenarios.
Required AWS Permissions
The AWS profile used for deployment must have at least the following policies and permissions:
CloudFrontFullAccessAmazonAPIGatewayAdministratorAWSLambdaFullAccess- If you don't specify a pre-existing execution role, at least the following IAM permissions
iam:DeleteRolePolicyiam:DeleteRoleiam:CreateRoleiam:AttachRolePolicyiam:PutRolePolicy
- Your lambda functions will have whichever permissions the profile used to create them has, so keep this in mind if certain policies are required to access your data store.
Features
Configuration Selection and Storage
Multiple map configuration .xml files can be included in your project's src/tiler/src/config to be dynamically loaded at run-time. Use the config path parameter with the name of your configuration file (without the file extension) to select which file gets loaded when rendering tiles.
- Example: if you have a configuration file named
my-good-map.xml, you can tell Tilegarden to use it with the endpoint/tile/my-good-map/{z}/{x}/{y}.png
Configuration files can also be optionally loaded from an AWS S3 bucket, which allows you to add/remove/replace available maps without having to redeploy your entire project. To do so, add &s3bucket=<bucket-name> to your query string, and treat the config parameter as the desired config file's key (omitting the .xml extension). You can use the build-xml script to generate config files for this purpose without having to spin up the entire development environment, run ./scripts/build-xml --help for more info.
- Accessing config files stored in S3 requires you to have deployed your Tilegarden instance using an AWS profile with read/write permissions for the desired S3 bucket. These permissions are passed on to the AWS Lambda function. You must also manually add the
AmazonS3ReadOnlyAccesspermission to the IAM role that gets created for your lambda, or otherwise use a custom role that has that permission, sinceclaudiadoesn't add it during creation. build-xmlhas a-b/--s3_bucket <name>flag which automatically uploads built files to S3 instead of saving them to disk. This requires you to have the AWS CLI installed on your machine, and uses the AWS credentials you've specified in.env.
Datasources
Tilegarden supports the use of any geospatial data source that Mapnik/Carto does (shapefile, postgis, pgraster, raster). However, bear in mind that only PostGIS data sources support custom queries, and are thus the only ones that allow you to perform additional filtering on your data (see Filters). Also, attempting to bundle large local data files into your Tilegarden deployment could lead to rejection by AWS Lambda due to size restrictions.
Map Styles
Map styles are specified in CartoCSS, the specification for which can be found here. The default stylesheet is located at src/tiler/src/config/style.mss. Multiple stylesheets can be used by adding them to the parameter “Stylesheet” in src/tiler/src/config/map-config.mml. Each .mss “class” refers to a map layer (its “id” property), specified in map-config.mml.
Filters
Filtering your geospatial data can be done in two ways:
Custom Layer Queries
Filters can be specified on the back-end by altering the query of one of your map's layers in src/tiler/src/config/map-config.mml, and can then be fetched through the API. To create a filtered layer, modify the “table” property of the layer to have the format (QUERY) as VARIABLE.
- Make sure each layer has a unique “id” value.
- Make sure to include the geometry column in this query, along with whatever columns you are filtering by.
- Example:
(SELECT geom,homes FROM table_name WHERE homes='big') as qwould create a filtered layer for all points that have a “homes” value of “big” in your database.
Layers can be accessed from the API by adding ?layers=["layer1","layer2",etc.] (a valid JSON array)as a query string to a tile or UTF grid URL. If no layers are specified in the query string, all layers specified in map-config.mml are displayed simultaneously.
Client-specified Filtering
Layers can also be filtered dynamically by passing a JSON object as a value of a layer in the ?layers array. The schema for this sort of request looks like the following:
[
{
"name":"layer1",
"mode": "AND" // optional, operator to use to combine filters, can either be AND or OR, defaults to AND
"filters": [ // optional, applies no filters if missing
{
"col":"column1", // table column you wish to filter by
"val":"value1", // value to filter on
"op": ">" // optional, boolean operator to use when comparing col to val, = by default
}
]
}
]
The resulting url would have the query string ?layers=[{"name":"layer1","filters":[{"col":"column1","val":"value1","like": true}]}]. Filtering layers in this way appends WHERE col='val' to the existing query defined for the layer. Defining multiple filters in the "filters" array of the layer object ANDs the filters together in the query.
- NOTE: Be careful with
%s in your JSON! Browsers/Tilegarden automatically handle most percent-encodable characters, but%s especially can cause your JSON to become unparsable. CallencodeURIComponent()on your filter objects for safety.
UTF Grids
UTF grids can be generated at the url /grid/{z}/{x}/{y}?utfFields=field1,field2,etc., where “utfFields” is a comma-separated list of one or more table columns from the database. Each grid url MUST have a “utfFields” query string. If you have already modified the table query of a layer to specify a filter, be sure the include the utf field column in the query. Ex. (SELECT geom,homes,dog... ), if you want to base a UTF grid off the value of “dog”.
Vector Tiles
Vector tiles of your data can be generated at the endpoint /vector/{z}/{x}/{y}. Filtering by layer works via query string but can also be handled client side, as can styling. See here for more info about the Mapbox Vector Tile specification.
Local Development
Dependencies: docker, docker-compose
- Clone the Tilegarden repository to your machine.
- Add any development data:
- Add your geospatial data to the
data/directory. .zipped shapefiles and gzipped SQL dumps will be loaded in to the development database automatically. - Make sure the name of the file matches the name of the SQL table that contains the data.
- Add your geospatial data to the
- Configure your map:
- Open the file
src/tiler/src/config/map-config.mml. This is where you specify your map settings. A full specification for Carto's .mml format can be found here. - Of particular note is the “Layer” property, which specifies the layers of your map (as an array). Odds are you won't have to change the target srs (at the top of the file), but make sure that the srs of each layer is specified properly. By default, all layers of your map are displayed at once, but different combinations of layers can be selected using the
layersquery string. See Filters for more info.
- Open the file
- Create a copy of
env-templatenamed.env. This contains production-specific configuration options and doesn't need to be filled out right now (but must exist in order to run the development server). - Run
./scripts/updateto create your Docker containers and populate the development database. The optional flag--downloadwill download the data sets used for our demos. - Run
./scripts/serverto start the development server. It will be available at localhost:3000, your tiles can be found at/tile/{z}/{x}/{y}.png. - The local development server exposes a node.js debugger, which can be attached to by Chrome DevTools or your IDE of choice (see below).
- (Optional) If you downloaded the example data using
./scripts/update --download, openingindex.html(or any of the pages indemo/) should show you working demos.
- (Optional) If you downloaded the example data using
- The local development environment can be reset by running
./scripts/clean, which removes all development artifacts including Docker containers and volumes.
Debugging
The local development server exposes a Node.js process WebSocket that can be attached to your IDE of choice to step through and debug your code. Here are instructions on how to do so using Google Chrome's Dev Tools:
- Start the development server with
./scripts/server. - Open Google Chrome and navigate to
about:inspect. - There should be an option listed as something along the lines of "Target (v8.10.0)" with the Node.js logo and a path like
file:///home/tiler/bin. Click "inspect", underneath. - A new window will open up. Enter
ctrl+pto open a search bar that lets you navigate to your source code. The transpiled code insrc/tiler/bin/is what is actually being tracked by the debugger, not the source code insrc/tiler/src/.
The 34.1M bundle size doesn't account for mapnik (the tool I used couldn't figure out what to make of the C++ bit of it, I think), which should push the whole thing over AWS's limit of 70M (apparently).
409 Dec 30, 2022
52 Jul 21, 2022
1.8k Dec 28, 2022
27 Nov 9, 2022
125 Dec 20, 2022
2 Oct 25, 2021
9.4k Jan 7, 2023
50 Dec 12, 2022
1.2k Dec 30, 2022
111 Dec 15, 2022
203 Dec 7, 2022
379 Dec 23, 2022
266 Dec 17, 2022
3k Jan 4, 2023
1.4k Jan 2, 2023
39 Jun 20, 2022