PEARL (Planetary Computer Land Cover Mapping) Platform API and Infrastructure

Overview

PEARL API & Infrastructure

image

PEARL is a landcover mapping platform that uses human in the loop machine learning approach. This repository contains the API and infrastructure to run the platform. The frontend is maintained at pearl-frontend.

Overview

PEARL infrastructure uses Kubernetes to manage resources and provide on-demand GPU support. The following is a high-level architecture. A websocket router manages available GPU/CPU and allocates them to clients.

pearl backend

This repository contains code for the api, gpu, socket and tiles services in the services directory. Code to deploy all the services and infrastructure on Azure can be found in the deployment directory. We have also included terraform resource definitions in deployment/terraform and a Helm chart in deployment/helm. The Helm chart is published separately at https://devseed.com/pearl-helm-chart and instructions for using the helm chart can be found here.

The machine learning components are documented here.

Development

Initial development can be bootstrapped by running all of the services via docker-compose

docker-compose up --build -d

This script will ensure that you have a postgres database set up, and will configure and start all necessary services locally for a fully functional dev environment

API Documentation

API documentation can be found by opening the following location in your browser file://<path-to-git-repo>/lulc-infra/api/doc/index.html

or ideally, once your development environment has been started, API documentation can be found by navigating to http://localhost:2000/docs.

Deployment and CI

Notes on deploy process and CI integration can be found here.

You can also helm to deploy directly onto a Kubernetes cluster. Instructions can be found here.

Note: This has currently been tested using AKS on Azure, but should ideally work on any Kubernetes cluster with minimal changes.

Debugging and Logs

Instructions for accessing the Grafana UI to inspect application logs can be found here

LICENSE

MIT

Comments
  • Irregular AOIs

    Irregular AOIs

    Context

    Adds support for non-rectangular AOIs

    Investigation

    • AOI Table already supports arbitrary polygons
    • Inference Websocket Messages already support arbitrary polygons

    Closes: https://github.com/developmentseed/pearl-frontend/issues/21

    opened by ingalls 1
  • Model Storage Checks

    Model Storage Checks

    Context

    Ensure Model.storage & Model.active are true on project creation

    Closes: https://github.com/developmentseed/pearl-backend/issues/10

    cc/ @geohacker

    bug 
    opened by ingalls 0
  • Codebase Updates

    Codebase Updates

    Context

    This PR updates the codebase to more modern ES6 which is increasingly becoming a necessity in order to use underlying project dependencies.

    • [x] Update API deps to latest versions where possible
    • [x] Update API Codebase to ES6

    Actions

    The following tests are currently failing:

    • [x] tiles.test.js
    • [x] project.test.js
    • [x] model-osmtag.js
    • [x] instance.test.js
    • [x] checkpoint-upload.js
    • [x] checkpoint.test.js
    • [x] checkpoint-geoms.test.js
    • [x] batch.test.js
    • [x] aoi.test.js
    • [x] aoi-share.test.js
    • [x] aoi-patch.test.js
    enhancement 
    opened by ingalls 0
  • lock azurerm version in providers.tf

    lock azurerm version in providers.tf

    Specify exact versions of azurerm and helm providers to use. Refs #12 .

    It seems like there are two ways of "locking" versions of providers. One if this, to specify it in the providers.tf. The other would be to create a .terraform.lock.hcl file with the locked dependencies.

    The problem with that approach is that I needed to create a separate lockfile for dev, staging, and production, and that seemed like a lot of hassle - to remember to update those all locally, keep in sync, etc.

    This seemed cleaner, but we can use the other approach if that seems better.

    cc @geohacker

    opened by batpad 0
  • Making staging work again

    Making staging work again

    This is all a bit terrible.

    So, changes to the azurerm version caused some properties to resources to change. Since terraform isn't great with updating Azure resources in place when properties changed, it wanted to re-create both the cluster and the IP address. This was causing cascading failures.

    For now, what I've done is add a lifecycle argument to the Cluster and IP address Azure resources to just not do any updates on those resources.

    @geohacker - let's discuss, but I think we should keep it that way. Unfortunately, allowing changes to cluster or IP address properties seems like it just results in too many problems. This is not ideal at all, but I don't have a better way.

    For now, going to merge this and have staging work again.

    Also, using $refs in the values.schema.json didn't quite work on CI, and had to revert that change as well.

    opened by batpad 0
  • Do not allow activation if model has no file

    Do not allow activation if model has no file

    We need to add a constraint on setting active=true on models that have storage=false, my understading is that a model file is required to create and run projects.

    bug 
    opened by geohacker 0
  • Json schema util

    Json schema util

    What I am changing

    Fairly minor - get the $refs in the json schema to actually work - file://./ did the trick.

    Tested that the helm chart seems to publish fine. We should keep an eye out on staging, but this shouldn't affect the deploy - just cleans up the values.schema.json file massively.

    cc @geohacker

    opened by batpad 0
  • Project TS

    Project TS

    Context

    Mosaic data is currently stored at the project level with subsequent AOIs of a given project inheriting the single mosaic type from their parent project. This has worked up until now as we have essentially only used the naip.latest mosaic for all projects.

    Adding timeseries data complicates this model as it means that there is no relationship to be made between similiar AOIs that use mosaics with different timestamps. Although a user would be able to manually select mosaics that might overlap and compare them, a DB relationship wasn't possible.

    This PR changes the way mosaic data is stored in the database and surfaced via the API. Mosaic data has been removed from the project data and is instead stored

    • Project - Just a container for a given Model ID
    • AOI - Contains Bound Data & Name
    • AOI_TimeFrame - Contains most of the data that AOI formerly did (classes, patches, etc) as well as a new mosaic ID & mosaic Timestamp field

    Closes: https://github.com/developmentseed/pearl-backend/issues/36

    cc/ @geohacker

    opened by ingalls 0
  • Add mosaic id and date info to the AOI

    Add mosaic id and date info to the AOI

    Currently we don't store the mosaic and date range an AOI (inference) is generated from. We should do this so downstream consumers can perform better change management.

    @developmentseed/pearl

    opened by geohacker 0
  • Prepare some predefined search ids for sentinel 2 mosaic in Mexico

    Prepare some predefined search ids for sentinel 2 mosaic in Mexico

    We should look at https://planetarycomputer.microsoft.com/explore?c=-104.9008%2C22.8615&z=4.40&v=2 and prepare a few different seasons of mosaics that we can use as a starter. This is related and required for #32.

    We'll support advanced filters but this is important for a start.

    cc @developmentseed/pearl

    opened by geohacker 0
  • Support large AOIs

    Support large AOIs

    Our current AOI limits are:

    • Live inference 100 km²
    • Batch inference 200 km²

    We should increase this based on what geometry sizes Reforestamos wants to run for. I'll look at that provide the numbers.

    cc @ingalls

    opened by geohacker 0
  • Add Sentinel Support

    Add Sentinel Support

    We should support Sentinel imagery from Planetary Computer. Unlike NAIP, with Sentinel we want to allow the frontend decide a time range for creating a mosaic to inference to run on.

    I think the inference endpoint will now have to not only take the imagery id/name but also look for a time range which we need to use to query tiles.

    @ingalls @vincentsarago let's chat more specifics about this. The frontend design has some implications. cc @LanesGood @vgeorge @batpad

    opened by geohacker 1
Owner
Development Seed
Development Seed
[WIP] n8n nodes that cover the Zoho "Books" API

n8n-nodes-starter This repo contains example nodes to help you get started building your own custom integrations for n8n. It includes the node linter

Vrishin Patel 6 Nov 26, 2022
Open Harvest is a technology designed farmers to publicly share the details of their crops/land to allow all farmers to stay informed on the best crops they can grow without risking a surplus crisis.

OpenHarvest OpenHarvest is a web application designed to balance farming production in India. Farmers in India are struggling with marketing their cro

Call for Code® with The Linux Foundation 11 Dec 8, 2022
typesafe nodejs client for transit.land

TransitLand Graphql Client About Since the Transitland released its new and shiny GraphQL API, it is possible to query the API using GraphQL. The clie

ioki 5 Jan 9, 2023
Package publishing for deno.land/x/polkadot

deno/polkadot Experimental This is the first release of the Deno interfaces for the polkadot-js family, as such it still needs a lot of testing and co

@polkadot{.js} 22 Dec 3, 2022
ChainJet is a workflow automation platform specialized in blockchain and cloud infrastructure.

A workflow automation platform specialized on blockchain, with more than 300 integrations. Description ChainJet is a workflow automation platform spec

ChainJet 38 Dec 24, 2022
Open source data infrastructure platform. Designed for developers, built for speed.

Gigahex is a web based data infrastructure platform to deploy and manage Apache Spark™, Apache Kafka and Apache Hadoop clusters. Currently, it support

Gigahex 22 Dec 6, 2022
Open source data infrastructure platform. Designed for developers, built for speed.

Gigahex is a web based data infrastructure platform to deploy and manage Apache Spark™, Apache Kafka and Apache Hadoop clusters. Currently, it support

Gigahex 21 Apr 1, 2022
Extensión para crear el icono y cover de un curso en Platzi para organizar la toma de apuntes en Notion

?? Platzi Cover Extensión para extraer el icono y cover de un curso en Platzi para organizar la toma de apuntes en Notion ?? Instalación Descargar el

Juan Galvis 7 Nov 15, 2021
A cover page transition based on Vitalii Burhonskyi's Dribbble shot.

Cover Page Transition A cover page transition based on Vitalii Burhonskyi's Dribbble shot. Article on Codrops Demo Installation Install dependencies:

Codrops 37 Dec 31, 2022
API4AI is cloud-native computer vision & AI platform for startups, enterprises and individual developers

API4AI is cloud-native computer vision & AI platform for startups, enterprises and individual developers. This repository contains sample mini apps that utilize Brand Recognition API provided by API4AI.

api4ai 11 May 24, 2022
NoExGen is a node.js express application generator with modern folder structure, namespace/project mapping and much more! It contains preconfigured Settings and Routing files, ready to be used in any project.

Installation $ npm install -g noexgen Quick Start You can use Node Package Execution to create your node-express application as shown below: Create th

Souvik Sen 7 Oct 8, 2022
awsrun 189 Jan 3, 2023
Collect and generate mapping from file-based routers

roullector: route collector Collect and generate route data from a file-based router such as svelte-kit's What this does: show / hide <!-- before -->

Quang Phan 4 Apr 9, 2022
Mind Mapping to excel, or excel to .xmind file

Mind Mapping To Excel Project setup Prepare project npm install 1、Fetch data and generate excel Open the Mind Mapping like this Process On Mind Mappi

xuzelin995 3 May 5, 2022
Userland module that implements the module path mapping that Node.js does with "exports" in package.json

exports-map Userland module that implements the module path mapping that Node.js does with "exports" in package.json npm install exports-map Usage co

Mathias Buus 9 May 31, 2022
Atlan is a CLI that helps you to manage local dockerized infrastructure without any needed knowledge on Docker and Docker Compose.

Atlan CLI Atlan is a CLI that helps you to manage local dockerized infrastructure without any needed knowledge on Docker and Docker Compose. What is A

Myastro 10 Aug 17, 2022
Easy-to-use CDK constructs for monitoring your AWS infrastructure

CDK Monitoring Constructs Easy-to-use CDK constructs for monitoring your AWS infrastructure. Easily add commonly-used alarms using predefined properti

CDK Labs at AWS 214 Jan 6, 2023
A tool for managing production-grade cloud clusters, infrastructure as code

Cloudy Description Cloudy is an "infrastructure as code" tool for managing production-grade cloud clusters. It's based on Pulumi that mostly using Ter

Cloudy 24 Jan 1, 2023
AWSGoat : A Damn Vulnerable AWS Infrastructure

AWS GOAT Compromising an organization's cloud infrastructure is like sitting on a gold mine for attackers. And sometimes, a simple misconfiguration or

INE Lab Infrastructure 993 Dec 28, 2022