The AI Data Platform

Annotate, Manage and Deploy Training Data

The end-to-end AI data management platform that helps ML teams annotate, manage and deploy training data at scale.








LayerX is on a mission to accelerate AI application development by empowering ML teams with modern tools.

Features

• Image and video annotations with multiple annotation types. Just drag and drop any image or video file and start annotating in minutes.
• Class and Attribute level annotations. For example, class can be Vehicle, attributes can be Car, Van, Truck.
• Annotation task management. Byte size tasks make it easy to annotate large volumes of data at scale with high accuracy.
• Dataset management. Create, manage and deploy training datasets from annotated data. Version control your datasets to track changes over time.
• Data Augmentation. Increase your dataset size by adding slightly modified copies via built-in augmentation functions like crop, rotation, flip, blur and more.
• Export your annotated dataset to the training machine in a single click using the dataset sync tool.

Install the Open Source version using the instructions below. If you need a fully hosted version of LayerX you can start a free trial of LayerX Cloud

High Level System Architecture

Backend - Annotation Manager (NodeJS) :

• REST API endpoint for the Annotation tool frontend
• Communicates with a Python backend (Flask App) to offload background tasks such as annotations and dataset creation
  
  

Backend - data processor (Python Flask) :

• Creates annotation tasks from uploaded videos or images
• Creates datasets and augments images
  
  

Frontend- (Angular Web) :

• User interface to manage annotation projects and data sets
• Rich web interface for annotating objects
  
  

Deploy LayerX-ai with docker

Pre-requirements

Tested OS : Ubuntu 20.04.2 LTS

Install git

apt install git-all

Install Docker - https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

Install Docker Compose - https://docs.docker.com/compose/install/#install-compose-on-linux-systems

Configure to manage Docker as a non-root user

Create an AWS S3 bucket - https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html

Useful

How to find AWS S3 Access key / Secret key

Clone Repository

layerx-community

git clone https://github.com/LayerX-AI/layerx-community.git

Environment Specific Configurations

.env file (at the root of local repository)

URL which system is hosted

BASE_URL = <server  url>
Note: if hosted in local, then put this as "http://localhost"

AWS S3 bucket configurations

AWS_ACCESS_KEY = <AWS_ACCESS_KEY>
AWS_SECRET_KEY = <AWS_SECRET_KEY>
AWS_REGION = <AWS_REGION  eg:us-east-1>
AWS_BUCKET_NAME = <S3 Bucket name>

Support email sending config with sendgrid (optional)

SENDGRID_API_KEY = <SENDGRID_API_KEY>
SUPPORT_EMAIL = <YOUR_EMAIL_ADDRESS_FOR_SUPPORT_EMAIL_SENDING>

google drive configuration for content upload (optional)

GOOGLE_API_KEY = <GOOGLE_API_KEY>
GOOGLE_CLIENT_ID = <GOOGLE_CLIENT_ID.apps.googleusercontent.com>
GOOGLE_CLIENT_SECRET = <GOOGLE_CLIENT_SECRET>
GOOGLE_REFRESH_TOKEN = <GOOGLE PLAYGROUND REFRESH TOKEN>

Run Configure script to update configurations of each components

./configure.sh

Run the system

docker-compose up

LayerX is now ready to use :)

Additional Notes

Directory structure looks like

.
+-- layerX-enterprise/
	---.env
	---default.env
	---docker-compose.yml
	+--contents/
	+--uploads/
	+--mongoData/
	+--DB_initial_data/
	---nginxData/
		+--nginx/
	+--layerx-angular-frontend/
	+--layerx-python-backend/
	+--layerx-nodejs-backend/
	+--layerx-sync-tool

Services and ports

• mongodb - 1521
• python_app - 8081
• node_backend - 8080
• frontend - 8085
• nginx - 80

Other Commands

Start the system

docker-compose up

Stop the system

docker-compose stop

Restart the system

docker-compose restart

Start a single service

docker-compose up <service name>
eg: docker-compose up node_backend

Build a container service (node_backend, python_app and frontend)

docker-compose build <service name>
eg: docker-compose build frontend

Build a service from image (mongodb and nginx)

docker-compose up --force-recreate <service name>
eg: docker-compose up --force-recreate nginx