An AWS Cloud Native application using CDK that defines a Serverless Event Driven application for interacting with Twitter and utilising Machine Learning / AI as a Service.

Overview

AWS Serverless Event Driven Twitter Bot

An AWS Cloud Native application using CDK (Written in TypeScript) that defines a Serverless Event Driven application for interacting with Twitter and utilising Machine Learning / AI as a Service.

Overview

System Diagram

First point to realise for this application, is that it could be simplified down to a few lambdas without Events - and for this single use-case it would work fine. The premise here is to demonstrate many different concepts and a Serverless Event Driven architecture that works for simple applications as well as for the Enterprise.

This whole application is defined here within CDK, apart from the Ad-Hoc Athena queries, some examples can be seen below.

Serverless

The application only costs money when it is processing a message from Twitter, otherwise it is sitting in a standby state - with the only cost being the minor cost of S3 and CloudWatch log persistence. More can be seen in the Cost section below.

Event Driven

It is Event Driven because it is reactive based on the webhooks coming from Twitter, as soon as a message is received then AWS will take care of provisioning the resources, as defined in this application, to process it and then shut down again. This works whether it's a single message or a burst of 100's of messages.

Ingress

The Ingress Microservice at the top of the diagram takes care of accepting the messages from Twitter, this includes the authentication (Using API Keys stored in Secrets Manager). Each message is sent to a Lambda using a Proxy integration - this takes care of Authentication and then sending the payload into EventBridge. Note, the messages could be sent directly to EventBridge from API Gateway, but a Lambda Authoriser would be needed anyway - so using a Lambda Proxy Integration keeps this simple.

The raw message is sent to EventBridge with the type of message set as the detail-type - Twitter lists all of the types that could be sent to the endpoint in it's documentation.

A simple Anti-Corruption Layer also publishes a simplified Domain event of MESSAGE_RECEIVED.

Plumbing

The Plumbing Microservice consists of the Event Hub within EventBridge utilised for effectively linking all the Microservices. It also is configured with an Event Archive to allow replaying of events if required.

A catch-all rule exists to push every event into CloudWatch logs for two purposes:

  • Debugging
  • Metrics

A Metric Filter is deployed to the CloudWatch group for each type of message which generates a customer Metric that can then be used to visualise the events flowing through the system.

Analysis

The Analysis Microservice deploys a rule into the Plumbing Event Hub to catch all MESSAGE_RECEIVED events - this starts an execution on a StepFunction to allow orchestration of multiple services:

  • The downloading of any media into a local bucket and then parallel processing of these through different Rekognition endpoints to look for Labels, Text, Faces, etc.
  • The parallel processing of text through different Comprehend endpoints to gather insights into the text.
  • Pushing the result of all the analysis insights back to the Event Hub with a MESSAGE_ANALYSED event.

Alerting

The Alerting Microservice deploys a rule into the Plumbing Event Hub to catch all MESSAGE_ANALYSED events which have found text with a Sentiment of NEGATIVE. This targets an SNS Topic that can be sub-scribed to from Email addresses, Phone Numbers, etc to get alerts when Negative messages are received. It could integrate into Slack and other notification systems.

Responding

The Responding Microservice deploys two rules into the Plumbing Event Hub to catch all MESSAGE_ANALYSED events. One catches those with Images in them, and one catches those with no images, only text.

Images

The Image rule executes a Lambda which pulls the image from S3 (downloaded earlier) and uses the Rekognition insights to add Celebrity names onto the image next to the faces - and then generates a command Event onto the Plumbing Event Bus with the list of celebrities (or message saying none found) called SEND_TWEET.

This can also blur faces of none-celebrities, highlight certain objects, etc.

Text

The Text rule executes a Lambda which calls Amazon Lex - an AI Conversational Bot which was built/defined in CDK. Lex utilises a separate Lambda to fulfilment any highlighted topics (Jokes/Facts) and returns the response. A SEND_TWEET command is generated and pushed to the Plumbing Event Bus.

Egress

The Egress Microservice deploys a rule into the Plumbing Event Hub to catch all SEND_TWEET commands and executes a Lambda which pulls API credentials from Secrets Manager, pulls any images from S3 and then calls the Twitter API to create a reply tweet. If it wasn't for the Images, then this could be done with a EventBridge API Destination in theory.

Analytics

The Analytics Microservice deploys a rule into the Plumbing Event Hub to catch all messages. This targets Kinesis Data Firehose, which is configured to use a Lambda for transformation (simply adds a newline after each message so they can be parsed later) and then store in S3 as a Data Lake - it has a 1 minute buffer configured so is not real-time.

A Glue Table has also been defined for MESSAGE_ANALYSED that can be used by Athena - Glue could also be configured to crawl the S3 bucket and build the tables automatically, but for the purposes of the reporting required here a single static table (based on an Internal event that is known) makes most sense.

Athena can be used against the bucket, and using the Glue table to run SQL queries about the messages received and all the analysis data (Such as find all messages that are Positive, or all messages that contained an image with a car in it).

Utilised AWS Services

  • AWS Glue
  • AWS Identity and Access Management (IAM)
  • AWS Lambda
  • AWS Secrets Manager
  • AWS Step Functions
  • Amazon API Gateway
  • Amazon Athena
  • Amazon CloudWatch
  • Amazon Comprehend
  • Amazon EventBridge
  • Amazon Kinesis Data Firehose
  • Amazon Lex
  • Amazon Rekognition
  • Amazon S3
  • Amazon Simple Notification Service (SNS)

Building and Deploying

CDK Deploy

If not already setup for CDK then you will need:

  • AWS CLI installed and your workstation configured to the correct account: aws configure
  • Node & NPM installed
  • CDK installed globally: npm install -g aws-cdk
    • This can be verified by running cdk --version

Within the root of this application you should be able to then run a npm install to restore the dependencies.

Once installed, then you can run cdk deploy --all --context twitterAccountId=999999 to build and deploy all stacks to your default AWS account/region. Fill in your own account ID here. For other CDK commands then check documention.

The API Gateway URL should be output to the console as part of the deployment, but may be hard to find in the output - it will look something like: IngressStack.APIGateway = https://99dd9d9dd.execute-api.eu-west-1.amazonaws.com/prod/

If you cannot find it, then navigate to API Gateway in your console and you should have an API called ingress-api - if you navigate to this and then Stages and prod you can see the url there.

Twitter Developer Account

The application is reactive to webhooks from Twitter utilising the Account Activity API. For this a Developer Account is needed.

  1. Sign up for a Twitter Developer Account
  2. Apply for Elevated Access
  3. Create an Application and grab all the API Keys/Secrets, Auth Tokens, etc.
  4. Follow the Twitter Secrets below to add these details to your AWS account.
  5. Create a Dev Environment for that application to use the Account Activity API.
  6. Register a Webhook with the https://api.twitter.com/1.1/account_activity/all/{{environment}}/webhooks.json?url={{your_api_gateway_url}}/prod/twitter API. More details in theTwitter API Doc.
  7. Register a subscription for the account. More details in the Twitter API Doc.

Once a webhook is registered then an API call will be made to the API Gateway to verify, this can be seen in the logs for the IngressStack-TwitterActivitylambda lambda for debugging.

Twitter Secrets

Create a Secret in Secret Manager manually in the correct AWS account and region with the name TwitterSecret and value of the below. (In the UI this is added as a Key/value pair or plaintext of the raw JSON like below):

{
  ApiKey: 'TODO',
  ApiSecret: 'TODO',
  AccessToken: 'TODO,
  AccessTokenSecret: 'TODO'
}

Athena

You can use the below ad-hoc queries in Athena by selecting the messages-data-lake Glue table created as part of this CDK app. Some sample searches below:

Select all Celebrities

SELECT time, detail.author, celebrityfaces.name
FROM "analysed-messages-table"
CROSS JOIN UNNEST(detail.analysis.images) as t(images)
CROSS JOIN UNNEST(images.analysis.celebrityfaces) as t(celebrityfaces)
WHERE "detail-type" = 'MESSAGE_ANALYSED'
ORDER BY time DESC

Select all Image Labels

SELECT time, detail.author, detail.text, labels.name
FROM "analysed-messages-table"
CROSS JOIN UNNEST(detail.analysis.images) as t(images)
CROSS JOIN UNNEST(images.analysis.labels) as t(labels)
WHERE "detail-type" = 'MESSAGE_ANALYSED'
ORDER BY time DESC

Select all Positive Text

SELECT time, detail.author, detail.analysis.textsentiment, detail.text
FROM "analysed-messages-table"
CROSS JOIN UNNEST(detail.analysis.images) as t(images)
WHERE "detail-type" = 'MESSAGE_ANALYSED' AND detail.analysis.textsentiment='POSITIVE'
ORDER BY time DESC

Cost

The cost of processing can be broken down into a few areas:

  • AI (Rekognition, Lex, Comprehend)
  • Code/Infrastructural (StepFunction, Lambda, EventBridge, API Gateway, SNS)
  • Storage (S3, Cloudwatch, Secrets Manager)

AI

This is the most expensive part, especially since the Step Function in this application is currently running the text and images through multiple APIs and not using all the results - obviously in a production ready system that could be handling millions of requests then this should be optimised. As is though, to deal with 1 million requests (50/50 on text and image), it would cost about £1500. This works out as £0.0015 per message - or about 15p for 100 messages.

Code/Infrastructural

The Code/Infrastructural is the second most expensive part, but mainly because of using a Standard Step Function - which as mentioned above is not optimised as it calls the AI services for everything - adding extra steps, and in production would be better suited to Express step functions for each specific use-case potentially. Even so, this works out at around £130 to handle 1 million messages, this works out as £0.00013 per message, less than 1p for 100 messages.

Interestingly Lambda would be less than a few pounds to handle 1 million messages with the use-case in this application. This works out as a ridiculously small number per message, or even per 100 messages.

EventBridge would be less than £1 top handle 1 million messages. Firehose would be less than £0.30.

Storage

Storage is cheap on AWS, so to store a lot of history in the Data Lake and in CloudWatch then this would probably work out as up to around £5 (assuming 1 million messages). With more realistic numbers this would be pence.

Overall

So overall it would be about £1700 to handle 1 million messages with the current design. Or £0.0017 per message - £0.17 for 100 messages.

You might also like...

This application provides the CDK project and a frontend that allows you to build a serverless chat application based on API Gateway's WebSocket-based API feature.

This application provides the CDK project and a frontend that allows you to build a serverless chat application based on API Gateway's WebSocket-based API feature.

Serverless chat application using ApiGateway Websockets This project lets you provision a ready-to-use fully serverless real-time chat application usi

Sep 14, 2022

Sample code for resizing Images with [email protected] using the Custom Origin. You can deploy using AWS CDK.

Sample code for resizing Images with Lambda@Edge using the Custom Origin. You can deploy using AWS CDK.

Resizing Images with [email protected] using the Custom Origin You can resize the images and convert the image format by query parameters. This [email protected]

Aug 23, 2022

A monorepo that uses the AWS Cloud Development Kit to deploy and configure nanomdm on AWS lambda.

NanoMDM on AWS This repo builds and configures a nanomdm server to run on AWS lambda. It uses the Cloud Development Kit and tries to follow best pract

May 26, 2022

A sample CICD Deployment Pipeline for your Alexa Skills, using AWS CDK, CodeBuild and CodePipeline

A sample CICD Deployment Pipeline for your Alexa Skills, using AWS CDK, CodeBuild and CodePipeline

Alexa Skils - CI/CD CDK Pipeline This repository will help you setting up a CI/CD pipeline for your Alexa Skills. This pipeline is powered by AWS Clou

Feb 4, 2022

A sample code that implements a simple Web app using AWS CDK v2

A sample code that implements a simple Web app using AWS CDK v2

A sample code that implements a simple Web app using AWS CDK v2. This code will be introduced in a live coding session at AWS Summit Online Japan 2022 Developer Zone in 2022/5/25.

Aug 17, 2022

AWS CDK compiled for web (and Node!)

cdk-web 🚀 DEMO 💪 AWS CDK compiled for web (and Node!) cdk-web and aws-cdk-web are functionally identical packages on npm. read about the differences

Jul 19, 2022

This project provides a CDK construct creating AWS organizations.

AWS Organizations This project provides a CDK construct creating AWS organizations. Currently, there is no @aws-cdk/aws-organizations available. See t

Sep 19, 2022

Easy-to-use CDK constructs for monitoring your AWS infrastructure

CDK Monitoring Constructs Easy-to-use CDK constructs for monitoring your AWS infrastructure. Easily add commonly-used alarms using predefined properti

Sep 21, 2022

Functionless-based mini-framework for DynamoDB migrations in AWS CDK.

dynamodb-migrations This repo is heavily in progress! Readme describes desired contract and functionality. Please do not try using it yet!. I'm not ev

Aug 30, 2022
Owner
Principal Software Architect working with AWS and web services at scale
null
Learn Web 2.0 and Web 3.0 Development using Next.js, Typescript, AWS CDK, AWS Serverless, Ethereum and AWS Aurora Serverless

Learn Web 2.0 Cloud and Web 3.0 Development in Baby Steps In this course repo we will learn Web 2.0 cloud development using the latest state of the ar

Panacloud Multi-Cloud Internet-Scale Modern Global Apps 58 Sep 22, 2022
Under the Sea is an official AWS workshop delivered by AWS SAs and AWS Partners to help customers and partners to learn about AIOps with serverless architectures on AWS.

Under the Sea - AIOps with Serverless Workshop Under the Sea is an exciting MMORPG developed by the famous entrepreneur behind Wild Rydes, the most po

AWS Samples 2 Sep 7, 2022
AWS Lambda & Serverless - Developer Guide with Hands-on Labs. Develop thousands line of aws lambda functions interact to aws serverless services with real-world hands-on labs

AWS Lambda & Serverless - Developer Guide with Hands-on Labs UDEMY COURSE WITH DISCOUNTED - Step by Step Development of this Repository -> https://www

awsrun 15 Sep 15, 2022
'event-driven' library aims to simplify building backends in an event driven style

'event-driven' library aims to simplify building backends in an event driven style(event driven architecture). For message broker, light weight Redis Stream is used and for event store, the well known NoSQL database, MongoDB, is used.

Sihoon Kim 9 Aug 4, 2022
MerLoc is a live AWS Lambda function development and debugging tool. MerLoc allows you to run AWS Lambda functions on your local while they are still part of a flow in the AWS cloud remote.

MerLoc MerLoc is a live AWS Lambda function development and debugging tool. MerLoc allows you to run AWS Lambda functions on your local while they are

Thundra 100 Sep 20, 2022
Example Serverless DynamoDB integration tests using Jest, TypeScript and the AWS CDK

serverless dynamodb integration tests ?? Example Serverless DynamoDB integration tests using Jest, TypeScript and the AWS CDK Introduction How to inte

Lee Gilmore 6 Jun 11, 2022
Sample AWS microservices app with service discovery defined using the CDK. Uses Docker + Fargate & ELB.

AWS Microservices Demo with CDK and Fargate About Simple AWS microservice-based app. Consists of two Spring Boot based services: Name Service GET /nam

Nick Klaene 5 Jan 13, 2022
io-ts Typed Event Bus for the runtime of your Node.js application. A core for any event-driven architecture based app.

Typed Event Bus Based on io-ts types, this bus provides a handy interface to publish and consume events in the current runtime of the Node.js process.

Konstantin Knyazev 3 May 23, 2022
A serverless AWS expense tracker API. AWS Lambda functions, API gateway, and Dynamodb are among the ingredients.

AWS-Serverless-API A serverless AWS expense tracker API. AWS Lambda functions API gateway Dynamodb Endpoints Create a new expense: Method: POST Body f

Ondiek Elijah Ochieng 1 Jul 16, 2022