A flexible gateway for running ML inference jobs through cloud providers or your own GPU. Powered by Replicate and Cloudflare Workers.

Overview

Cogflare

(Working title)

Cogflare is a Cloudflare Workers application that aims to simplify running distributed ML inference jobs through a central API. The jobs can (currently) be run using Replicate or anywhere that Docker and a GPU is available, such as RunPod or your own hardware.

How does it work?

Cogflare provides an HTTP API that is similar in surface area to Replicate's API, but enables additional flexibility for running predictions. It provides a websocket server that workers connect to, and a lightweight queue with state managed by Durable Objects for running jobs on those workers. R2 is leveraged to provide storage for results with high speed and no bandwidth costs, and the workers KV store is used as a record of data. The entire setup should cost $5 + $0.015/GB of storage + GPU costs, at the scale tested.

Why does this exist?

I've been through a few iterations of the backend for NightmareBot. I found Replicate a great solution overall but needed the flexibility to run wherever I wanted. I've found a number of people trying to solve the same problems I have so I created this project in the hopes of keeping others from having to reinvent the wheel so many times.

What's it useful for?

You tell me! Bots are the big use case so far, but anywhere you need API access to ML inference with flexibility could be a fit.

How do I use it?

Contact [email protected] if you're interested in using my hosted version - I'm not currently charging for it but I can only let people use it who are willing to act as testers. Currently there's a hard dependency on the Replicate API using your own key, so you'll first need to sign up for API access there and get a key.

No, not how do I use yours, how do I use it?

This is a preliminary guide and needs a lot of work, operation is subject to drastic changes and it's not really recommended to do this yourself yet

If you'd like to host it yourself, it should be as simple as updating the wrangler.toml file with your own account and resources (which you'll need to create) and deploying - no special sauce. The setup of Cloudflare workers is beyond the scope of this document for now, but I hope to add some basic instructions soon.

You'll need an entry in the TOKENS_KV, with the key acting as an authorization token and the contents being a JSON structure like this:

{"allow": true, "replicate": "YOUR_REPLICATE_TOKEN", "worker": "some-random-string" }

You can now make prediction requests by POSTing to the /predictions endpoint and retrieve their status with GET /predictions/id.

Jobs will run on Replicate unless there are workers available. There is simple overflow logic place right now as well that sends jobs to Replicate if queue depth is greater than available workers, but this is temporary and subject to change.

Currently workers have to run Docker images built using a fork of Replicate's Cog, however the public image r8.im/nightmareai/disco-diffusion has been built using this so I'll use it as an example. To start a worker for Disco Diffusion, a typical command would look like:

docker run --rm --gpus=all r8.im/nightmareai/disco-diffusion python -m cog.server.websockets wss://[WORKER-URL]/v1/models/nightmareai/disco-diffusion/websockets/[WORKER-TOKEN] https://[WORKER-URL]/v1/models/nightmareai/disco-diffusion/files nightmareai/disco-diffusion

This attaches the worker to your queue for this model, and it will continue to reconnect and run jobs until killed.

This all sounds really complicated, I just want to run stuff and I don't want to talk to you!

Stay tuned! Easy sign up and setup are on the list!

You might also like...

A Raycast Extension to view your replicate.com image collection

A Raycast Extension to view your replicate.com image collection

Replicate API Raycast Extension View your prediction history and search for predictions by prompt. Create an account and get an API token here (requir

Sep 18, 2022

Google-Drive-Directory-Index | Combining the power of Cloudflare Workers and Google Drive API will allow you to index your Google Drive files on the browser.

🍿 Google-Drive-Directory-Index Combining the power of Cloudflare Workers and Google Drive will allow you to index your Google Drive files on the brow

Jan 2, 2023

This is a boilerplate for creating your own languages for various use cases. You can even create your own programming language from scratch!

Bootstrap compiler This is a bootstrap compiler that can be used to compile to compiler written in the target language. You can write a compiler in th

Nov 14, 2022

Attempt to replicate the visuals of late 80s / early 90s flight simulators

Attempt to replicate the visuals of late 80s / early 90s flight simulators

Retro Fligh Simulator Quick and dirty attempt to replicate the visuals of late 80s / early 90s flight simulators, using as a reference MicroProse's F-

Dec 15, 2022

A web GUI for inpainting with Stable Diffusion using the Replicate API.

🎨 Inpainter A web GUI for inpainting with Stable Diffusion using the Replicate API. Try it out at inpainter.vercel.app cherries-oranges-bananas.mp4 H

Dec 27, 2022

Abusing Cloudflare Workers to establish persistence and exfiltrate sensitive data at the edge.

Abusing Cloudflare Workers This repository contains companion code for the blog post MITM at the Edge: Abusing Cloudflare Workers. malicious-worker/ c

Sep 16, 2022

Lightweight universal Cloudflare API client library for Node.js, Browser, and CF Workers

Cloudflare API Client Lightweight universal HTTP client for Cloudflare API based on Fetch API that works in Node.js, browser, and CF Workers environme

Nov 13, 2022

A Cloudflare Workers service that fetches and renders Notion pages as HTML, Markdown, or JSON.

notion-fetch A Cloudflare Workers service that fetches and renders Notion pages as HTML, Markdown, or JSON. Powered by Durable Objects and R2. Usage P

Jan 6, 2023

A zero-dependency, strongly-typed web framework for Bun, Node and Cloudflare workers

nbit A simple, declarative, type-safe way to build web services and REST APIs for Bun, Node and Cloudflare Workers. Examples See some quick examples b

Sep 16, 2022
Owner
NightmareBot
NightmareBot
A simple Prometheus (aggregated) push gateway allowing stateless/serverless workloads, ephemeral and batch jobs to easily expose their metrics.

Serverless Prometheus (aggregated) Push Gateway A simple Prometheus (aggregated) push gateway allowing stateless/serverless workloads, ephemeral and b

Adam Janiš 17 Dec 4, 2022
Functional-style Cloudflare Durable Objects with direct API calls from Cloudflare Workers and TypeScript support.

durable-apis Simplifies usage of Cloudflare Durable Objects, allowing a functional programming style or class style, lightweight object definitions, a

Dabble 12 Jan 2, 2023
Node starter kit for semantic-search. Uses Mighty Inference Server with Qdrant vector search.

Mighty Starter This project provides a complete and working semantic search application, using Mighty Inference Server, Qdrant Vector Search, and an e

MAX.IO LLC 8 Oct 18, 2022
Use Cloudflare Gateway DNS/VPN to block ads, malware and tracking domains - free alternative to NextDNS, Pi-hole and Adguard

Cloudflare Gateway Pi-hole Scripts (CGPS) Cloudflare Gateway allows you to create custom rules to filter HTTP, DNS, and network traffic based on your

null 70 Aug 13, 2023
CLI Progress Bar implemented in NodeJS to track Time, ETA and Steps for any long running jobs in any loops in JS, NodeJS code

NodeJS-ProgressBar CLI Progress Bar for NodeJS and JavaScript to track Time, ETA and Steps for any long running jobs in any loops in JS, NodeJS code D

Atanu Sarkar 5 Nov 14, 2022
Grupprojekt för kurserna 'Javascript med Ramverk' och 'Agil Utveckling'

JavaScript-med-Ramverk-Laboration-3 Grupprojektet för kurserna Javascript med Ramverk och Agil Utveckling. Utvecklingsguide För information om hur utv

Svante Jonsson IT-Högskolan 3 May 18, 2022
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
awsrun 189 Jan 3, 2023