An example repository on how to start building graph applications on streaming data. Just clone and start building 💻 💪

Last update: Dec 20, 2022

Related tags

Overview

Example Streaming App 🚀 🚀

This repository serves as a point of reference when developing a streaming application with Memgraph and a message broker such as Kafka.

KafkaProducer represents the source of your data. That can be transactions, queries, metadata or something different entirely. In this minimal example we propose using a special string format that is easy to parse. The data is sent from the KafkaProducer to Kafka under a topic aptly named topic. The Backend implements a KafkaConsumer. It takes data from Kafka, parses it and sends it to Memgraph for graph analysis, feature extraction or storage.

Installation

Install Kafka and Memgraph using the instructions in the homonymous directories. Then choose a programming language from the list of supported languages and follow the instructions given there.

List of supported programming languages

How does it work exactly

KafkaProducer

The KafkaProducer in ./kafka/producer creates nodes with a label Person that are connected with edges of type CONNECTED_WITH. In this repository we provide a static producer that reads entries from a file and a stream producer that produces entries every X seconds.

Backend

The backend takes a message at a time from kafka, parses it with a csv parser as a line, converts it into a openCypher query and sends it to Memgraph. After storing a node in Memgraph the backend asks Memgraph how many adjacent nodes does it have and prints it to the terminal.

Memgraph

You can think of Memgraph as two separate components: a storage engine and an algorithm execution engine. First we create a trigger: an algorithm that will be run every time a node is inserted. This algorithm calculates and updates the number of neighbors of each affected node after every query is executed.

Comments

Updating neighbors doesn't work

Bug Description When running kafka/producer/stream_producer.py, the neighbors property on each node always stays 0. Sometimes the node doesn't even contain the property.

To Reproduce Just run property initialized Memgraph + any backend + stream_producer.py.

Expected behavior As new nodes/edges come in, the neighbors should be incrementally updated.

Desktop: OS: Ubuntu 20.04

opened by gitbuda 2
Using the appropriate Memgraph Docker image

If memgraph/memgraph:latest already exists on the machine, memgraph/run.sh memgraph won't actually use the latest Memgraph image.

Probably the best would be to hardcode Memgraph version where everything is working as expected, e.g. v1.6.0.

opened by gitbuda 2
Bump tmpl from 1.0.4 to 1.0.5 in /backend/node
Bumps tmpl from 1.0.4 to 1.0.5.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Port to Memgraph 2.0 features
[x] Create kafka.transform module

[x] Fix or document problem with neighbors

[x] Polish Python backend

[x] Remove redundant code from all backends

[x] Update README page
opened by gitbuda 0

fix memgraph version

closes #21

I added the --pull always flag to the docker run command. It seems weird that docker verifies the hash multiple times before running:

 memgraph fix-memgraph-version ✗ bash run.sh memgraph
19a3f4efbe827fb1bdb5d67241f424d5dec1820627b265b25e5eeda13e65c213
Starting memgraph...
latest: Pulling from memgraph/memgraph
Digest: sha256:9a45e6ad18f28edcc1e8317eb225041a3c017b7841b43d994d90155c97efd905
Status: Image is up to date for memgraph/memgraph:latest
latest: Pulling from memgraph/memgraph
Digest: sha256:9a45e6ad18f28edcc1e8317eb225041a3c017b7841b43d994d90155c97efd905
Status: Image is up to date for memgraph/memgraph:latest
latest: Pulling from memgraph/memgraph
Digest: sha256:9a45e6ad18f28edcc1e8317eb225041a3c017b7841b43d994d90155c97efd905
Status: Image is up to date for memgraph/memgraph:latest
latest: Pulling from memgraph/memgraph
Digest: sha256:9a45e6ad18f28edcc1e8317eb225041a3c017b7841b43d994d90155c97efd905
Status: Image is up to date for memgraph/memgraph:latest

opened by MasterMedo 0

fix errors
[x] fix stream producer

[x] print number of neighbours in all backends

[x] add a script that runs memgraph queries (why are there so many queries?)

~~[ ] add documentation that explains all memgraph queries~~

[x] add create trigger on nodes that sets neighbors to 0

[x] fix update trigger not to do bfs (only immediate neighbours are needed)

[x] update drawing in readme (bigger arrows)
opened by MasterMedo 0
init java backend
~~[ ] add code docs~~

[x] decouple parts of code into functions/classes

[x] simplify pom.xml

~~[ ] remove test class or add tests~~

~~[ ] store kafka message split parts to a Map instead of using String.format()~~

[x] improve print messages

[x] add kafka consumer to try arguments along with the session if possible

~~[ ] extract magic strings and numbers to global variables~~

~~[ ] keep accepting messages even if error occurs after parsing a faulty message~~

~~[ ] use csv reader instead of splitting~~

~~[ ] print number of neighbours~~

~~[ ] use gradle~~
opened by MasterMedo 0
Add Node.js example
[x] Init Express and Jest

[x] Init linter

[x] Fetch data from Kafka

[x] Store data to Memgraph

[x] Calculate something by using BFS

[x] Log to console and/or return data to the user

[x] Report bugs in Memgraph

Trigger recovery + containing query module call.

Trigger --> comment syntax.

Property map feature, e.g., (n:Node $map) (Memgraph does not support that yet).

[x] Improve count correctness (transitive is not working yet)

[x] Improve README installation / setup details
opened by gitbuda 0
Add deployment example with Ansible

Combine Memgraph, Kafka, Docker, Docker Compose, and Ansible to showcase simple deployment for each backend. Similar, just more polished, to https://github.com/memgraph/orbicon.

opened by gitbuda 0

Owner

Memgraph

Real-time operational graph analytics.

GitHub https://memgraph.com

just a graphql example created by typescript + fastify + mikro-orm(postgresql) + mercurius(graphql adaptor) + type-graphql

fastify-mikro-orm-mercurius-graphql-example A MikroORM boilerplate for GraphQL made with Fastify, Mercurius, Typescript using TypeGraphQL ?? Packages

10 Aug 28, 2022

An open source cybersecurity protocol for syncing decentralized graph data.

GUN is an ecosystem of tools that let you build community run and encrypted applications. Currently, Internet Archive and HackerNoon run GUN in produc

16.7k Dec 27, 2022

NestJS + MikroORM example repository for testing within transactional contexts. Achieve a much faster test suite.

Description Nest NestJS + MikroORM example repository for testing within transactional contexts. Running tests in transactions will speedup your test

5 Dec 20, 2022

Streaming and playing on the Nintendo Switch remotely!

Switch-Stream This is a semi-convoluted application as a proof-of-concept that someone could play their Switch from a distance. A server is connected

8 May 2, 2022

HLS, DASH, and future HTTP streaming protocols library for video.js

videojs-http-streaming (VHS) Play HLS, DASH, and future HTTP streaming protocols with video.js, even where they're not natively supported. Included in

2.2k Jan 5, 2023

Like JSON-RPC, but supports streaming.

Earthstar Streaming RPC Similar to JSON-RPC, but also supports streaming (soon). Written to be used in Earthstar (github, docs). Table of Contents Usa

5 Feb 10, 2022

A progressive Node.js framework for building efficient and scalable server-side applications

A light template to easily setup your backend with a simple jwt authentication with Nest.js and TypeScripit. Powered with Prisma and Hot-reload features

3 Feb 25, 2022

A progressive Node.js framework for building efficient and scalable server-side applications

A progressive Node.js framework for building efficient and scalable server-side applications. Description Nest framework TypeScript starter repository

3 Oct 31, 2022

Composable data framework for ambitious web applications.

Orbit Orbit is a composable data framework for managing the complex needs of today's web applications. Although Orbit is primarily used as a flexible

2.3k Dec 18, 2022

A Gmail Clone which built with ReactJS and Redux. You can sign in with your Google Account, compose a new e-mail and send realtime emails to the project.

Gmail Clone with ReactJS A Gmail Clone that you can sign in with your Google Account, compose a new e-mail and send realtime emails to the project. Cl

49 Nov 14, 2022

Python ELT Studio, an application for building ELT (and ETL) data flows.

PELT Studio The Python Extract, Load, Transform Studio is an application for performing ELT (and ETL) tasks. Under the hood the application consists o

55 Nov 18, 2022

A Full Stack Amazon Clone which created using ReactJS with full E-Commerce Functionality!!

Amazon Clone with ReactJS A small web app that tries to imitate the desktop web version of amazon site, you can add items to the basket, delete them,

50 Oct 3, 2022

Uniswapv2-pool-funding - A "Nugget Standard for Funding" compliant smart contract to provide liquidity to UniswapV2 (clone) pools. (Quickswap in this case)

uniswap(clone)-v2-pool-funding A smart contract that makes it easy to: supply liquidity to a uniswap(clone)-v2 pool using ERC20 or the network native

2 May 14, 2022

Dev.to clone

Dev.to-clone A Dev.to clone created with MERN stack and more (Essentially FB without stories feature) Tech used with vite ?? : Frontend React Redux to

64 Dec 13, 2022

AlaSQL.js - JavaScript SQL database for browser and Node.js. Handles both traditional relational tables and nested JSON data (NoSQL). Export, store, and import data from localStorage, IndexedDB, or Excel.

Please use version 1.x as prior versions has a security flaw if you use user generated data to concat your SQL strings instead of providing them as a

6.1k Jan 9, 2023

Azure Data Studio is a data management tool that enables you to work with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

Azure Data Studio is a data management tool that enables working with SQL Server, Azure SQL DB and SQL DW from Windows, macOS and Linux.

7k Dec 31, 2022

ReCord - really i dont even know, start obs recording by joining voice channel

ReCord starts recording Voice Calls in Discord How To Use Requirements: NodeJS A text editor OBS webhook plugin Step 1: create a config.json file Ste

1 May 1, 2022

Very easy graphQL example made by Bobby Chao

Very easy graphQL example made by Bobby Chao. The folder has been organized, the module has been split, and it can be directly used as a development scratch. It using graphQL + node.js + express, and MySQL as datasource.

4 Sep 18, 2022

Welcome to the LEGO Games Repository, where you can enjoy anytime, anywhere. This is the 2021 KNU Advanced Web Programming team project.

Welcome to LEGO git repository! Here are some useful information about LEGO service. 0. Docker image Link : https://hub.docker.com/r/leibniz21c/legoga

16 Jul 21, 2022