Open-source API to securely share data with customers.

Related tags

Learning resource data-sharing data-pipelines data-infrastructure
Overview

Pipebird Logo


Offer secure data pipelines to your customers' data warehouses, directly from your product.

Slack · Website · Docs

About Pipebird

GitHub starsLicense License

Direct, secure, and cost efficient data sharing. Pipebird is open source infrastructure that enables businesses to share data directly to their customers' data warehouses or databases.

Once integrated, Pipebird eliminates the need for your customers to use a third-party ETL provider or build their own pipelines to access data from your product.

Offering data pipelines natively in your product:

  • minimizes security and compliance risks created by third-parties. Launch Pipebird within your virtual cloud with the press of a button, your data never hits our servers.
  • eliminates pipeline complexity for customers and partners. Create customer-defined pipelines directly from your product using a declarative configuration language.
  • internalizes revenue previously captured by third-party ETL providers. Customers pay you for providing higher-quality data in a more secure manner.

We believe that native product data pipelines are the future of how companies will retrieve data from their vendors - we've already seen major companies like Stripe and Customer.io make big investments in building out data sharing features for their customers.

Pipebird makes it easy for any company to offer customers the same powerful data sharing features in just a few of hours.

Product features

Quick start

How it works

License

Comments
  • feat: initial implementation for BigQuery destination

    feat: initial implementation for BigQuery destination

    Fixes #114

    What is the problem?

    • Need direct loads into BigQuery

    Changes being made

    • Handles load logic for BigQuery as a destination
    • Adds staging bucket details to destination in cases where provisioned S3 is not the best option (GCS is faster in this case)

    How did you test this code?

    • Manually retested all destinations with Postgres as a source
    opened by ianedwards 0
  • refactor: added base_url to admin-ui env

    refactor: added base_url to admin-ui env

    What is the problem?

    admin-ui is missing NEXT_PUBLIC_PIPEBIRD_BASE_URL Furthermore, deployed instance don't allow inbound connections to 375 (admin-ui service)

    Changes being made

    Added NEXT_PUBLIC_PIPEBIRD_BASE_URL to docker-compose env Also allowed inbound connections to port 375

    How did you test this code?

    ran locally

    opened by cumason123 0
  • build: changed volume size from 50GB -> 20GB

    build: changed volume size from 50GB -> 20GB

    What is the problem?

    Previously when docker was failing due to insufficient storage space, we increased the volume size from 8GB -> 50GB. This was likely an over correction

    Changes being made

    Decreased volume size from 50GB -> 20GB

    opened by cumason123 0
  • feat: added pipebird-admin-ui service to port 375

    feat: added pipebird-admin-ui service to port 375

    Fixes #96

    What is the problem?

    Pipebird agent needs admin-ui

    Changes being made

    Added admin-ui to docker-compose

    How did you test this code?

    Ran locally

    opened by cumason123 0
  • Add pipebird provider UI to docker compose

    Add pipebird provider UI to docker compose

    Is your feature request related to a problem? Please describe. Right now the API layer is cumbersome for just uploading sources and views given they require a lot of json fields, and users may not want to go through the API to change these one time setup fields.

    Describe the solution you'd like We want to provide a white labeled provider UI such that companies deploying pipebird can in the ui setup sources and views without having to use the pipebird API.

    enhancement 
    opened by cumason123 0
  • Environment license key should have a default

    Environment license key should have a default

    Describe the bug Server will error out if no license key supplied. We use the license key to monitor application state on the license server. We should give pipebird a default license key if none is defined.

    bug good first issue 
    opened by cumason123 0
  • build(deps): bump vm2 from 3.9.10 to 3.9.11

    build(deps): bump vm2 from 3.9.10 to 3.9.11

    Bumps vm2 from 3.9.10 to 3.9.11.

    Release notes

    Sourced from vm2's releases.

    3.9.11

    New Features

    https://github.com/patriksimek/vm2/commit/58478a58c6f4af3c54faf4117ed5ab72d2cc8cd5: Add option require.strict to allow to load required modules in non strict mode.

    Fixes

    https://github.com/patriksimek/vm2/commit/d9a7f3cc995d3d861e1380eafb886cb3c5e2b873: Security fix.

    Changelog

    Sourced from vm2's changelog.

    v3.9.11 (2022-08-28)

    [new] Add option require.strict to allow to load required modules in non strict mode.
    [fix] Security fix.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Add documentation content and support with Mintlify

    Add documentation content and support with Mintlify

    Summary

    This PR adds the content that will power the Pipebird documentation. To edit the documentation in the future, edit the markdown files as you would with other code and push it to the main to see it reflected live or a branch to get a preview link.

    Screen Shot 2022-09-21 at 10 43 14 AM
    opened by hanywang2 0
  • temporal workers needs their own service

    temporal workers needs their own service

    Is your feature request related to a problem? Please describe.
    Temporal workers are expected to horizontally scale
    _

    Describe the solution you'd like
    They need to be separated from the deployment of the actual api. Preferably in their own container, potentially one worker per container.

    enhancement 
    opened by cumason123 0
Owner
Pipebird
Open source infrastructure for data sharing
Pipebird
GitHub https://pipebird.com/
Easy and simple way to share data via mobile’s built-in share module.

React-Mobile-Share Provides an easy and simple way to share data (such as text, url and media) via mobile’s built-in share module. It is based on Web

EncoreSky Technologies 36 Dec 28, 2022
Marketing-redirect - Redirect your customers from your marketing page to the application they're paying for

Marketing Redirect Do you have a marketing page and an app with two separate, bu

Elliot Plant 5 Feb 11, 2022
CarCollectors is website for customers who want to rent a car

CarCollectors is website that consumes the car api and the involvement api. It allows the user to browse a large collection of cars, like them, comment on them, and place a reservation order.

Soufiane Boursen 3 May 3, 2022
A superfast and easy to use knowledge base to help your customers get the info they need, when they need it most.

A superfast and easy to use knowledge base to help your customers get the info they need, when they need it most. helpkb is an open-source Next.js (A

Mark Moffat 11 Dec 5, 2022
Under the Sea is an official AWS workshop delivered by AWS SAs and AWS Partners to help customers and partners to learn about AIOps with serverless architectures on AWS.

Under the Sea - AIOps with Serverless Workshop Under the Sea is an exciting MMORPG developed by the famous entrepreneur behind Wild Rydes, the most po

AWS Samples 4 Nov 16, 2022
Alluo Decentralised App (dapp) allowing customers to interact with the Alluo protocol

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: ya

Alluo 4 Oct 21, 2022
Infisical — Sync your .env securely in seconds.

Infisical Infisical is a simple, end-to-end encrypted secrets manager for your .env files. It enables teams to securely sync and manage .env files in

null 2.7k Jan 4, 2023
Updog is an open-source social media webapp intended to allow everyday people to share their thoughts in a welcoming community.

SE701-Updog Updog is an open-source social media webapp intended to allow everyday people to share their thoughts in a welcoming community. This proje

SE 701 Team 2 UoA 14 Apr 18, 2022
Reference for How to Write an Open Source JavaScript Library - https://egghead.io/series/how-to-write-an-open-source-javascript-library

Reference for How to Write an Open Source JavaScript Library The purpose of this document is to serve as a reference for: How to Write an Open Source

Sarbbottam Bandyopadhyay 175 Dec 24, 2022
An Open-Source Platform to certify open-source projects.

OC-Frontend This includes the frontend for Open-Certs. ?? After seeing so many open-source projects being monetized ?? without giving any recognition

Open Certs 15 Oct 23, 2022
This project is for hacktoberfest to encourage new developer and open source developers to contribute to open source and improve skills which require debugging, write testable code, industry standards, problem solving and many more,

?? Blog project for hacktoberfest In this repository, you can find issues related to a blog project that is built on top of Next.js. The project is a

Shikhar 4 Oct 9, 2022
This is a project for open source enthusiast who want to contribute to open source in this hacktoberfest 2022. 💻 🎯🚀

HACKTOBERFEST-2022-GDSC-IET-LUCKNOW Beginner-Hacktoberfest Need Your first pr for hacktoberfest 2k22 ? come on in About Participate in Hacktoberfest b

null 8 Oct 29, 2022
LunaSec - Open Source Security Software built by Security Engineers. Scan your dependencies for Log4Shell, or add Data Tokenization to prevent data leaks. Try our live Tokenizer demo: https://app.lunasec.dev

Our Software We're a team of Security Engineers on a mission to make awesome Open Source Application Security tooling. It all lives in this repo. Here

LunaSec 1.2k Jan 7, 2023
🧙 Mage is an open-source data management platform that helps you clean data and prepare it for training AI/ML models.

Intro Mage is an open-source data management platform that helps you clean data and prepare it for training AI/ML models. What does this do? The curre

Mage 2.5k Jan 4, 2023
This project is built with JavaScript, Webpack, HTML & CSS, Leaderboard api. When user clicks on Refresh button it hits the api and responds with the data, The user can also post data to the api

leaderboad Description the project. this project is about the leaderboad i did during Microverse to build a website for adding Data to the API and fet

Emmanuel Moombe 4 May 30, 2022
Open Harvest is a technology designed farmers to publicly share the details of their crops/land to allow all farmers to stay informed on the best crops they can grow without risking a surplus crisis.

OpenHarvest OpenHarvest is a web application designed to balance farming production in India. Farmers in India are struggling with marketing their cro

Call for Code® with The Linux Foundation 11 Dec 8, 2022
Ready to manipulate partitions file? Create a custom partition, apply custom security system, hide the partition and share your hidden data on the www

Paranoia ?? Ready to manipulate partitions file? Create a custom partition, apply custom security system, hide the partition and share your hidden dat

Alice Snow 3 Dec 29, 2022
Fun way to share some of your (already) public StackOverflow user data to the internet.

ProfileOverflow A simple app leveraging StackExchange APIs to show some StackOverflow accounts information. Made for fun during a weekend as practice.

Jacky Efendi 5 Sep 14, 2022
An open source API wrapper for TechHost API.

TechHost API Wrapper An open source API wrapper for TechHost API. Badges Installation Install techhost-api-wrapper with npm. npm install techhost-api-

Eight∞ 4 Jun 23, 2022