An API that allows you to scrape blog posts and articles and get a list of notes or a summary back.

Overview

EZAI-Web-Scraper

An API that allows you to scrape blog posts and articles and get a list of notes or a summary back.

Recommendations

  1. Use browserless.io for scraping instead of the headless chromium that comes with puppeteer. It's faster and more reliable.
  2. For better results changed the language model to text-curie-001 or text-davinci-002. The default model is cheap, but not the greatest.
  3. For easiest deployment, use the docker file in the dist folder (Make sure you add the .env variables).
  4. If you make a cool feature or find a bug, please consider contributing!

Enviroment Variables

OPENAI_API_KEY={YOUR API KEY} (Optional. Key can be provided in the request headers)

BROWSERLESS_API_KEY={YOUR API KEY} (Optional. Only needed if you plan on using browserless.io)

PORT={YOUR CHOSEN PORT} (Required)

How To Run

Command Line

Developement: npm run test

Production: npm run start

Docker

(Does not work on Apple M1 chips)

CD into the "dist" folder and build the image.

Run the image, but make sure to include the enviroment variables.

I have tested this project on render.com and Google Cloud Run. Both work well and are a good choice.

API ENDPOINTS

/notes

  • Method: POST
  • Parameters:
    • Headers:
      • Key
        • Description: Your OpenAI API key (Only use if you didnt set the OPEN_API_KEY enviroment variable)
        • Type: String
        • Required: False
    • Body
      • URI
        • Description: A link to the website you would like to have notes made from.
        • Type: String
        • Required: True

/summary

  • Method: POST
  • Parameters:
    • Headers:
      • Key
        • Description: Your OpenAI API key (Only use if you didnt set the OPEN_API_KEY enviroment variable)
        • Type: String
        • Required: False
    • Body
      • URI
        • Description: A link to the website you would like to have summarized
        • Type: String
        • Required: True

Example Request

const notes = await fetch(https://myapi.com/notes, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Key: "Your OpenAI key"
  },
  body: {
    URI: "https://blog.com/1234"
  }
)
You might also like...

A Zotero add-on that scans your Markdown reading notes, tags the associated Zotero items, and lets you open notes for the Zotero items in Obsidian.

A Zotero add-on that scans your Markdown reading notes, tags the associated Zotero items, and lets you open notes for the Zotero items in Obsidian.

Zotero Obsidian Citations Adds colored tags to Zotero items that have associated Markdown notes stored in an external folder. Open an associated Markd

Sep 18, 2022

📝 You Can Create Your Own Short Notes With The Help of Sticky-Notes Website.

Hi 👋 , I'm Sneh Agrawal A passionate Web developer from India 🔭 I’m currently working on Chatting Website Chit-Chat 📫 How to reach me on My Gmail A

Feb 23, 2022

"To-do list" is a tool that helps to organize your day. It simply lists the things that you need to do and allows you to mark them as complete. You will build a simple website that allows for doing that, and you will do it using ES6 and Webpack!

To-do-list Description "To-do list" is a tool that helps to organize your day. It simply lists the things that you need to do and allows you to mark t

Aug 19, 2022

An ongoing curated list of frameworks, books, articles, talks, screencasts, recordings, libraries, learning tutorials and shiny resources about Javascript Development.

An ongoing curated list of frameworks, books, articles, talks, screencasts, recordings, libraries, learning tutorials and shiny resources about Javascript Development.

Javascript Frameworks Development Welcome to the world of Javascript Frameworks. An ongoing curated list of frameworks, books, articles, talks, screen

Jul 31, 2022

Free to use and not for sale. This repo uses scrape or data.json

Aine-MD This script is free, if caught sold, this script will be deleted immediately. Don't forget to follow my github Script ini gratis, Jika ketahua

Sep 17, 2022

It uses JavaScript and a web browser (for example, Firefox) to scrape tweets.

It uses JavaScript and a web browser (for example, Firefox) to scrape tweets.

Twitter JS Scraper Introduction There are many tools available for collecting tweets. Some of these tools make use of the official Twitter API, which

Jun 24, 2022

Scrape From primbon.com

scrape-primbon Instalation : npm i scrape-primbon Usage: const { Primbon } = require('scrape-primbon') const primbon = new Primbon() Example const {

Apr 9, 2022

node.js script to scrape all top meme templates from https://imgflip.com/memetemplates

imgflip-scraper node.js script to scrape all top meme templates from https://imgflip.com/memetemplates Acknowledgements imgflip Run Locally Clone the

Sep 18, 2022

Backend API Rest application for ShortLink, a URL shortening service where you enter a valid URL and get back an encoded URL

ShortLink - The Shortest URL (API) Sobre o Projeto | Como Usar | Importante! Sobre o projeto The Shortest URL é um projeto back-end de ShortLink, um s

Mar 22, 2022
Owner
null
Easily open daily notes and periodic notes in new pane; customize periodic notes background; quick append new line to daily notes.

Obsidian daily notes opener This plugin adds a command for opening daily notes in a new pane (so that a keyboard shortcut could be used!) and gives ex

Xiao Meng 13 Aug 22, 2022
Show a helpful summary of test results in GitHub Actions CI/CD workflow runs

Test Summary Produce an easy-to-read summary of your project's test data as part of your GitHub Actions CI/CD workflow. This helps you understand at-a

Test Summary 273 Sep 19, 2022
This blog is still under development! I present a project scope for science articles, it can now be used in production! But there are some details that need to be put up front.

Science-Blog ?? Attention! This blog is still under development! I present a project scope for science articles, it can now be used in production! But

Raissadev 2 Sep 19, 2022
📈 AI powered web scraper that let's you scrape anything you want from the web including google search results

AI powered web scraper that let's you scrape anything you want from the web including google web search results from your terminal. And at the same ti

Udhay rajeev 22 Sep 11, 2022
simple-remix-blog is a blog template built using Remix and TailwindCSS. Create your own blog in just a few minutes!

simple-remix-blog is a blog template built using remix.run and TailwindCSS. It supports markdown and MDX for the blog posts. You can clone it and star

José Miguel Álvarez Vañó 2 Sep 14, 2022
Scrape data from Instagram without applying for the authenticated API 🎯

scraper-instagram Scrape data from Instagram without applying for the authenticated API. Getting started Prerequisites NodeJS NPM or Yarn Install From

fabi.stehle 8 Sep 22, 2022
Discovers and parses news, blog and podcast posts from any website

post-feed-reader A library to fetch news, blog or podcast posts from any site. It works by auto-discovering a post source, which can be an RSS/Atom/JS

Guilherme Chaguri 4 Mar 14, 2022
Follow along with blog posts, code samples, and practical exercises to learn how to build serverless applications from your local Integrated development environment (IDE).

Getting started with serverless This getting started series is written by the serverless developer advocate team @AWSCloud. It has been designed for d

AWS Samples 51 Sep 18, 2022