simple metadata scrapper for node.js

Overview

meta-fetcher

GitHub Workflow Status (branch) npm

Simple metadata scrapper for node.js. Under the hood it uses isomorphic-unfetch
to fetch the metadata, parses it and returns it as json object.


Installation

npm install meta-fetcher

or use yarn

yarn add meta-fetcher

Usage

import metaFetcher from 'meta-fetcher';

(async () => {
  const result = await metaFetcher('https://hoppscotch.io/');
  console.log(result);

  /*
        {
            metadata: {
                website: 'https://hoppscotch.io/',
                title: 'Hoppscotch - Open source API development ecosystem',
                description: 'Helps you create requests faster, saving precious time on development.',
                banner: 'https://hoppscotch.io/banner.jpg',
                themeColor: '#202124'
            },
            socials: {
                'twitter:site': '@hoppscotch_io',
                'twitter:creator': '@hoppscotch_io'
            },
            favicons: [
                'https://hoppscotch.io/_nuxt/icons/icon_64x64.9834b3.png',
                'https://hoppscotch.io/_nuxt/icons/icon_512x512.9834b3.png'
            ]
        }
    */
})();

API

metaFetcher(input)

Takes one url string as a parameter and returns an object containing the meta-information.

input

type: string
default: 'none'

The url string to be scrapped.


Related

Here are some related projects

  • page-scrapper: 📦 node.js scrapper that pulls out all links and images of a given site.

License

MIT

You might also like...

Node.js library that provide a cache for file metadata or file content.

@file-cache A cache library for file metadata or file content. It is useful for process that work a given series of files and that only need to repeat

Aug 6, 2022

Manage HTML metadata in Vue.js components with SSR support

Manage HTML metadata in Vue.js components with SSR support

✌️ 🥉 Check the next branch for Vue3 support Manage HTML metadata in Vue.js components with SSR support template ... /template script expor

Nov 23, 2022

A util for getting data and metadata for all markdown files in a given dir. Useful for building static site generators

extract-md-data A util for getting data and metadata for all markdown files in a given dir. Useful for building static site generators. Usage Given th

Jan 6, 2022

👾 Discord links with Metadata

👾 Discord links with Metadata

OnDiscord.io 👾 Wrap invite links from Discord with Metadata Uses @napi-rs/canvas to render images server-side. Server metrics are shown to source leg

Mar 26, 2022

The ICPverse service to extract the metadata from an existing NFT project on the Internet Computer.

Infinity_Rank The ICPverse service to extract the metadata from an existing NFT project on the Internet Computer. Requirements for Use: npm installed

Nov 5, 2022

Dual ownership for NFTs, preventing rugs when collection metadata/royalties are changed

Multisig Ownable Opensea only allows EOAs to make changes to collections, which makes it impossible to use multisigs to secure NFT contracts since whe

Nov 24, 2022

Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collections

eznft Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collec

Sep 21, 2022

A collection of crypto tokens (metadata, logos, links)

@deliquid/token-list (beta) A multi-chain collection of crypto tokens, including metadata, logos, and more. Installation npm i @deliquid/token-list Ba

Jun 3, 2022

Fetch publication metadata from the INSPIRE-HEP database

Zotero INSPIRE Metadata Updater This is an add-on for the excellent open-source reference manager Zotero. It is useful as most papers from arXiv will

Nov 14, 2022

Passport NFT using on-chain metadata

Passport NFT Collection Setup yarn Environment File (.env) RINKEBY_URL: Ethereum (Rinkeby) Testnet Provider endpoint MATIC_URL: Polygon (Matic Mumbai)

Jun 2, 2022

An Obsidian plugin to upload your notes' metadata to your database.

An Obsidian plugin to upload your notes' metadata to your database.

Obsidian + PostgreSQL = ❤️ An Obsidian plugin to upload your notes' metadata to your database. Features send the Dataview annotations of a page to a P

Nov 18, 2022

A plugin for Obsidian that can create input fields inside your notes and bind them to metadata fields.

Obsidian Meta Bind Plugin This plugin can create input fields inside your notes and bind them to metadata fields. New docs I am currently working on n

Nov 16, 2022

A GitHub Action to enable Pages and extract various metadata about a site

A GitHub Action to enable Pages and extract various metadata about a site. It can also be used to configure various static site generators we support as starter workflows.

Nov 29, 2022

For data quality enthousiasts (and dataview lovers): manage the metadata of your notes.

Metadata Menu This plugin is made for data quality enthousiasts: access and manage the metadata of your notes in Obsidian. Metadata Menu adds context

Nov 22, 2022

🎨 Dynamic Token Images and Metadata

ERC721K - Dynamic Token Images and Metadata Overview ERC721K is an ERC721 implementation with zero dependencies for off-chain data. Both the token ima

Nov 18, 2022

the music metadata you love, with IPFS hosting on nft.storage :rainbow:

the music metadata you love, with IPFS hosting on nft.storage :rainbow:

Music NFT IPFS Metadata Standard for musicians pioneered by Catalog & Mint Songs. Compatible with contracts created by: Catalog Sound.xyz Manifold - W

Oct 17, 2022

🎨 Dynamic Token Images and Metadata

ERC721K - Dynamic Token Images and Metadata Overview ERC721K is an ERC721 implementation with zero dependencies for off-chain data. Both the token ima

Sep 30, 2022

Node 18's node:test, as a node module

node-core-test This is a user-land port of node:test, the experimental test runner introduced in Node.js 18. This module makes it available in Node.js

Nov 10, 2022

Run Node.js on Android by rewrite Node.js in Java

node-android Run Node.js on Android by rewrite Node.js in Java with the compatible API. third-party: libuvpp, libuv-java JNI code by Oracle. Build Clo

Nov 15, 2022
Comments
  • fetch does not wait for JS to execute

    fetch does not wait for JS to execute

    some websites will have their meta tags populated by JS. Since we are fetching the HTML contents of the page before JS has been loaded, the meta tags will never be filled

    eg. the results of fetching meta tags for https://twitter.com/ManjaroLinux will all be empty strings because the meta tags are loaded by the JS

    a way to solve this problem would be to use somthing like Selenium or Puppeteer to wait for the page to load before getting the HTML contents.

    This is a major blocker since a lot of websites now use client side rendering and this library does not work in those cases.

    opened by burhanuday 2
  • change the meta tag name

    change the meta tag name

    Hi, First of all, thank you for this npm package. It actually helps me a lot. While I was working on a project, I have noticed it has a tiny issue when I tried to get og:image from head. Upon researching, I have found that you are trying to find 'meta[name="og:image"]' where I think you should find 'meta[property="og:image"]'.

    bug 
    opened by shoaibsharif 1
  • Implement bug fix for paths

    Implement bug fix for paths

    Sometimes path may be a CDN url. Eg. when url = https://medium.com the favicon href = https://cdn-static-1.medium.com/_/fp/icons/Medium-Avatar-500x500.svg, and we need to just handle this case and return it.

    opened by hwrdtm 1
  • Replace jest with ava for testing

    Replace jest with ava for testing

    Even tho I love ava's simplicity. Ava is still not widely used for ts project. I am still experimenting and putting together a boilerplate for future use. Once ready, I will use Ava in this project.

    opened by rocktimsaikia 1
Releases(v3.1.1)
  • v3.1.1(Jun 12, 2021)

  • v3.1.0(May 4, 2021)

    Notable changes (SEMVER-MINOR)

    • Merge similar properties into one (#8)
    • Remove unnecessary fields from the output ie. twitter-card
    • Add new field themeColor
    • Rename basic-metadata to metadata
    Source code(tar.gz)
    Source code(zip)
  • v3.0.2(May 4, 2021)

    Notable changes (SEMVER-MINOR)

    • Restructure the project.
    • Remove tsdx and use rollup to create builds.

    v3.0.1 tag is not released here on Github as I had to fall back to older v2 when the new version which used tsdx, didn't have default import support for common.js. Now with this v3.0.2 cjs default import is supported again.

    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Apr 30, 2021)

    Notable changes (SEMVER-MAJOR)

    • Remove the additional options ie, the option to change the user agent and email
    • Switch to isomorphic-unfetch from node-fetch for smaller build size
    • Move to GitHub actions from TravisCI
    • Use tsdx to redo the project structure
    • Switch to jest from ava for testing

    The version v3.0.0 starts with the new branch main. All the other previous versions are on the branch master

    Source code(tar.gz)
    Source code(zip)
  • 2.0.1(Mar 8, 2021)

  • 2.0.0(Oct 14, 2020)

    Breaking changes

    • added default export support (removed export)

    Tweaks

    • Switched to Ava from Jest
    • Updated the test files accordingly
    • removed redundant packages
    Source code(tar.gz)
    Source code(zip)
  • 1.0.1(Sep 13, 2020)

  • 1.0.0(Sep 12, 2020)

Owner
Rocktim
Tweaking design and code
Rocktim
Node.js Application Configuration

Configure your Node.js Applications release notes Introduction Node-config organizes hierarchical configurations for your app deployments. It lets you

Loren West 5.8k Nov 22, 2022
Run any command on specific Node.js versions

Run any command on specific Node.js versions. Unlike nvm exec it: can run multiple Node.js versions at once can be run programmatically is 5 times fas

ehmicky 595 Nov 23, 2022
Node.js object hash library with properties/arrays sorting to provide constant hashes. It also provides a method that returns sorted object strings that can be used for object comparison without hashes.

node-object-hash Tiny and fast node.js object hash library with properties/arrays sorting to provide constant hashes. It also provides a method that r

Alexander 73 Oct 7, 2022
Node.js CLI tool to visualize an aggregate list of your dependencies' licenses

licenseye Node.js CLI tool to visualize an aggregate list of your project's dependencies' licenses. Install Yarn yarn global add licenseye NPM npm ins

Liran Tal 35 Oct 28, 2022
Abstracts execution of tasks in parallel using Node.js cluster.

cluster-map Abstracts execution of tasks in parallel using Node.js cluster. It is a high level abstraction around a common pattern used to delegate a

Gajus Kuizinas 27 Jul 3, 2022
Clock and task scheduler for node.js applications, providing extensive control of time and callback scheduling in prod and test code

#zeit A node.js clock and scheduler, intended to take place of the global V8 object for manipulation of time and task scheduling which would be handle

David Denton 12 Dec 21, 2021
IDLIX Scrapper API with cookie

IDLIX Scrapper About IDLIX Scrapper, IDLIX API Hak Cipta Projek ini dilindungi oleh MIT yang dimana penggunanya boleh menggunakan, mendistribusikan, m

Annurdien Rasyid 9 Oct 27, 2022
Grupprojekt för kurserna 'Javascript med Ramverk' och 'Agil Utveckling'

JavaScript-med-Ramverk-Laboration-3 Grupprojektet för kurserna Javascript med Ramverk och Agil Utveckling. Utvecklingsguide För information om hur utv

Svante Jonsson IT-Högskolan 3 May 18, 2022
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 13 Jul 1, 2022