meta-fetcher

Simple metadata scrapper for node.js. Under the hood it uses isomorphic-unfetch
to fetch the metadata, parses it and returns it as json object.

Installation

npm install meta-fetcher

or use yarn

yarn add meta-fetcher

Usage

import metaFetcher from 'meta-fetcher';

(async () => {
  const result = await metaFetcher('https://hoppscotch.io/');
  console.log(result);

  /*
        {
            metadata: {
                website: 'https://hoppscotch.io/',
                title: 'Hoppscotch - Open source API development ecosystem',
                description: 'Helps you create requests faster, saving precious time on development.',
                banner: 'https://hoppscotch.io/banner.jpg',
                themeColor: '#202124'
            },
            socials: {
                'twitter:site': '@hoppscotch_io',
                'twitter:creator': '@hoppscotch_io'
            },
            favicons: [
                'https://hoppscotch.io/_nuxt/icons/icon_64x64.9834b3.png',
                'https://hoppscotch.io/_nuxt/icons/icon_512x512.9834b3.png'
            ]
        }
    */
})();

API

metaFetcher(input)

Takes one url string as a parameter and returns an object containing the meta-information.

input

type: string
default: 'none'

The url string to be scrapped.

Here are some related projects

page-scrapper: 📦 node.js scrapper that pulls out all links and images of a given site.

License

MIT

Node.js library that provide a cache for file metadata or file content.

@file-cache A cache library for file metadata or file content. It is useful for process that work a given series of files and that only need to repeat

Aug 6, 2022

Manage HTML metadata in Vue.js components with SSR support

✌️ 🥉 Check the next branch for Vue3 support Manage HTML metadata in Vue.js components with SSR support template ... /template script expor

Jan 1, 2023

A util for getting data and metadata for all markdown files in a given dir. Useful for building static site generators

extract-md-data A util for getting data and metadata for all markdown files in a given dir. Useful for building static site generators. Usage Given th

Jan 6, 2022

👾 Discord links with Metadata

OnDiscord.io 👾 Wrap invite links from Discord with Metadata Uses @napi-rs/canvas to render images server-side. Server metrics are shown to source leg

Mar 26, 2022

The ICPverse service to extract the metadata from an existing NFT project on the Internet Computer.

Infinity_Rank The ICPverse service to extract the metadata from an existing NFT project on the Internet Computer. Requirements for Use: npm installed

Nov 5, 2022

Dual ownership for NFTs, preventing rugs when collection metadata/royalties are changed

Multisig Ownable Opensea only allows EOAs to make changes to collections, which makes it impossible to use multisigs to secure NFT contracts since whe

Nov 24, 2022

Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collections

eznft Yet another library for generating NFT artwork, uploading NFT assets and metadata to IPFS, deploying NFT smart contracts, and minting NFT collec

Sep 21, 2022

A collection of crypto tokens (metadata, logos, links)

@deliquid/token-list (beta) A multi-chain collection of crypto tokens, including metadata, logos, and more. Installation npm i @deliquid/token-list Ba

Jun 3, 2022

Fetch publication metadata from the INSPIRE-HEP database

Zotero INSPIRE Metadata Updater This is an add-on for the excellent open-source reference manager Zotero. It is useful as most papers from arXiv will

Nov 27, 2022

Passport NFT using on-chain metadata

Passport NFT Collection Setup yarn Environment File (.env) RINKEBY_URL: Ethereum (Rinkeby) Testnet Provider endpoint MATIC_URL: Polygon (Matic Mumbai)

Jun 2, 2022

An Obsidian plugin to upload your notes' metadata to your database.

Obsidian + PostgreSQL = ❤️ An Obsidian plugin to upload your notes' metadata to your database. Features send the Dataview annotations of a page to a P

Dec 19, 2022

A plugin for Obsidian that can create input fields inside your notes and bind them to metadata fields.

Obsidian Meta Bind Plugin This plugin can create input fields inside your notes and bind them to metadata fields. New docs I am currently working on n

Jan 4, 2023

A GitHub Action to enable Pages and extract various metadata about a site

A GitHub Action to enable Pages and extract various metadata about a site. It can also be used to configure various static site generators we support as starter workflows.

For data quality enthousiasts (and dataview lovers): manage the metadata of your notes.

Metadata Menu This plugin is made for data quality enthousiasts: access and manage the metadata of your notes in Obsidian. Metadata Menu adds context

Dec 23, 2022

🎨 Dynamic Token Images and Metadata

ERC721K - Dynamic Token Images and Metadata Overview ERC721K is an ERC721 implementation with zero dependencies for off-chain data. Both the token ima

Dec 29, 2022

the music metadata you love, with IPFS hosting on nft.storage :rainbow:

Music NFT IPFS Metadata Standard for musicians pioneered by Catalog & Mint Songs. Compatible with contracts created by: Catalog Sound.xyz Manifold - W

Oct 17, 2022

Sep 30, 2022

Node 18's node:test, as a node module

node-core-test This is a user-land port of node:test, the experimental test runner introduced in Node.js 18. This module makes it available in Node.js

Dec 15, 2022

Run Node.js on Android by rewrite Node.js in Java

node-android Run Node.js on Android by rewrite Node.js in Java with the compatible API. third-party: libuvpp, libuv-java JNI code by Oracle. Build Clo

Nov 15, 2022

fetch does not wait for JS to execute

some websites will have their meta tags populated by JS. Since we are fetching the HTML contents of the page before JS has been loaded, the meta tags will never be filled

eg. the results of fetching meta tags for https://twitter.com/ManjaroLinux will all be empty strings because the meta tags are loaded by the JS

a way to solve this problem would be to use somthing like Selenium or Puppeteer to wait for the page to load before getting the HTML contents.

This is a major blocker since a lot of websites now use client side rendering and this library does not work in those cases.

opened by burhanuday 2
change the meta tag name

Hi, First of all, thank you for this npm package. It actually helps me a lot. While I was working on a project, I have noticed it has a tiny issue when I tried to get og:image from head. Upon researching, I have found that you are trying to find 'meta[name="og:image"]' where I think you should find 'meta[property="og:image"]'.
bug

opened by shoaibsharif 1
Implement bug fix for paths

Sometimes path may be a CDN url. Eg. when url = https://medium.com the favicon href = https://cdn-static-1.medium.com/_/fp/icons/Medium-Avatar-500x500.svg, and we need to just handle this case and return it.

opened by hwrdtm 1
Replace jest with ava for testing

Even tho I love ava's simplicity. Ava is still not widely used for ts project. I am still experimenting and putting together a boilerplate for future use. Once ready, I will use Ava in this project.

opened by rocktimsaikia 1

v3.1.1(Jun 12, 2021)

Adds the property attribute as a fallback to og: image lookup if not found in the name attribute. #9
Source code(tar.gz)
Source code(zip)
v3.1.0(May 4, 2021)
Notable changes (SEMVER-MINOR)

Merge similar properties into one (#8)

Remove unnecessary fields from the output ie. twitter-card

Add new field themeColor

Rename basic-metadata to metadata

Source code(tar.gz)
Source code(zip)
v3.0.2(May 4, 2021)
Notable changes (SEMVER-MINOR)

Restructure the project.

Remove tsdx and use rollup to create builds.

v3.0.1 tag is not released here on Github as I had to fall back to older v2 when the new version which used tsdx, didn't have default import support for common.js. Now with this v3.0.2 cjs default import is supported again.

Source code(tar.gz)
Source code(zip)
v3.0.0(Apr 30, 2021)
Notable changes (SEMVER-MAJOR)

Remove the additional options ie, the option to change the user agent and email

Switch to isomorphic-unfetch from node-fetch for smaller build size

Move to GitHub actions from TravisCI

Use tsdx to redo the project structure

Switch to jest from ava for testing

The version v3.0.0 starts with the new branch main. All the other previous versions are on the branch master

Source code(tar.gz)
Source code(zip)
2.0.1(Mar 8, 2021)

Early return the url if the favicon path is absolute, fixed in #7
Source code(tar.gz)
Source code(zip)
2.0.0(Oct 14, 2020)
Breaking changes

added default export support (removed export)

Tweaks

Switched to Ava from Jest

Updated the test files accordingly

removed redundant packages

Source code(tar.gz)
Source code(zip)
1.0.1(Sep 13, 2020)

Add missing repo name and homepage details in package.json
Source code(tar.gz)
Source code(zip)
1.0.0(Sep 12, 2020)

Initial release
Source code(tar.gz)
Source code(zip)