⁂ The simple file storage service for IPFS & Filecoin

Overview


web3.storage

The simple file storage service for IPFS & Filecoin.

Getting started

This project uses node v16 and npm v7. It's a monorepo that use npm workspaces to handle resolving dependencies between the local packages/* folders.

npm install

To add a new workspace to the repo:

npm init -w ./packages/website

To run an npm script in one or more workspaces

npm run test --workspace=a --workspace=b

Testing

Each workspace has its own suite of testing tools, which you can learn more about in the relevant packages/* directory. Check out highlights in each readme using the links below, then dig into the relevant package.json file for a full list of available scripts.

Our docs website is currently hosted in a separate repo, but you can test it too!

Learn more

To learn more about the Web3.Storage service, upload a file through our friendly UI, or find detailed documentation for the JS client library, please head over to https://web3.storage

Comments
  • small (24K) file fails repeatedly to upload (thousands like it have succeeded) unless I change file extension

    small (24K) file fails repeatedly to upload (thousands like it have succeeded) unless I change file extension

    Using the upload script, I have a single file (among around 12000 files) that fails over and over. Every other file succeeds. Here's the output every time I try to run the getting-started example upload script with it:

    file:///Users/trott/Desktop/Projects/ipfs/node_modules/web3.storage/src/lib.js:155
                    throw new Error(res.message)
                          ^
    
    Error: Network connection lost.
        at pRetry.retries (file:///Users/trott/Desktop/Projects/ipfs/node_modules/web3.storage/src/lib.js:155:23)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at async RetryOperation._fn (/Users/trott/Desktop/Projects/ipfs/node_modules/p-retry/index.js:50:12) {
      attemptNumber: 6,
      retriesLeft: 0
    }
    

    I've tried uploading via the web interface too and it also seems to fail, albeit silently.

    The file is a text file with .ocr extension because it is the output from an OCR process. I have uploaded thousands of similar files so far without a problem, but this one refuses to upload. If I add .txt after the .ocr extension, the upload works. (See https://bafybeibkozbrqfcbrtvxt5j5ifswz27lvf5u3wscckicd7d7kbjw6rdlry.ipfs.dweb.link/). I have not had to do this with the other files.

    The file in question is available at https://gist.githubusercontent.com/Trott/53d20175e63466777b6c84d7a1468cf7/raw/39c3233ce0085afa354ef726e74d85ab9224b7a4/sfpl_sfh4_009_005.ocr.

    kind/bug effort/days 
    opened by Trott 25
  • Focus the website on the users content

    Focus the website on the users content

    • [ ] Removing the Docs and API cta containers (as seen Focus the website on the users content #1311 (comment) )
    • [ ] Add alternating row bg to table
    • [ ] Normalize all fonts in a row
    • [ ] Rework storage providers: EITHER OPTION 1. Reducing number of storage providers (placing them into a tooltip/modal/whatever OR reducing the total number displayed further to 2 or 3) OPTION 2. Rework into an aggregate status of either "Queued" or "Stored (xN)". Stored shows count of Active Deals. If there is less than 1, then it is Queued.
    • [ ] Move "name" to the first column
    • [ ] consider reducing the padding at the top of the page (above the main heading), maybe from 4.5rem to 2rem ?
    • [ ] drastically reduce padding/margin/min-height that is being applied between rows. (awaiting feat: Split file manager table into uploaded & pinned #1363 and will reassess after to see if additional adjustments needed)

    Rough mock of page without api and doc cta sections, and tweaks to table

    Screen Shot 2022-06-01 at 3 59 35 PM

    Original Request

    The logged in view on the web3.storage website should put the focus on the users content, and give a comfortable information density for folks working with lots of files.

    Some examples

    estuary.tech 🔗

    Screenshot 2022-05-11 at 11 50 49

    IPFS Web UI 🔗

    Screenshot 2022-05-11 at 12 07 22

    Cloudflare R2

    Screenshot 2022-05-11 at 12 00 12

    Amazon S3

    Screenshot 2022-05-11 at 11 51 06

    Digital Ocean Spaces

    kind/enhancement P2 stack/website pi/dotstorage-site-improvements 
    opened by olizilla 21
  • Add status endpoint to check pin and deal info per cid

    Add status endpoint to check pin and deal info per cid

    GET /user/uploads/:cid

    Proposed respsonse shape:

    {
      "cid": "bafy",
      "dagSize": 101,
      "pins": [{
        "peerId": "12D3KooWR1Js",
        "peerName": "who?",
        "region": "where?",
        "status": "Pinned"
      }],
      "deals": [{
        "dealId": 12345,
        "miner": "f99",
        "status": "active",
        "activation": "<iso timestamp>",
        "pieceCid":  "baga",
        "dataCid":  "bafy",
        "dataModelSelector": "Links/0/Links"
      }]
    }
    

    This is the subset of properties in the current w3 schema that have some relevance to the user, flattened. Most Batch info is removed, apart from the pieceCid, the dataCid (the cid of the batch) the dataModelSelector, which, I'm told, could be used to extract the cid from the Piece.

    The full shape of the schema that is available can be seen in the Fauna GraphQL explorer below for comparison Screenshot 2021-07-13 at 10 16 57

    opened by olizilla 21
  • feat: js client accepts user provided fetch function

    feat: js client accepts user provided fetch function

    I need to process some headers, like the cookie. because I post the request to my service and do some validation and then I create a proxy request to forward the request to your server. so, I need to add the credentials: "include" option to the requestInit option.

    therefore, I add the pre middleware to achieve that, it will pass the requestInit to the user, and the user modifies and returns it to the client before the request is sent.

    opened by duxiaofeng-github 17
  • Files Page UX Issues / Update requests

    Files Page UX Issues / Update requests

    1. "Next" button appears on 25 uploads, but there is no new page (button should appear on 26th upload)

    See screenshot here: image

    Requested fix: only show the button when 26 items are uploaded (/ not deleted from the users account).

    1. Size calculations are not immediate for all items (see above screenshot - seems like this happens for anything >10MiB).

    Requested fix: Can we put "Calculating..." for the size column if it's not available?

    1. Column definitions - is it possible to put in "tool tips" that can give definition for the domain specific information? Eg CID -> definition for a CID (cc @terichadbourne do we have definitions for each of these columns in the docs? if so natural idea might be to have a link to the docs for definitions.

    Relevant columns:

    • CID
    • Storage Providers
    • Pin Status
    1. Include a link below the table that mentions the JS library (can use the same copy + link as on the manual upload page)

    Requested fix: Include this line underneath the table "You can also upload files using the JS Client Library.

    kind/bug 
    opened by jnthnvctr 16
  • Old issue needs to be properly addressed

    Old issue needs to be properly addressed

    The import line import { Web3Storage, getFilesFromPath } from 'web3.storage'; leads to the error Module not found: Can't resolve 'ipfs-car/blockstore/memory' on ReactJS.

    I know this issue has been mentioned here #312 and here #343 but it seems it hasn't been solved yet.

    kind/bug need/triage 
    opened by OxMarco 15
  • feature: API pubkeys

    feature: API pubkeys

    So, we have this very reasonable feature request (#560), people want to delete things.

    The only problem is, a few people have been putting their API tokens in their web frontends. Not a lot, but enough of them that we don’t want to go and give those keys the ability to destroy user data and we doubt this pattern of behavior will go away no matter how many warnings we might try to put on it.

    We also have another very reasonable feature request (#571), people want to do a one-time-use signature for an upload so they can guard their token in their backend without having to proxy the upload.

    We’ve also wanted to get wallet auth for a long time (https://github.com/nftstorage/nft.storage/issues/619) and we’ve made progress on a prototype for a MetaPlex integration (https://github.com/nftstorage/nft.storage/issues/431) that is giving us a much better understanding of the developer ergonomics.

    We also have a bunch of finer grained permissions we’d like to do that we’ve explored as token scoping (#314).

    We’ve also had requests for a feature, similar to Pinata metadata, that would allow users to tag their uploads with metadata they can query against in the service.

    Finally, we want to get an API going for mutable references and keep it open to multiple decentralized mutable naming/reference systems in the future (#511).

    API PubKeys

    • Users create a public/private keypair locally and add the public key to their account as an API PubKey.
    • An API PubKey is tied to an account, same as a token is today.
    • An API PubKey is given explicit permission grants for each service feature it can use:
      • PUT
      • DELETE
      • TAG:{NAMESPACE} (details below)
    • The signer signs a message with even more fine grained permissions and tags (see below) that functions as a one-time-use token.
      • PUT:{CID}:{CARHASH}
      • DELETE:{CID}:{CARHASH}
      • TAG:{NAMESPACE}:{ USERNAME | USER WALLET ADDRESS | CHAIN | ETC }
      • TIMESTAMP

    This resolves #571 (one-time-signature), #560 (deletes), and has pretty obvious future extensions to handle everything in #314 (token scoping).

    Tagging

    Since tags have to be in the signature they are secured by the developer’s private key and can’t be tampered with by the end-user.

    The thing that sucks about tagging is that it’s in our database, which is not decentralized. But, we could just write this data to a chain. It’s quite small.

    Since the TAG:{NAMESPACE} is an explicit permission grant the developer sets in their acocunt, we could allow developers to configure our service to write data into specific protocols set on each namespace.

    I can easily imagine a feature in which we mint an NFT for each USER/TAG:{NAMESPACE} and use https://github.com/Gozala/ipnftx/blob/main/Readme.md to write this into the chain. Then developers can use the DID of the NFT as a reference rather than anything specific to our service, and if they ever leave the service we can just transfer the NFT to them. You could imagine doing the same thing if you wanted to attach an IPNS or ENS client to a tag namespace, but we’d need to do some key custody for that so we shouldn’t take it lightly (maybe next year).

    I’ve been talking about these as “tags” but we could also have a feature called /last which would function as a mutable reference to the last tag.

    In the integrations we do like Metaplex, we could embed the wallet address and signature and all the other data we want to use in order to control abuse in these endpoints that are open to any wallet signature.


    This may sound like a bunch of features all jammed together, and therefor a lot of work, but it’s actually not. This is actually pretty simple and relatively boring security (we should consider just using hawk https://github.com/mozilla/hawk).

    It also has some obvious incremental steps we can break it into and deliver user value along each step without getting stuck in a long timeline, and some good alignment with where we want to go long term (everything in the service is just a passthrough to a decentralized protocol or an index of it).

    kind/enhancement need/triage need/go-nogo-decision stack/api-protocols 
    opened by mikeal 14
  • chore: add countly analytics integration for user tracking

    chore: add countly analytics integration for user tracking

    Creating this PR for visibility of the work in progress.

    • [x] Setup countly init
    • [x] Track page views
    • [x] Track 404s
      • Simple 404 page added for this
    • [x] Track header and footer links
    • [x] Track other links to docs and other pages
    • [x] Track CTA clicks and other actions
      • [x] Login / Logout
      • [x] Files (Upload / Delete)
      • [x] Tokens (Create / Copy / Delete)

    Deployment

    • [x] Add env vars to build #202
      • NEXT_PUBLIC_COUNTLY_KEY
      • NEXT_PUBLIC_COUNTLY_URL

    (Requirements here).

    opened by zebateira 14
  • File load & pagination issues for user with many files

    File load & pagination issues for user with many files

    User observations

    • Users with lots of uploads in web3.storage experience really slow loading times.
    • Users with > 1000 uploads are not currently able to see them all in the web interface. It stops at 100 pages of 10 files (loads 1000 first ones only).

    Acceptance criteria

    • As a user I’m able to navigate through all my uploads
    • Loading uploads should be reasonably quick

    Notes

    There are 2 parts to this:

    • Have proper paginations to the page ( not loading 1000 items and then paginate in memory)
    • Look into query optimisation
    • This was first raised and looked into for ticket 1147

    https://images.zenhubusercontent.com/378893240/704a835d-8695-4f22-853d-85ad199a1058/screen_recording_2022_03_21_at_10_38_42_720p.mp4

    topic/pot topic/pagination 
    opened by mbommerez 12
  • Add contact page and social links to website

    Add contact page and social links to website

    Issue description

    The website currently lacks clear paths to contact Web3.Storage support or make contact requests. This feature would facilitate more traditional communication, as is typical of most SaaS and business websites.

    Solution

    • A dedicated contact page
    • A path to get to social channels in the footer

    In the solution shown below, we have a traditional contact page with a form and some social icons added to the footer. However, a service would have to be used to submit the form.

    Options

    @dchoi27 can you weigh in here on what you think would be better?

    1. We could use a 3rd party form, ticketing or email service with an API
    • Pro: requests are private
    • Con: For services that require a token/key, it would be exposed by the static site
    1. We could send the pre-populated query to Github to create an issue with the content from the form on the website (format described here)
    • Pro: Does not require a new service
    • Con: All submissions would be public, so for example, if a high profile publication makes a media inquiry, info that they might have considered private will be publicly exposed

    Design

    (Comps below are from the design files, which also contain field validation states) contact page comp

    footer-social-icons kind/enhancement P2 need/triage stack/website 
    opened by orvn 11
  • feat!: new website frontend and app ui

    feat!: new website frontend and app ui

    Ships new NextJS website including:

    • Marketing site
    • App UI
    • Docs

    See commit list for details, since this is an initiative with a large collection of epics

    Note: working on getting the build running correctly with Cloudflare pages

    1-home

    2-appui

    3-faq

    kind/enhancement effort/weeks 
    opened by orvn 11
  • Remove the confusing

    Remove the confusing "/mo" from the Free tier description on the Payment page

    Users that have the Free Tier available see it displayed on their Payments page https://web3.storage/account/payment/ There is a line in the description that states "...You can use our service up to 5GiB/mo without being charged." This is confusing and should instead just say "5GiB" without the "/mo".

    • [ ] remove the "/mo" copy from the free tier description.

    Looks like the line in question is here: https://github.com/web3-storage/web3.storage/blob/3ed273d75d6ebd7f733cbcb4c818d5775eca073f/packages/website/components/contexts/plansContext.js#L58

    stack/website topic/pricing 
    opened by JeffLowe 0
  • chore(main): release website 2.34.0

    chore(main): release website 2.34.0

    :robot: I have created a release beep boop

    2.34.0 (2023-01-02)

    Features

    • add synchronized, persistent tab groups (#2172) (3be2028)
    • website has /.well-known/did.json so did:web can resolve in prod (#2169) (3ed273d)

    Bug Fixes


    This PR was generated with Release Please. See documentation.

    autorelease: pending 
    opened by github-actions[bot] 0
  • quick start guide for w3up

    quick start guide for w3up

    We are going to make this quick start more substantial than the previous versions:

    • [ ] P1: w3cli
    • [ ] P2: start on server side JS app (whilst working on other docs)
    opened by heyjay44 0
Releases(website-v2.33.1)
Dustbin - Just Another Text Storage Service

Dustbin It's just another text storage service built in fastify. API Ofcouse we

Dustbin Server 25 Dec 3, 2022
💾 Offline storage, improved. Wraps IndexedDB, WebSQL, or localStorage using a simple but powerful API.

localForage localForage is a fast and simple storage library for JavaScript. localForage improves the offline experience of your web app by using asyn

localForage 21.5k Jan 4, 2023
💾 Offline storage, improved. Wraps IndexedDB, WebSQL, or localStorage using a simple but powerful API.

localForage localForage is a fast and simple storage library for JavaScript. localForage improves the offline experience of your web app by using asyn

localForage 21.5k Jan 1, 2023
Cross-browser storage for all use cases, used across the web.

Store.js Cross-browser storage for all use cases, used across the web. Store.js has been around since 2010 (first commit, v1 release). It is used in p

Marcus Westin 13.9k Dec 29, 2022
Cross domain local storage, with permissions

Cross domain local storage, with permissions. Enables multiple browser windows/tabs, across a variety of domains, to share a single localStorage. Feat

Zendesk 2.2k Jan 6, 2023
JS / CSS / files loader + key/value storage

bag.js - JS / CSS loader + KV storage bag.js is loader for .js / .css and other files, that uses IndexedDB/ WebSQL / localStorage for caching. Conside

Nodeca 86 Nov 28, 2022
A lightweight vanilla ES6 cookies and local storage JavaScript library

?? CrumbsJS ?? A lightweight, intuitive, vanilla ES6 fueled JS cookie and local storage library. Quick Start Adding a single cookie or a local storage

null 233 Dec 13, 2022
:sunglasses: Everything you need to know about Client-side Storage.

awesome-web-storage Everything you need to know about Client-side Storage. Table of Contents Introduction Browser Support Cookies Pros Cons API Useful

Varun Malhotra 420 Dec 12, 2022
An AngularJS module that gives you access to the browsers local storage with cookie fallback

angular-local-storage An Angular module that gives you access to the browsers local storage Table of contents: Get Started Video Tutorial Development

Gregory Pike 2.9k Dec 25, 2022
local storage wrapper for both react-native and browser. Support size controlling, auto expiring, remote data auto syncing and getting batch data in one query.

react-native-storage This is a local storage wrapper for both react native apps (using AsyncStorage) and web apps (using localStorage). ES6 syntax, pr

Sunny Luo 2.9k Dec 16, 2022
This is an upload script which allows you to upload to web3 storage using JS.

This is an upload script which allows you to upload to web3 storage using JS. first make sure to run npm install on the directory run script using nod

null 1 Dec 24, 2021
A javascript based module to access and perform operations on Linode object storage via code.

Linode Object Storage JS Module A javascript based module to access and perform operations on Linode object storage via code. Code Guardian Installing

Core.ai 3 Jan 11, 2022
Store your data in the world's fastest and most secure storage, powered by the blockchain technology⚡️

Store your data in the world's fastest and most secure storage, powered by the blockchain technology.

BlockDB 3 Mar 5, 2022
Expirable data storage based on localStorage and sessionStorage.

Expirable storage About The Project Expirable data storage based on localStorage and sessionStorage. Getting Started To get a local copy up and runnin

Wayfair Tech – Incubator 5 Oct 31, 2022
Browser storage interface for IndexedDB, WebSQL, LocalStorage, and in memory data with Schema and data validator.

Client Web Storage Browser storage interface for IndexedDB, WebSQL, LocalStorage, and in memory data with basic Schema and data validation. Installati

Before Semicolon 19 Sep 30, 2022
jStorage is a simple key/value database to store data on browser side

NB! This project is in a frozen state. No more API changes. Pull requests for bug fixes are welcomed, anything else gets most probably ignored. A bug

Andris Reinman 1.5k Dec 10, 2022
A simple, lightweight JavaScript API for handling browser cookies

JavaScript Cookie A simple, lightweight JavaScript API for handling cookies Works in all browsers Accepts any character Heavily tested No dependency S

null 20.2k Jan 3, 2023
Simple window.localStorage, with type safety

mini-local-storage simple window.localStorage, with type safety example // localStorage.ts import { createLocalStorage } from "mini-local-storage";

Kipras Melnikovas 4 Jan 8, 2023
This is for homework submission of Filecoin Chinese Education Series - Coding with Filecoin.

Coding-with-Filecoin-Homework 课程简介 随着互联网和大数据技术的发展,我们正愈发依赖中心化的服务来存储和处理相关数据。但这背后有两个潜在的问题:用户不能完全控制自身数据的使用与传播,且很难验证公开数据的完整性与可靠性。为了解决这两个问题,新一代的协议和点对点网络已经问世

IPFS & Filecoin 15 Jul 14, 2022
🪐 The IPFS gateway for NFT.Storage is not "another gateway", but a caching layer for NFTs that sits on top of existing IPFS public gateways.

nftstorage.link The IPFS gateway for nft.storage is not "another gateway", but a caching layer for NFT’s that sits on top of existing IPFS public gate

NFT.Storage 37 Dec 19, 2022