Reference to #172 that was recently merged. @vasco-santos

This update caused a significant amount of NFTs to become useless on major marketplace platforms.

NFTs can be rendered as an iframe with HTML/CSS/JS:

Animation_url also supports HTML pages, allowing you to build rich experiences and interactive NFTs using JavaScript canvas, WebGL, and more. Scripts and relative paths within the HTML page are now supported. However, access to browser extensions is not supported. https://docs.opensea.io/docs/metadata-standards#metadata-structure

This change forced content-security-policy headers to be overwritten; now, all of those NFTs relying on iframe rendering cannot load content.

While I agree with the sentiment mentioned in #172 (wrt @Gozala), IPFS (nftstorage.link) as a protocol should not be responsible for setting security standards that could be damaging to legitimate use cases.

I've noticed a behavior where the range request for a large file instead returns a 200, and starts serving the entire file. This is less than ideal, as it may cause a very large download to start.

Ideally, the request can wait until the range request can be satisfied, or alternatively, some sort of error should be returned. Ex:

curl -v -r 29788705059-29788770616 https://bafybeic3w7mp6pteuvzjxnmdt65r6pttth6mzgcjfzna6fugb66xs2z3tq.ipfs.nftstorage.link/ > /dev/null

Observe that the response is a 200 response and curl attempts to download the entire file:

< HTTP/2 200 

This PR adds forbidden content types to avoid abuse as we were seeing ~28% of traffic with bin files from last 12 hours https://dash.cloudflare.com/fffa4b4363a7e5250af8357087263b3a/nftstorage.link/analytics/traffic?content-type=bin

This can also be problematic for gateway being blocked by other parties and we should investigate other types to block.

CF may still just put bin on other kind of types if does not know about. Anyway, we should just block octet-stream to start. This should likely evolve into a wrangler secret / Admin feature.

Integrate goodbits into nftstorage.link.

Motivation

We introduced CSP https://github.com/nftstorage/nftstorage.link/pull/172 aiming to block phishing websites from operating. More than phishing websites, this measure brings to the table additional long term benefits:

• prevent NFTs in the wild to have centralized dependencies instead of same origin content addressable content
• ipfs:// browser handling will require all external sources to also rely on IPFS protocol (we can overcome these pains earlier on)

There are already occurrences on the wild of NFTs relying on primitives that are now blocked by default via CSP. We are shipping https://github.com/nftstorage/goodbits as a place where we can manually add these NFTs, in order to bypass CSP for them after manual validation.

Implementation details

Similarly to what we do in reads pipeline for denylist, a new KV is created for optimal access from CF Workers on the gateway operation.

This PR includes:

• GOODBITS KV
• Cron job that recurrently downloads goodbits list and updates KV with the new occurrences
• gateway to bypass goodbits with appropriate tag to bypass CSP

Needs:

• [x] https://github.com/nftstorage/goodbits/pull/1

Hi,

Issue

When generative tokens (html CAR packages) are viewed from the gateway. Direct links to ipfs (i.e ipfs://<CID>) are not sanitized:

DOES NOT WORK -> https://bafybeifsiea24k46cgebr6pcleqmqe6kyqzh4pw4rexep4xdr337625oyi.ipfs.nftstorage.link/ WORKS -> https://ipfs.io/ipfs/bafybeifsiea24k46cgebr6pcleqmqe6kyqzh4pw4rexep4xdr337625oyi/

Though looking at the source code I'm not such of the artist's logic there.

Addresses wasm module loading reported in #175 For details see https://github.com/WebAssembly/content-security-policy/blob/main/proposals/CSP.md#the-wasm-unsafe-eval-source-directive

This PR adds perma-cache client to client package. There are potential improvements on the pipeline (like converting given URLs to nftstorage.link if they are IPFS urls), but keeping scope smaller for initial implementation.

It includes:

• all 4 initial methods (put, delete, list, status)
  • Support for batching, retries, and rate limit control as needed
• smoke tests with mock API (we can move on later to spin up the actual API + DB + Gateway)
• workflow tests updated to run bundlesize and pre mocking
• client docs separated between scopes, one (unchanged) Markdown file for gateway utils and other for perma cache docs.

Closes #92

BREAKING CHANGE: GET /perma-cache/status renamed to /perma-cache/account

We currently don't have sourcemaps in sentry. These are the only differences I could find from the working source maps in web3storage project per changes in https://github.com/web3-storage/web3.storage/pull/1033

Per Cloudflare durable limits that lead to temporary fix on https://github.com/nftstorage/nftstorage.link/pull/32 , this PR adds solution proposed in #17 by replacing Durable Object logic to prevent rate limits by a DNS based solution that enables us to rely on CF dashboard to stipulate rate limits.

Added rate limit rule https://dash.cloudflare.com/fffa4b4363a7e5250af8357087263b3a/nftstorage.link/security/waf/rate-limiting-rules pinata.nftstorage.link by X-Forwarded-For. Internally within the worker, we track 429 errors already and wey will be counted in metrics.

TODO:

• [x] Add domain to CF nftstorage.link zone
• [x] Setup rate limits to pinata.nftstorage.link

Closes #17

:robot: I have created a release beep boop

1.15.0 (2023-01-03)

Features

• add csp report endpoint (#209) (c1d3b26)

This PR was generated with Release Please. See documentation.

:robot: I have created a release beep boop

1.14.6 (2022-12-16)

Bug Fixes

• add dotstorage apis to csp allowlist (#204) (8c75d93)

This PR was generated with Release Please. See documentation.

:robot: I have created a release beep boop

2.3.3 (2023-01-03)

Bug Fixes

• api testing setup (#206) (0bd363e)
• api wrangler routes need pattern (#210) (b2b1c85)
• normalized url with subdomain must be w3s.link (182641a)

This PR was generated with Release Please. See documentation.

:robot: I have created a release beep boop

2.0.1 (2023-01-02)

Bug Fixes

• api testing setup (#206) (0bd363e)

This PR was generated with Release Please. See documentation.

Being able to fetch content as a CAR from nftstorage.link would be rad. It'd allow folks to fetch and verify content themselves, with the nftstorage.link speed boost.

We need it for the web3.storage client.get method. We could hit the nftstorage.link endpoint directly from that rather than via the api worker. We're getting reports of larger files failing to fetch via GET /cid/:cid which is currently just a proxy for ipfs.io. e.g https://github.com/web3-storage/web3.storage/issues/840

The implementation should just use the new /ipfs/:cid?format=car search param that landed in go-ipfs v0.13. If other gateways take a while to adopt this we should implement gets as hitting ipfs.io only, so at least we start to benefit from the cdn / caching behavioiur for them in the meantime.

Expected

When I visit https://nftstorage.link/ipfs/f0170122041cd1126759340060ea717da677ccf97619097af67a60e25fa526452d47e2e34, that is, an "f" base CID, the resource should resolve correctly

Actual

Server responds with 400 invalid CID: f0170122041cd1126759340060ea717da677ccf97619097af67a60e25fa526452d47e2e34: To parse non base32 or base58btc encoded CID multibase decoder must be provided in body

Notes

Taking the equivalent base32 encoding from https://cid.ipfs.io/#f0170122041cd1126759340060ea717da677ccf97619097af67a60e25fa526452d47e2e34, I can resolve the resource on the same gateway (bafybeicbzuism5mtiada5jyx3jtxzt4xmgijpl3huyhcl6ssmrjni7rogq)