There is something incredibly important to note- projects leading up to ID 3.8 million or so should be saved as .sb files, not .sb3.

Additionally: Projects made in Scratch 1 will return a large binary file with a "ScratchV0" header (the file is the entire project.) Projects made in Scratch 2 and 3 will return a .JSON file (the .json, only, not any of the project's assets.)

Some sort of routine should be written to determine if the file is a Scratch 1 project or a Scratch 2/3 project.

Isn't it better to save just the JSON files? this way less space is taken up and I'm pretty sure assets will remain publicly accessible.

PR #8269 in scratch-gui adds the token to the project URL, but not the asset URL, and it's already been merged. Line #62 is the URL for the asset, and as you can see it doesn't contain an extra query parameter, unlike line #44, whose URL also contains the token as a query param. Since it has already been merged, and PR #6773 doesn't modify the assets url, it's pretty safe to assume that assets will remain public.

The file modified in both the PRs that actually bring about this change is src/lib/storage.js

Unfortunately, uploading all of the project files to this repository is pretty much impossible.

GitHub only allows you to upload 100 files at a time. So I tried compressing 10,000 project files into a 7zip archive. That doesn't work either. GitHub has a 25mb limit on file sizes.

So I decided to just upload the archive to The Internet Archive.

Here's the link: https://archive.org/details/scratch-projects

@MagentaSuperNova you appear to be the only person with the ability to upload to the scratch-projects item on Archive.org. Is there a way that this could be more community accessible so that more project uploads don't have to go through you?

I'm archiving projects 200000-210000 using SB-Auto, and some projects aren't being downloaded--for example, 201169 and 201171 were downloaded, but 201170 was skipped. Is this normal?

I got the below error shortly after starting. I updated everything and re-cloned the repo, still no luck.

Wrote project 50738
Wrote project 50737
Wrote project 50736
node:events:505
      throw er; // Unhandled 'error' event
      ^

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at new NodeError (node:internal/errors:377:5)
    at _write (node:internal/streams/writable:320:11)
    at Writable.write (node:internal/streams/writable:335:10)
    at PassThrough.ondata (node:internal/streams/readable:766:22)
    at PassThrough.emit (node:events:527:28)
    at Readable.read (node:internal/streams/readable:539:10)
    at flow (node:internal/streams/readable:1023:34)
    at resume_ (node:internal/streams/readable:1004:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on WriteStream instance at:
    at WriteStream.onerror (node:internal/streams/readable:785:14)
    at WriteStream.emit (node:events:527:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ERR_STREAM_WRITE_AFTER_END'
}

Node.js v18.1.0

In the data collection, there are gaps of projects that are not archived, that we need to archive.

also, on an unrelated note, we can possibly archive projects after unshared projects block, because of scratchdb, we can try and get their jsons, and download the assets they reference, and zip it up, as I don’t think scratch has restrictions on what assets you can access.

I've seen some messages about archiving scratch project files. That led me to this repository and so I saw this and wrote a script to get the json from the scratch api. I've successfully checked and downloaded files from the projects with ids up to a little less than 100k. I was wondering how the json files downloaded are converted to sb3 as when opening an actual sb3 I downloaded off the sight it was filled with garbage not json.

It appears all of the .sb3 files that have been archived so far are 0 bytes in size and are blank (excluding the first one for some reason). However, when you go to https://projects.scratch.mit.edu/5 (or any other project) and download the file, everything is there just as it should be.

This appears to be a problem with getsb3 (https://sb3.micahlindley.com) and because the script is going to getsb3 to download the project files and not the Scratch API, blank project files are being archived.

I am working on an implementation of Forkphors's sb downloader that automates the process of downloading Scratch project files, zipping the project JSON and the assets, and selecting the correct file extension.

The goal is to have the user enter a starting and ending project IDs, and simply click a button to start the process. Since this is all done in the browser, it will be super easy for anyone to use.

Since the original index.js file wasn't actually archiving projects (#2), and was causing errors (#1), I decided to re-write it.

It gets the project file from the Scratch API.

I am aware it is not the fastest, but you don't want to flood the Scratch API (and get IP banned), and this way the next project download isn't started until the current one has finished.

Archive Contribution - 210001 through 210500

I wanted to contribute a little bit. Hopefully, everything is in the correct format :).

Link: https://archive.org/details/scratch-projects_202208 Project IDs: 210001-210500

According to the readme, there have already been archived 100,000 projects, but dogcatpuppylover said they were working on 200,000-210,000, so it should prob be updated

Or in the instructions, add a step that asks you to make a new folder called projects or run mkdir projects. Currently, if you follow the instructions you'll get an error saying that the projects directory isn't found.