A TurboRepo local cache server which uploads artifact cache to GH artifacts.

Overview

TurboRepo Github Artifacts action

This action allows you use Github artifacts as TurboRepo remote cache server.

How it works?

It's starts a local TurboRepo server (on port 9080) and uses Github artifacts as a caching storage.

Setup

  1. Add in your workflow.yml the following section before TurboRepo runs:
- name: TurboRepo local server
  uses: felixmosh/turborepo-gh-artifacts@v1
  with:
    repo-token: ${{ secrets.GITHUB_TOKEN }}
    server-token: ${{ secrets.TURBO_SERVER_TOKEN }}
  1. Make turbo repo work with the local server
- name: Build
  run: yarn build --api="http://127.0.0.1:9080" --token="${{ secrets.TURBO_SERVER_TOKEN }}" --team="foo"

That's it 😋 .

Action inputs

The action has 2 required inputs:

  1. repo-token - A Github token with repo permission, usually the default secrets.GITHUB_TOKEN is enough.
  2. server-token - An auth token to ensure that your code interacting with the local server.

Working Example

Working example of the entire setup, based on npx create-turbo@latest.

Useful Links

Comments
  • `outputs: []` not cached

    `outputs: []` not cached

    We're trying to use turborepo-gh-artifacts in a workflow, and we've got it integrated, but it never caches anything for our lint/tests:

    Post job cleanup.
    Found server pid: 1881
    Killing server pid: 1881
    There is nothing to upload.
    Server logs:
    Cache dir: /home/runner/work/_temp/turbo-cache
    Local Turbo server is listening at http://127.0.0.1:9080
    

    Looking at your example run, it's clear that it is uploading cache items. However, you're also calling build vs. lint, which is what we're trying to do (i.e., run tests).

    From the Turbo Repo docs, it implies that such tasks would still get cached:

    If your task does not emit any files (e.g. unit tests with Jest), set outputs to an empty array (i.e. []). Turborepo will automagically cache the logs for you.

    Your code appears to assume there will be a file produced: https://github.com/felixmosh/turborepo-gh-artifacts/blob/master/src/utils/uploadArtifacts.ts#L33-L44

    Is this a bug, or confusion on our part?

    opened by humphd 13
  • GitHub Action issue [Need help]

    GitHub Action issue [Need help]

    Hello, hope you are having a wonderful day.

    I copied https://github.com/felixmosh/turborepo-gh-artifacts-example/blob/master/.github/workflows/test.yaml, and ran it with the requirements this repositories README.md says, I receive Unknown or unexpected option: --api during Build. Could someone please assist me with this setup as I am certain it is incorrect?

    The TurboRepo.yml file contents are displayed below.

    name: TurboRepo Github Artifacts action
    
    on:
      workflow_dispatch:
      push:
        branches:
          - main
    jobs:
      turborepo-github-artifacts-action:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v3
    
          - uses: actions/setup-node@v3
            with:
              node-version: 16
    
          - name: Get yarn cache directory path
            id: yarn-cache-dir-path
            run: echo "::set-output name=dir::$(yarn cache dir)"
    
          - name: Cache node_modules
            uses: actions/cache@v3
            id: yarn-cache
            with:
              path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
              key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
              restore-keys: |
                ${{ runner.os }}-yarn-
    
          - name: Install Packages
            run: yarn install --frozen-lockfile
            env:
              CI: true
    
          - name: Turborepo local server
            uses: felixmosh/turborepo-gh-artifacts@v2
            with:
              repo-token: ${{ secrets.TOKEN_TURBOREPO }} # GITHUB_TOKEN
              server-token: ${{ secrets.TURBO_SERVER_TOKEN }}
    
          - name: Build
            run: yarn build --api="http://127.0.0.1:9080" --token="${{ secrets.TURBO_SERVER_TOKEN }}" --team="team_name"
    
    
    opened by NorkzYT 11
  • feat: download artifacts from current workflow run, improve API usage

    feat: download artifacts from current workflow run, improve API usage

    I've been playing around with this action, which currently seems to be the only sane option for using Turborepo caching on GH actions. Unfortunately because my workflow is split into separate phases, I ran into a few limitations which this PR is supposed to solve:

    Artifacts from previous jobs of the same workflow are not found This is a limitation of Github Actions - apparently list and delete only works when the workflow was completed. This resulted in caching not working at all in subsequent jobs. I worked around this by downloading all previously uploaded artifacts from the same workflow before starting the server. Then when handling a request, the cache files are checked before falling back to listing other artifacts.

    Reduce API usage In the middle of testing, I actually hit my Github Actions rate limit of 1000/h. I believe this can be avoided using the following tricks:

    • Check file system first, then the API
    • Cache list request for serving GET requests. The turbo server is usually short-lived, so it should be relatively unlikely that we miss an artifact this way. And if we do, it's not the end of the world
    • Re-upload "remote" artifacts as workflow artifacts. I'm not 100% sold on this, but in theory we should be able avoid the list request on subsequent job runs this way, since these are automatically downloaded in followup jobs.

    Oh and I implemented a stub for the statistics endpoint that newer versions of turborepo are using. Otherwise you get a lot of unnecessary errors in the turborepo logs.

    opened by AlCalzone 11
  • refactor(turboServer): use environment variables for configuration

    refactor(turboServer): use environment variables for configuration

    Using environment variables to initiate TurboRepo into local caching mode creates cleaner package.json scripts that are usable in local development and CI.

    Fixes #18

    References

    github.com/vercel/turbo ReadRepoConfigFile source code github.com/vercel/turbo Issue #1324

    opened by johnhooks 6
  • docs: example TurboRepo cli team flag

    docs: example TurboRepo cli team flag

    Is the team flag to TurboRepo irrelevant, just fill TurboRepo cli expectations?

    I don't see any where that the local server created by this action uses it.

    opened by johnhooks 4
  • feat: check if artifact exists locally before attempting download

    feat: check if artifact exists locally before attempting download

    And here's the 3rd part of what we discussed. This PR is based on #11. I can rebase it depending on what we do with #11.

    The main point of this PR is to enable other means of persisting the turbo cache between jobs, and avoiding re-downloads when running tasks that re-check the cache for the same hash as previously.

    opened by AlCalzone 4
  • Cache artifacts not found

    Cache artifacts not found

    I think there is indeed an issue with artifact discovery. Maybe that's why array is undefined.

    I am seeing cache misses, yet I do see the artifact with the same name listed in the API.

    And after a run, there are 2 artifacts now with the same name.

    opened by moltar 4
  • Not getting a cache hit for some reason

    Not getting a cache hit for some reason

    Run felixmosh/turborepo-gh-artifacts@v1
      with:
        repo-token: ***
        server-token: /home/runner/work/_temp/_runner_file_commands/add_path_54db2817-4e5b-4ad7-bc55-29ad818f363b
    TURBO_LOCAL_SERVER_PID: 1904
    

    npx turbo run build --api="http://127.0.0.1:9080/" --token="/home/runner/work/_temp/_runner_file_commands/add_path_213b4c67-ecc2-4ddc-911f-b8dc1f8abafe" --team="moltar" --scope="sub-project-1" --include-dependencies
    

    sub-project-2:build: cache miss, executing 65c0a770b1950c86
    

    Post job cleanup.
    Found server pid: 1904
    Killing server pid: 1904
    There is nothing to upload.
    Server logs:
    Cache dir: /home/runner/work/_temp/turbo-cache
    Local Turbo server is listening at http://127.0.0.1:9080/
    Got a GET request /v8/artifacts/65c0a770b1950c86
    Got a GET request /v8/artifacts/37e71657dbacff1d
    Got a PUT request /v8/artifacts/65c0a770b1950c86
    Got a PUT request /v8/artifacts/37e71657dbacff1d
    

    Then if I re-run the workflow again, I get the same cache cache key 65c0a770b1950c86, and yet still a miss.

    Any ideas?

    Project demo is here: https://github.com/moltar/projen-turborepo-test

    opened by moltar 4
  • Refactor logs

    Refactor logs

    There were a few things I found confusing looking at this project for my other PR, so I thought I'd suggest some very minor refactors. I kept it separate so you can just ignore this one if you like!

    • If the artifact doesn't exist at all, the server still logs out "not found locally, downloading it" which sounded (to me) like it definitely does exist remotely. I slightly reworded it to make it clear that we're only looking for it remotely.
    • I removed the log line about it being successfully downloaded because it looks like that may trigger in certain situations even if the process has failed. I looked at putting it in downloadArtifact() or artifactApi.ts but it seems like Axios will error if it fails, and there is already error logging if the compression fails, so it seemed both misleading and unnecessary.
    • Made the default port a const as that (or a config file) is where I'd expect to find it.
    • Type hinting for downloadArtifact() that I added when putting the above logline in there before I decided against it. Left it in just for helpfulness.
    opened by s-leigh 3
  • License for using and contributing

    License for using and contributing

    In order to be able to use the content here and even contribute to it, we need a license statement which provides usage rights. Due to the copyright laws, there is currently no right to use the content of this repo for anyone else than the copyright owner.

    Since there is a mentioning of MIT in the package.json where it is unclear, what is meant by it, I recommend to add a standard MIT license text with the copyright addition typical for the license. Afterwards, add a section in the README that states that the usage is licensed by MIT and that contributions have to be done under the MIT license as well.

    I saw, that there is one additional contributor to the repository. To play this legal, I recommend to get his agreement on the license.

    opened by blaumeiser-at-bosch 3
  • Error: TypeError: Cannot read property 'map' of undefined

    Error: TypeError: Cannot read property 'map' of undefined

    In the post step, getting the following:

    Post job cleanup.
    Found server pid: 1643
    Killing server pid: 1643
    Error: TypeError: Cannot read property 'map' of undefined
    

    Sorry, doesn't provide any more info than that.


    screenshot-20220208T102136-SQSGOUx6

    opened by moltar 3
  • refactor(turboServer): use of `PORT` env variable

    refactor(turboServer): use of `PORT` env variable

    I'm concerned that using an env variable so generic as PORT could lead to a hard to track down error if someone uses another action that also uses PORT.

    https://github.com/felixmosh/turborepo-gh-artifacts/blob/985e228d924df5bc0cbba735de374e0e5f8e2138/src/turboServer.ts#L11

    opened by johnhooks 3
Releases(v2.1.0)
Owner
Felix Mosheev
Felix Mosheev
GitHub Action that configures Nix to read/write to a cache

Set up Nix Cache Action This is a GitHub Action that configures the Nix package manager to read from (and optionally write to) a remote cache. Usage U

Ross Light 5 Nov 22, 2022
Async node.js implementation of the UDP Minecraft Server Query Protocol and TCP Minecraft Server List Ping Protocol

?? Mc Server Status Async node.js implementation of the UDP Minecraft Server Query Protocol and TCP Minecraft Server List Ping Protocol. Also availabl

Daniel 5 Nov 10, 2022
HTTP server mocking and expectations library for Node.js

Nock HTTP server mocking and expectations library for Node.js Nock can be used to test modules that perform HTTP requests in isolation. For instance,

Nock 11.9k Jan 3, 2023
SPDY server on Node.js

SPDY Server for node.js With this module you can create HTTP2 / SPDY servers in node.js with natural http module interface and fallback to regular htt

SPDY & HTTP2 in JavaScript 2.8k Jan 4, 2023
:dash: Simple yet powerful file-based mock server with recording abilities

?? smoke Simple yet powerful file-based mock server with recording abilities Just drop a bunch of (JSON) files in a folder and you're ready to go! Bas

Yohan Lasorsa 159 Dec 13, 2022
An HTTP Web Server for Chrome (chrome.sockets API)

An HTTP Web Server for Chrome (chrome.sockets API)

Kyle Graehl 1.2k Dec 31, 2022
A server side agnostic way to stream JavaScript.

JS in JSN A server side agnostic way to stream JavaScript, ideal for: inline hydration network free ad-hoc dependencies bootstrap on demand libraries

Andrea Giammarchi 26 Nov 21, 2022
Node.js web server framework for Http/1.1 or Http/2

Node.js web server framework for Http/1.1 or Http/2 Description: This is http framework, you can use it to create Http/1.1 or Http/2 service。 Now let'

Jeremy Yu 10 Mar 24, 2022
A remote nodejs Cache Server, for you to have your perfect MAP Cache Saved and useable remotely. Easy Server and Client Creations, fast, stores the Cache before stopping and restores it again!

remote-map-cache A remote nodejs Cache Server, for you to have your perfect MAP Cache Saved and useable remotely. Easy Server and Client Creations, fa

Tomato6966 8 Oct 31, 2022
sveltekit + turborepo + histoire in a turborepo

swyx's SvelteKit monorepo starter This is my starter for a monorepo with 2022 tech: SvelteKit Turborepo Histoire pnpm - brew install pnpm Demo Proof t

swyx 36 Nov 25, 2022
open-source implementation of the Turborepo custom remote cache server.

This project is an open-source implementation of the Turborepo custom remote cache server. If Vercel's official cache server isn't a viable option, th

Maksim Sinik 362 Dec 30, 2022
Install the latest artifacts automatically and launch your server in a production ready mode with accessible convars.

Setup Edit the first three lines of the index.ts file to set your custom folder layout, follow the folder structure below for a example. const server_

null 2 Feb 4, 2022
A Node.JS tool to automatically install or update your FiveM server artifacts.

ItsANoBrainer FiveM Artifact Updater FiveM Artifact Updater is an application created with Node.JS to easily and quickly install/update your artifacts

null 23 Dec 8, 2022
An action that deletes an artifact associated with given workflow.

GitHub Action to delete workflow artifact(s) An action that deletes an artifact associated with given workflow. Report Bug · Request Feature Descripti

Philips Labs 2 Jan 5, 2022
A JavaScript library providing multiple simultaneous, stable, fault-tolerant and resumable/restartable file uploads via the HTML5 File API.

Flow.js Flow.js is a JavaScript library providing multiple simultaneous, stable and resumable uploads via the HTML5 File API. (Demo) The library is de

HTML5 File upload 2.9k Dec 18, 2022
Plugin to backup & restore your strapi installation (database + uploads) from admin panel.

Strapi Backup & Restore plugin Add backup and restore features directly inside your strapi admin panel. Supported databases: mysql sqlite postgre mong

Hugues BUREAU 41 Dec 19, 2022
Next-multipart - Easy & Simple File Uploads for Next.js

Next-Multipart Next-multipart is a small utility library to ease the process of file uploads with Next.js. It uses formidable under the hood, but with

Tim Raderschad 10 Nov 11, 2022
Automatically uploads a list of patrons to VRChat everytime a patron joins or leaves, without reuploading the world

Automatically uploads a list of patrons to VRChat everytime a patron joins or leaves, without reuploading the world. For use with Miner28/AvatarImageReader

Paul 10 Nov 13, 2022
Script written in JavaScript (Node) that uploads CGM readings from LibreLink Up to Nightscout

Nightscout LibreLink Up Uploader/Sidecar Simple Script written in JavaScript (Node) that uploads CGM readings from LibreLink Up to Nightscout. The upl

Timo Schlüter 87 Jan 7, 2023