I've successfully cloned the turborepo-remote-cache and I've hooked it up in AWS. Everything is working fine, up until I try to use the cache from docker. Here's my dockerfile:

FROM docker.artifactory.moveaws.com/node:16 as build
COPY . /app
WORKDIR /app
RUN yarn install
RUN yarn build
RUN npx turbo run build

Both RUN yarn build and RUN npx turbo run build are not updating the cache. Originally I thought that my docker instance couldn't communicate with the deployed service, but when I added a CURL inside the container, I could verify that outbound calls are properly triggered.

My question is, has anyone run into this? Maybe someone has an example of a dockerfile already out there? Or what could possibly be the issue that the commands won't trigger the cache during the build phase?

Looking at the implementation, it does feel like it can be deployed anywhere, however, the S3 read/writes might be tricky – especially for large files, as Lambdas are essentially short-lived.

Do you folks have any guidance/tips on how to effectively run this on AWS Lambda?

for folks wanting to run this locally between worktrees, not needing docker or needing to clone + compile would be a big help.

just a lil npx turborepo-remote-cache or something would be great :D <3

From docs:

ADC is a strategy used by Cloud Client Libraries and Google API Client Libraries to automatically find credentials based on the application environment, and use those credentials to authenticate to Google Cloud APIs. When you set up ADC and use a client library, your code can run in either a development or production environment without changing how your application authenticates to Google Cloud services and APIs.

ADC searches for credentials in the following locations: GOOGLE_APPLICATION_CREDENTIALS environment variable User credentials set up with the Google Cloud CLI The attached service account, as provided by the metadata server

By simply not providing credentials to Storage() constructor we instruct it to use one of the above three ADC methods. The 2nd and 3rd scenarios are most common in my experience - SDK knows where to find local credentials, and it will also check the GCP metadata server for the Service Account information (this is used for Workload Identity auth in GKE - the recommended way to authenticate with GCP. Very similar to AWS's IAM roles for service accounts/IRSA)

A slightly different approach to an existing PR https://github.com/ducktors/turborepo-remote-cache/pull/66: *Use ADC if no explicit credentials were provided. Many tools that use GCP SDK follow this logic.

Turbo repo introduce support for dry-run to inspect caches and report on whether a cache entry in the local and/or remote cache exists for a given hash. In order to leverage, remote cache implementors must implement a HEAD route to check for the resource. This PR enables this.

See the following for references..

https://github.com/vercel/turborepo/issues/1437 https://github.com/vercel/turborepo/pull/1988

and this article for use cases on leveraging this new info reported by dry-run.

https://medium.com/@sppatel/maximizing-job-parallelization-in-ci-workflows-with-jest-and-turborepo-da86b9be0ee6

Description

This PR adds support for the AWS_SESSION_TOKEN variable, which is used when using shortlived AWS IAM credentials. The logic added will only add this variable if the value is defined, otherwise it will be skipped. This should hopefully not cause any issues for installations not using the sessionToken.

I did not find any tests related to connecting to AWS S3 so I wasn't able to improve test coverage; should I try and investigate how to test this particular feature or can I just leave the PR as is?

Docs

Implemented based on these SDK docs.

I found that artifacts from the end of my build pipeline (just before deploy) are not cached.
Bit of a show-stopper as it causes a full redeploy!

It looks like Fastify has a default body size limit of 1 MiB, and so I'm speculating this is the reason.

Could we have a configuration option to bump this considerably higher?

A backport to the last stable version (so 1.1.2 I guess) would be really helpful, BTW.

Tihs PR adds setting to use Application Default Credentials for authentication of google-cloud-storage.

reference:

• https://cloud.google.com/docs/authentication/application-default-credentials
• https://googleapis.dev/nodejs/storage/latest/Storage.html#Storage-examples

In this PR:

A guide on how to run the turborepo-remote-cache server in an AWS Lambda Function.

Credit to @cpitt for a starting point on the handler code in #28.

This is dependant on #86 - as this change is required to allow the Lambda's temporary credentials to access the S3 bucket.

Issues reference:

• #28

On deploying to Vercel and using S3 storage, I get this error when trying to run turbo commands –

{
	"severity": "WARNING",
	"level": 40,
	"time": 1655122987166,
	"pid": 9,
	"hostname": "169.254.33.177",
	"reqId": "q-nGx8L1TCaa2laKhCeJRA-25",
	"data": {
		"message": "Access Denied",
		"code": "AccessDenied",
		"region": null,
		"time": "2022-06-13T12:23:07.166Z",
		"requestId": "KRFCEX6FM17HZD2Z",
		"extendedRequestId": "xYHj4oepC5Eu4oD0QcrQcTGCbOyITRXV7rp2JSXa1RwAVSOKsiI0b2gkABewiDxQXa7GN+8RUGM=",
		"statusCode": 403,
		"retryable": false,
		"retryDelay": 26.181109834484474
	},
	"isBoom": true,
	"isServer": false,
	"output": {
		"statusCode": 412,
		"payload": {
			"statusCode": 412,
			"error": "Precondition Failed",
			"message": "Error during the artifact creation"
		},
		"headers": {}
	},
	"stack": "Error: Error during the artifact creation\n    at Object.handler (/vercel/path0/src/plugins/remote-cache/routes/put-artifact.ts:29:31)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
	"type": "Error",
	"message": "Error during the artifact creation"
}

I've made sure the environment variables are correctly set up, the IAM credentials are right, and that the user has full adminstrator access on AWS.

When running locally I can't get my repo to actually cache the results.

Setup in turborepo-remote-cache:

yarn build
NODE_ENV=development TURBO_TOKEN=\"123\" yarn start

setup in test-turbo-reop

.turbo/config.json

{
    "teamId": "team_FcALQN9XEVbeJ1NjTQoS9Wup",
    "apiUrl": "http://localhost:3000"
}

package.json

{
    ...
    "scripts":{
            "build:turbo": "turbo run build --token=\"123\"",
            ...
    }
}

Log output from turborepo-remote-cache

{"severity":"INFO","level":30,"time":1654715502319,"pid":83236,"hostname":"MacBook-Pro-2.local","message":"Server listening at http://0.0.0.0:3000"}
^[[A{"severity":"INFO","level":30,"time":1654715546148,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-0","req":{"method":"POST","url":"/v8/artifacts/events?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup","hostname":"localhost:3000","remoteAddress":"127.0.0.1","remotePort":59506},"message":"incoming request"}
{"severity":"INFO","level":30,"time":1654715546151,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-0","message":"Route POST:/v8/artifacts/events?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup not found"}
{"severity":"INFO","level":30,"time":1654715546155,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-0","res":{"statusCode":404},"responseTime":5.764100074768066,"message":"request completed"}
{"severity":"INFO","level":30,"time":1654715546156,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-1","req":{"method":"POST","url":"/v8/artifacts/events?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup","hostname":"localhost:3000","remoteAddress":"127.0.0.1","remotePort":59508},"message":"incoming request"}
{"severity":"INFO","level":30,"time":1654715546156,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-1","message":"Route POST:/v8/artifacts/events?teamId=team_FcALQN9XEVbeJ1NjTQoS9Wup not found"}
{"severity":"INFO","level":30,"time":1654715546156,"pid":83236,"hostname":"MacBook-Pro-2.local","reqId":"vmy6wThJSLCcGWwTxwgjpg-1","res":{"statusCode":404},"responseTime":0.36331605911254883,"message":"request completed"}

Is there something that I am missing with having to register a team so that it doesn't 404 with the server or a configuration step that I missed in the docs?

Hope anyone that reads this is having a great day!

I only have the following line in the Docker Container logs, and no other log appears when I use the remote cache.

54896/11/01 01:17PM 30 Server listening at http://0.0.0.0:3000 | severity=INFO pid=6 hostname=turborepo-remote-cache

How do I know when I run the npm script build command, the action is importing the cache, and/or exporting it?

Is the STORAGE_PATH environmental variable correctly used along with volumes?

The following contains the contents of the TurboRepo Remote Cache docker-compose.yml file.

version: '3.9'
services:

  turborepo-remote-cache:
    image: fox1t/turborepo-remote-cache:1.8.0
    container_name: turborepo-remote-cache
    hostname: turborepo-remote-cache
    environment:
      - NODE_ENV=production
      - TURBO_TOKEN='xxx'
      - LOG_LEVEL=debug
      - STORAGE_PROVIDER=local
      - STORAGE_PATH=/tmp
    volumes:
      - /mnt/appdata/repo-team/turborepo-remote-cache/tmp:/tmp
    ports:
      - 3535:3000
    networks:
      - proxy

networks:
  proxy:
    driver: bridge
    external: true

Originally posted by @NorkzYT in https://github.com/ducktors/turborepo-remote-cache/discussions/84

🚀 Feature Proposal

Create a (or modify the existing) health check endpoint that is not protected with the authorization header.

Motivation

Without this, it's not possible to scale the service behind an AWS ALB with a health check as ALB doesn't support passing headers.

Example

AWS ALB health checks