bojanbass / nx-aws Goto Github PK
View Code? Open in Web Editor NEWNX monorepo with AWS S3 caching
NX monorepo with AWS S3 caching
Right now, the only way of skipping the remote cache and using the local one is via the NXCACHE_AWS_DISABLE environment variable.
Ideally, we would like to disable the remote cache by default for local development, but have it enabled for CI. Since we need to set an environment variable to disable the cache, that'd mean we need .env
support or using something like crossenv
to set the environment var in a cross-os compatible way.
If we add this as an option, we could just do nx build --skip-aws-cache=false
on CI and have the default value be true in our NX config.
Happy to make a PR if you think this would be acceptable.
Hey there! Thanks for the plugin, seems like exactly what I was looking for.
I've noticed that in my CI runs, i am definitely producing cache files, I can see them showing up in my bucket. But on subsequent reruns I can see output from the builds, and I'm wondering if I'm actually hitting the cache? I've also tested turning the caching off, and it seems about the same speed.
Do you have any advice for debugging and determining that I am in fact using the cache?
I'm on NX v17
Using Github Actions
Thanks!
Since updating to version 3.2.0
we have the following error:
Error: endpointV2Middleware is not found when adding regionRedirectEndpointMiddleware middleware before endpointV2Middleware
at /node_modules/@smithy/middleware-stack/dist-cjs/index.js:154:17
at Array.forEach (<anonymous>)
at getMiddlewareList (/node_modules/@smithy/middleware-stack/dist-cjs/index.js:143:31)
at Object.resolve (/node_modules/@smithy/middleware-stack/dist-cjs/index.js:297:32)
at PutObjectCommand.resolveMiddleware (/node_modules/@aws-sdk/client-s3/dist-cjs/commands/PutObjectCommand.js:39:22)
at S3Client.send (/node_modules/@smithy/smithy-client/dist-cjs/index.js:110:29)
at Upload.__uploadUsingPut (/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:66:57)
at Upload.__doConcurrentUpload (/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:128:39)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.all (index 0)
> NX Storage Cache: Upload error - Error: endpointV2Middleware is not found when adding regionRedirectEndpointMiddleware middleware before endpointV2Middleware
The config used is:
NXCACHE_AWS_ACCESS_KEY_ID=key
NXCACHE_AWS_SECRET_ACCESS_KEY=key
NXCACHE_AWS_BUCKET=bucket
NXCACHE_AWS_REGION=eu-west-1
When rolling back to 3.1.0, everything works.
Hi, I'm trying out this package and ran into this issue while running init:
NX (0 , devkit_1.readRootPackageJson) is not a function
From what I can tell, that function doesn't exist in the @nx/devkit package. Could be a version mismatch? Not sure, but mine is v16.9.1
I was able to fix/bypass it by changing it to use readJsonFile('package.json')
instead
I use a self hosted aws, to use that I need the possibility to set the endpoint, otherwise s3.eu-de.amazonaws.com is always used
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/s3clientresolvedconfig.html#endpoint
The breaking change from nx 16.0.0-beta.1
.
When migrations are run @nrwl/devkit
is replaced by @nx/devkit
> git -c user.useConfigOnly=true commit --quiet --allow-empty-message --file -
> NX Affected criteria defaulted to --base=main --head=HEAD
> NX Cannot find module '@nrwl/devkit'
Require stack:
- /../../node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
- /../../node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
- /../../node_modules/nx/src/tasks-runner/run-command.js
- /../../node_modules/nx/src/command-line/affected.js
- /../../node_modules/nx/src/command-line/nx-commands.js
- /../../node_modules/nx/bin/init-local.js
- /../../node_modules/nx/bin/nx.js
Pass --verbose to see the stacktrace.
husky - pre-commit hook exited with code 1 (error)
Caused by 16069
The only S3 actions that are explicitly used are GetObject, PutObject
, PutObjectACL
, and GetObjectAttributes
, However, the code is written in a way that implicitly requires ListBucket
nx-aws/packages/nx-aws-cache/src/tasks-runner/aws-cache.ts
Lines 226 to 231 in ddcc209
private async checkIfCacheExists(hash: string): Promise<boolean> {
const tgzFileName = this.getTgzFileName(hash),
params: clientS3.HeadObjectCommand = new clientS3.HeadObjectCommand({
Bucket: this.bucket,
Key: this.getS3Key(tgzFileName),
});
try {
await this.s3.send(params);
return true;
} catch (err) {
if ((err as Error).name === 'NotFound') {
return false;
} else if (err instanceof CredentialsProviderError) {
return false;
}
throw new Error(`Error checking cache file existence - ${err}`);
}
}
if the HeadObjectCommand
is used, and the principle has ListBucket permissions, then NotFound
will be returned. However if they do not have ListBucket permissions, then 403 will be returned. So caching will fail
See API reasoning https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/s3/command/HeadObjectCommand/
Permissions
You need the relevant read object (or version) permission for this operation. For more information, see [Actions, resources, and condition keys for Amazon S3 ](https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html)
. If the object you request doesn't exist, the error that Amazon S3 returns depends on whether you also have the s3:ListBucket permission.
If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 error.
If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 error.
Solution. Check if a 403 was returned, and return false in that case
Would it be possible to extend this solution to support an S3-compatible API like Backblaze B2 or Wasabi?
It seems the only thing missing is the ability to provide an endpoint variable in the configuration and pass it on when creating the S3 object. For backblaze this is explained here.
If you want I can provide a PR.
I saw that in the package.json, there are these peerDependenies
entries:
"peerDependencies": {
"@angular-devkit/core": ">=11.0.1",
"@angular-devkit/schematics": ">=11.0.1",
"@nrwl/workspace": ">=9.2.2"
},
Since I don't use angular, and since nx14 does not need @nrwl/workspace
, can all of these peers be optional?
Related:
Following the instructions in the README, I ran in to this error:
> NX Cannot find module '@nrwl/workspace/tasks-runners/default'
Require stack:
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
- <repo>/node_modules/nx/src/tasks-runner/run-command.js
- <repo>/node_modules/nx/src/command-line/run-many.js
- <repo>/node_modules/nx/src/command-line/nx-commands.js
- <repo>/node_modules/nx/bin/init-local.js
- <repo>/node_modules/nx/bin/nx.js
I'm using nx 14 with no plugins, so the default task runner (in nx.json) is: "runner": "nx/tasks-runners/default",
Should that new nx runner path should be used instead?
Currently, when the task runner is configured to use the @nx-aws-plugin/nx-aws-cache
it always prints USING REMOTE CACHE
in the terminal.
https://github.com/bojanbass/nx-aws/blob/master/packages/nx-aws-cache/src/tasks-runner/runner.ts#L43
However, when the --skip-nx-cache
Nx ignores all the caches, local and remote.
When the --skip-nx-cache
CLI option is specified the USING REMOTE CACHE
should not be printed in the terminal.
Use the options.skipNxCache
to condition the USING REMOTE CACHE
log.
So I tried using the plugin, but no matter which version I use (latest, 16.2.0 or 17.0.0) I always get errors, that there is no compatibility or that nx couldn't be found.
That's my package.json
"dependencies": {
"@nx-aws-plugin/nx-aws-cache": "^3.1.0",
"@nx/devkit": "16.2.1",
"nx": "16.2.1"
},
Running yarn nx generate @nx-aws-plugin/nx-aws-cache:init --verbose
gives the following output:
✔ AWS Access Key Id ·
✔ AWS Secret Access Key ·
✔ AWS configuration profile to use ·
✔ AWS Endpoint of the web service if a custom endpoint is needed (e.g. when using MinIO) ·
✔ AWS region to which this client will send requests · us-east-1
✔ AWS S3 Bucket, where the cached Nx outputs will be stored · nx-rempte-cache.myblubilub.com
✔ Whether to force path style URLs for S3 objects (e.g. when using MinIO) (y/N) · false
Error: You must install Nx to enable Storage Cache
at isCompatibleVersion (/Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:10:15)
at /Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:42:14
at Generator.next (<anonymous>)
at /Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:167:75
at new Promise (<anonymous>)
at Object.__awaiter (/Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:163:16)
at default_1 (/Users/blabliblub/projects/myproject/node_modules/@nx-aws-plugin/nx-aws-cache/src/generators/init/generator.js:41:20)
at Object.<anonymous> (/Users/blabliblub/projects/myproject/node_modules/nx/src/command-line/generate/generate.js:250:36)
at Generator.next (<anonymous>)
at fulfilled (/Users/blabliblub/projects/myproject/node_modules/tslib/tslib.js:164:62)
Any ideas or which versions are compatible with this?
A recent bug in my pipeline was causing some files to be missing from the cache. After investigating I narrowed it down to a missing folder called source
. Renaming that folder made the cache work as expected.
Investigating further I found the root cause, which was the introduction of this filter:
I am reproducing the issue in this repo, but I may take a while to set the AWS part to make it a full reproduction. Anyway, I hope it helps to illustrate the issue.
The short-term solution was to rename the folder called source
to something else, but I wonder how could we better identify if the filtered source
is the actual file produced by NX.
I need to set NXCACHE_AWS_DISABLE=true
in a .env file and I see the codebase supports that. Every developer on the team is going to want to set this flag for local development, so we want to commit this to the repo. We then flip this using an env var in github actions for CI.
The .env file is also read by another tool for username/login tokens which I would like to not commit.
I currently cannot change where either of these .env files are read from, would it be possible to add API surface to allow passing a different path to a .env file in?
Alternatively, would it be possible to change/flip the nxcache_aws_disabled=true without a .env file?
I'd like to try out this package, but I'm getting the following output:
$ nx generate @nx-aws-plugin/nx-aws-cache:init
> NX Generating @nx-aws-plugin/nx-aws-cache:init
✔ AWS Region (optional - can be defined using Environment variables) · us-east-1
✔ AWS Bucket (optional - can be defined using Environment variables) · nx-distributed-cache
> NX You must use Nx >= 8.0 to enable Storage Cache
Pass --verbose to see the stacktrace.
$ nx --version
15.7.2
am I missing something to be able to make this work?
Using Latest NX version and RxJS causes the following issues:
$runner.subscribe is not a function
The issues is isolated to the latest version, and using the older 0.0.5 version with the same NX and RxJS still works.
I think it's because previous version wraps $runner in a from($runner), which creates an observable that can be subscribed.
I'm running code on different nodes on CI pipeline execution.
During execution, an attempt of loading the remote cache ends up with the error, as nx does not allow the use of cache from different machines.
Extend config with trustForeignMachineCache?: boolean
property, which will allow to reuse cache generated on machines different than current (by removing source
file from downloaded tar
).
There is a known workaround by setting REJECT_UNKNOWN_LOCAL_CACHE=0
env variable, but this approach is discouraged. It should be the responsibility of RemoteCache
provider (like nx-aws
) instead of nx
Please share your POV about this idea. If you have no red flags, let me know and I will prepare a PR for that.
> NX Cannot find module '@angular-devkit/schematics'
Require stack:
- <repo>/node_modules/@nrwl/workspace/src/utils/ast-utils.js
- <repo>/node_modules/@nrwl/workspace/src/utils/project-type.js
- <repo>/node_modules/@nrwl/workspace/index.js
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/logger.js
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/aws-cache.js
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/tasks-runner/runner.js
- <repo>/node_modules/@nx-aws-plugin/nx-aws-cache/src/index.js
- <repo>/node_modules/nx/src/tasks-runner/run-command.js
- <repo>/node_modules/nx/src/command-line/run-many.js
- <repo>/node_modules/nx/src/command-line/nx-commands.js
- <repo>/node_modules/nx/bin/init-local.js
- <repo>/node_modules/nx/bin/nx.js
I don't want to install @angular-devkit/*
, because I'm not using angular 😅
I'm using nx@14
, and ran in to this error after installing @nrwl/workspace
after running into this error: #126
does this library support IAM roles? seems to always fail for me and falls back to using the default task runner.
the issue i see is checking the config in the AwsCache class
new S3().config.credentials
always returns null, hence checkConfig
throws an error
i've tested IAM role support bypassing this check and everything seems to work as expected
Thanks for this plugin. Trying to set it up and got this question 😄
First of all, thank you very much for the plugin.
Problem:
Unfortunately, I'm developing on a Windows computer and have a problem where "/" is normalized to "" in this function https://github.com/bojanbass/nx-aws/blob/ddcc209d6a6f7612e0de30485edb865ed8cdbdb0/packages/nx-aws- cache/src/tasks-runner/aws-cache.ts#L247 .
Possible Solution:
The easiest way is to insert a replace("\","/") after the "join". This means that Windows can work properly and the files do not end up on the S3 storage from a Windows computer with “xyz\123.tar.gz”.
Hi,
thank you for this great project!
Locally, it works perfectly for me, but I have been struggling to run it on AWS CodeBuild. It did not pickup the credentials correctly. The current version 1.1.0 uses fromIni which is not working. Your current master already uses fromNodeProviderChain which works in AWS CodeBuild.
Could you please publish a new version to npm that uses fromNodeProviderChain?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.