khalidx / cfs

An easy way to discover and manage your cloud like a local filesystem.

Hi 👋, I'm Khalid

Let's connect!

Languages and Tools

Status Update

I'm currently working on cloudfs, an easy way to discover and manage your cloud like a local filesystem!

I'm also working on tree, a new language for computing, and the simplest language on the planet! It's in private beta.

Can't figure out how to use ESM + TypeScript + ts-node + Node.JS? Feel like you've been running in circles with tutorials and blogs claiming to have the right config? Look no further. If you want a working config, check out my gist with 490+ stars ⭐️. You'll be up and running in no time!

Repositories

Pinned repositories below are authored by me. Check out my Github Stars for notable projects from around the internet!

cfs's Issues

Recognize "Token is expired" error for AWS SSO

Add code to recognize the following error, which is currently reported and counted as an Unknown error by cfs.

{
  "type": "Unknown",
  "error": {
    "message": "Token is expired. To refresh this SSO session run 'aws sso login' with the corresponding profile.",
    ...
  }
}

Document the minimum role needed to perform all operations

Add a minimum-role.json file that contains an AWS IAM role with the minimum permissions needed to perform all cfs discovery operations for listing all supported resources in an AWS account.

Also, consider adding a command that checks to ensure the current logged in user has permissions that meet the minimum set.

A command like one of the following:

cfs verify
cfs validate
cfs check role
cfs check permissions
cfs status

Add granular context to the errors command

Building on the work done in #21.

Improve the cfs errors command and add detailed error explanations. Something like:

The following errors were encountered during the last execution:

- insufficient permissions for:
  - AWS S3 Buckets
  - AWS EC2 Instances
- other errors, logged here: .cfs/errors.log

We may also need to think about how to report errors that only happened in a specific region, like errors related to resources in a region that may be disabled or unused due to organization IAM policies.

Introduce support for plugins

Introduce the plugins concept, supporting plugins written in any language.

Issue #54 will advance the plugins feature.

Investigate using official actions for drafting a release

Rather than using the GitHub recommended, although third-party action for drafting a release and uploading files to it in .github/workflows/build.yml, use the default (although archived) actions provided in the actions repository for GitHub Actions:

• https://github.com/actions/create-release
• https://github.com/actions/upload-release-asset

Alternatively, rather than relying on archived actions projects, it seems that a solution can be put together manually based on the options mentioned here:

• actions/upload-release-asset#28 (comment)
• actions/upload-release-asset#47 (comment)

For reference on configuring the third-party action mentioned at the start of this issue, see the block below (which on initial test did not work - the step was skipped in the build even though a tag was pushed):

    - name: Draft Release
      # See more information about this step at https://github.com/softprops/action-gh-release
      uses: softprops/action-gh-release@v1
      if: startsWith(github.ref, 'refs/tags/')
      with:
        draft: true
        prerelease: true
        generate_release_notes: true
        fail_on_unmatched_files: true
        files: |
          ./exec/cfs-linux
          ./exec/cfs-macos
          ./exec/cfs-win.exe

Add a flag to allow listing of in-region resources only

When discovering resources with cfs --region <region-name>, global resources are also included in addition to resource that exist in that region. Add support for a new --ignore-global flag that doesn't include these global resources in the output. Alternatively, add a --region-only flag, which is like --region, but with the additional aforementioned logic.

Add a button to open a resource in the AWS Console

For the browser experience served by the cfs browse command, each resource item in the search results list should have a button or link that opens the resource in the AWS Console, to make resource discovery and navigation easier.

On Windows, the page shifts when the scrollbar appears

Prevent shifting of the page as the scrollbar appears when list items render as part of a search. Also, in JSON code blocks, the scrollbar is present (although disabled) even if there is nothing to scroll.

Invalid CLI args and flags should not be accepted

Currently, the CLI arg parsing is very lenient and won't fail if you pass an unknown flag, like --some-flag or use an unknown or extra arg. Change this behavior to fail and output a message if the provided args don't exactly match the CLI command definition.

Errors should mention the associated resource type, region, and operation

An error written to the errors.log array should include the name of the resource and the operation that was currently taking place (and the region of the operation).

Errors should also possibly be grouped by this in errors.log.

The browse command should show icons for all resources

When using the cfs browse command, a user should be able to view the icon of the shown resource beside the resource JSON definition, to make the resource search and discovery feature better.

All paths should be escaped to disallow filesystem manipulation

The names of paths should not come straight from external systems, like AWS ARNs or AWS SSM Parameter Store parameter names without being escaped. For example, one should not be able to write a resource file that is named .cfs/alarms/alarm123/../456 as that would result in writing a file in a directory other than the one intended. Escape these paths before using them to build resource names and to write to the filesystem.

Incorrect type found in AWS SDK v3 client for CloudFront

In @aws-sdk/client-cloudfront, the type of the field HttpVersion for a CloudFront Distribution is defined as the following union:

http1.1 | http2

In reality, the AWS API returns the following data:

HTTP1.1 | HTTP2

This issue is for tracking the submission of this minor fix to the appropriate repository for the AWS SDK.

No bug exists in the cfs code in this repository, since we use our own manually-defined zod union type for validating this data. This is actually how this bug was caught, since we define our own types and rely on parsing data with validation rather than just relying on the TypeScript declarations provided by the AWS SDK.

No paginator exists in AWS SDK v3 client for API Gateway v2

In @aws-sdk/client-api-gateway, a paginateGetRestApis exists. But, in @aws-sdk/client-apigatewayv2, no such paginator exists for getApis.

For now, writing an iterator manually. This issue will stay open until the paginator has been added to the SDK, and the cfs code is updated to use the paginator.

Add node v18 to the GitHub Actions build matrix

As soon as pkg supports building native binaries with node v18, add 18.x to the node-version build matrix in .github/workflows/build.yml.

Add a verbose mode flag to output debug logs

Do two things:

1. Output all logs by default to debug.log, like we do for errors in errors.log.
2. Add a --verbose flag that outputs debug logs to the console during execution.

Investigate the differences between DescribeStacks and ListStacks

Currently, we are using DescribeStacks for listing the stacks in an AWS account. Check how the data is different versus using the ListStacks operation.

Ensure all resources have correct Zod schemas

Not all resources have been tested, and some may fail due to a Zod schema that is too strict (actually, we've now made all object properties deepPartial(), removing the strictness).

Also, the following resources are not using a strict schema (to save time, the TypeScript types were huge), and are using .passthrough() instead, and will need to be updated to use a strict schema:

• AWS EC2 Instances
• AWS ELB Classic Load Balancers
• AWS ELB Application, Gateway, and Network Load Balancers
• AWS CodePipeline Pipelines
• AWS IAM Roles
• AWS IAM Users
• AWS IAM Policies
• AWS RDS Database Clusters
• AWS Secrets Manager Secrets
• AWS CloudWatch Logs Log Groups

Also:

• Ensure LoadBalancerName is unique for AWS ELB Classic Load Balancers and AWS ELB Application, Gateway, and Network Load Balancers
• Ensure name is unique for AWS CodePipeline Pipelines
• Ensure DatabaseName is unique for AWS RDS Database Clusters
• Ensure Name is unique for AWS Secrets Manager Secrets, and whether it is allowed to be path-style (assuming it is path style in the current code)
• Ensure logGroupName is unique for AWS CloudWatch Logs Log Groups

The CLI may not need a built-in server for the browse command

Since all assets are HTML/CSS/JS/SVGs, and the only thing we are loading from the web is a font, and since resources can be embedded as JSON or HTML or compiled into a single-file HTML page (that can be opened in a browser, emailed, etc) there may be no need to have a full-blown express server shipped with the CLI.

Think about this alternative.

Add an errors command to log error message categories

Add a new cfs errors command to log the categories of encountered error messages, like:

0 NoInternetAccess errors
0 AuthenticationMissing errors
0 AuthenticationExpired errors
3 InsufficientPermissions errors
0 SchemaValidationFailed errors
2 Unknown errors

Fix resource paths for alarms, topics, and queues

The alarms, topics, and queues resources should have better resource paths through proper escaping. Currently, the paths don't look good due to too much information other than the resource name being left in the path. Clean up the paths by:

• alarms: remove arn:aws:cloudwatch:<region>:<account#>:alarm:<name>, and leave <name>
• topics: remove arn:aws:sns:<region>:<account#>:<name>, and leave <name>
• queues: remove https://sqs.<region>.amazonaws.com/<account#>/<name>, and leave <name>

Also, paths are currently showing up as .cfs/topics/us-east-1/s%3Asns%3Aus-east-1... for topics, for example. Something is wrong with the substring used to extract the name. The arn:aws: remnant is still here, leaving just the s.

It turns out, the issue with the substring was indexOf returning -1 when the pattern being searched for does not exist in the string. Add validation to the zod schemas to ensure the pattern being searched for exists in the string before calling indexOf.

Add icons for newly added resources

Add icons for all recently added resources so that they show up properly when using the cfs browse command. This builds on the icons added in #7.

Add support for resource links

Add a button or a that when clicked, opens the specified resource's console page in the AWS console.

Something like:

<a href="${params.link}">
  <img width="24px" src="/img/external-link" alt="External link to the console for ${params.type}">
</a>

Also, see if there is a way to specifically open the resource in the correct account, using some parameter in the console URL, like: https://0123456789.signin.aws.amazon.com/console/s3/?bucket=some-fake-bucket-123.

Remove process hard exit

It is probably a bad idea to hard exit the process with process.exit(1). Some operations (like writing to the console) may not have completed yet.

See this Stack Overflow thread for more information.

Add typeahead search with autocomplete

The search bar in the browser experience (the cfs browse command) should support a typeahead experience that autocompletes resource names and/or paths.

Define additional example plugins and behavior

Define the application lifecycles that plugins can participate in.

Add some usable example plugins. Plugin ideas:

• a plugin to log and warn when any buckets are public
• a plugin that marks resources for cleanup and deletion
• a plugin that looks at certificate expiration and logs when certs are about to expire

Native binary fails to open the browser on Linux

The open npm package bundles xdg-open with its source. Problem is, when we bundle a native binary with pkg, this file does not get included and is not runnable. Therefore, the cfs browse command cannot open URLs when running from the cfs linux binary and xdg-open is missing on the host system.

This seems to be a long running bug/inconvenience in open/pkg/linux. Figure out how to solve.

Currently, it fails with an error that apparently cannot be caught:

code: 'ENOENT',
syscall: 'spawn xdg-open',
path: 'xdg-open'