Using data pulled from S3 buckets containing terraform states, this will pull all data from the account and compare against current state.

Additional Features:

1. add and remove exceptions of resources.

Example: A terraform state file is in S3 bucket and added to the list of TFSTATES. The state was abandoned and does no longer represent what was deployed in the account.

I want to check a TFSTATE file in S3 and confirm its resources were actually deployed and are RUNNING or has discrepancies. There are ways to do this through Terraform, but can also be codified.

The best solution (though it costs 150mb) is to add the whole .terraform folder to api.
Another elegant solution is to add a file "dummy_aws.tf" and invoke "terraform init" every time you want to initialize the api.

The point of this issue is to make a painless docker image.

Must add endpoint /login and endpoint /refresh. That should return the jwt token:

{
    "access": ...,
    "refresh": ...
}

The only endpoint that doesn't require the Authorization: header. All the other endpoints should have it built-in.

Login details may be passed as parameters or environment variables for now.

--login-password {password} --login-username {username}

Of course, must implement login page in the frontend with POST to that /login. And must append the authorization header in axios.

• Implement POST endpoint for the jwt token in the backend
• Use cmd flags instead of hardcoded to login with jwt
• Require the authorization header in all the other requests.
• Implement the login page in the frontend
• Implement the authorization header in axios

The program should fetch features on startup, write them to disk on /features, and add/remove from db on /remove and /add.

1. setup a dynamodb instance.
2. connect to it using go
3. save/restore features from it. maybe add some _test.go?

• AWS Credentials should not be hardcoded.
• VPC Flow Logs should be enabled
• Remove all rules associated with default route tables, ACLs, SGs, in the default VPC in all regions
• Resources should use encryption when they are created
• Resources should use encryption in transit
• Resources should follow basic naming standards
• S3 buckets should not be public unless approved by Security Engineering
• S3 buckets should have logging enabled
• Security groups should not be overly open. Exposure of insecure protocols should be limited for [ ] ingress traffic. Only selected ports and approved by Security Engineering should be open
• CloudTrail must be enabled in all Regions
• GuardDuty must be enabled in all VPCs, in all regions

Also log entries should have an "account".

Question: how to do ui?

opt1: url /tfstates/. Selects the account to show elements. Same for logs.

Some kind of abstraction to reuse this in /logs. FilterableTable, with some control above the table.

So it should act exactly like a table, except it should only show those which "column" account is of
a particular way?

Maybe:

<FilteredTable filter={filter} onSetFilter={filter => onSetFilter(filter)} objs={objs} 
columnsRender={() => {

}
rowRender={obj => 

} />

checklist

Implement the necessary feature in both server and client.

• Add to repo the frontend
• Implement /tfstates
• Implement /features
• implement /logs alias /. For each Kind, a different table.
• implement /unregisteredresources
• do details checklist, one for each kind.
• implement feature edition
• implement tfstate details
• implement log details.
• implement + for tfstates
• implement + for features
• implement delete for tfstates
• implement delete for features
• implement delete for logs

Reasons

Why?

• Make auth easier
• Dependencyless access. Just need the URL, no cli
• Better view of differences. The program handles large strings, both for input json and compliance strings.
• Easier to show to somebody else
• Easier feature edition
• Doesn't take too much effort. its get/post/delete, get/post/delete.
• More motivation, visual stuff is always refreshing

Why not?

• Added complexity
• Less time to invest in
• CLI is more "programmer-friendly", allows you to do things with pipes, etc.

For now, gonna have both. The CLI itself is really simple. However, output must be formatted for cli server-side. With web, backend should return stuf in json

How to implement

frontend

Using react, making the same endpoints with react router, /features, /features/id, etc. with tables or cards on each one.

backend

jsonTopLevel() interface{} and jsonDetails interface{} in restObject. And /endpoint/json/{...}.

This sounds good.
like:

func (feature *ComplianceFeature) jsonTopLevel(map map[string]interface{}) {
    result["name"] = feature.Name
}

The problem with this is that you won't be able to return arbitrary objects. But this doesn't seem neccessary.

Because loads all the foreign resources, but doesn't load states, which are used to determine if the resource is foreign or not.
Solution is to add two functions: losdAllTFStatesMinimal and loadAllTFStatesFull.
Also use the full version on state monitoring as well, to avoid a bunch of get requests.

The minimal/full postfix makes the meaning cleaner without needing documentation. Should be applied to all db.getAll* functions.

The point of labels is solving the following issue:
Sometimes I want specific states to run against specific features.

So, how do I do this?
Well, chances are:

1, Reference in every state which features you want.
2. Reference in every feature which states are affected.

Both suck. 1 sucks because it'll be very painful. For every state, I add which features I want. Imagine 20 states, and specifying which features I want for them. Also, when I add a new feature, I need to edit all states to include the new feature.
2 sucks because for every state I add, I will need to update the feature to include it.
Also, It'll be a pain in the ass to implement in Dynamo.

So, what's the solution?
The solution is tags. Tags are the bridge between states and features.
Every feature specify tags. And every state specify tags.
So, they won't be directly coupled, but will be connected through tags.

• Every state is affected by features that contains any of their tags.
• Every feature affects all states that contains any of their tags.

So, to implement:

• Ask which features in runComplianceTool. Use a tmp directory
• Persist a list of tags in tfstates and features.
• Function runComplianceToolWithTags(input, db, []tags). To make life easier
• Implement in backend/frontend ability to set tags in tfstates.
• Implement in backend/frontend ability to set tags in features.

Store documents as:

tfstate {
    stateJSON
    complianceResult: ComplianceResult
}

log {
    stateJSON
    oldStateJSON
    complianceResult complianceResult
    oldComplianceResult complianceResult

This will take a lot of complexity away.

Allows to:

• generalize, reuse parsing logic in frontend. because the compliance struct is a well defined and fields are not embedded in the top level object json using weird name conventions like "_old" suffixes.
• allows to save a lot of bandwith, the server doesn't need to fetch the compliance output (nor the json state when fetching the list, which is the biggest bottleneck).
• takes parsing complexity away in the rest handlers

Backend

• make sure dynamo driver supports nested structs. otherwise none of the above apply!
• In complianceResult, add a flag that indicates the state of the result. initialized, error, errorMessage.
• Move error parsing to the compliance result.
• make checkTFstate return complianceResults directly, instead of outputs. Outputs are only handled by the function that executes the tool, never passed above.
• Remove the complianceOutput strings in logs, add the necessary field for complianceResult
• Cleanup the handler for compliance results.
• Remove complianceOutput and prevComplianceOutput in logs, implement the structs as well
• Only fetch the necessary fields for logs and tfstates in GET.

Frontend

• Generalized func to show compliance label in Compliance.js, and a bool tooltip true/false if you want tooltips. Also may handle errors and stuff. Another one ComplianceChange(old,new).

How to get the current tfState?

Maybe add a -tf-state-s3-bucket and -tf-state-s3-key.

1. Fetch the file from this bucket.
2. Save it in the current dir as a .tfstate
3. Do terraform show -json > wanted.json
4. Read wanted.json. Now you have a way to read the current tfstate.

´/validate´ endpoint with hardcoded features.
plan file must be sent as a raw base64 string in the body.

cmd ./client <plan file> <ip>

Very simple: on get/{id} don't fetch the whole object set. Just the one that was GETted

A validation consists of:

type ValidationLog struct {
     Time time.Time
     InputJson string
     Output string
     WasSucessful bool
     ErrorCount int
     SkippedCount int
     PassedCount int
}

Ok. So... Problems to solve:

• Parse tool output into a ValidationRecord
• Find a way to save in dynamo log entries also.
• Implement endpoints /logs and /logs/{id}.

It's necessary to check what went wrong without having to check the app logs

Add server_test.go to check whether or not all endpoints behave as expected

Changes required:

• Add boolean validating, and show some loading signs? The PATCH /tfstate/{id}/forcecheck?
  That returns some error? like a long-running endpoint? Another one non-blocking, a flag forceCheck to check in the next iteration. That can be read by the frontend to show a spinner, or something like that.
  Also, some error reporting if the check fails, Report the aws error as "error" or whathever.
  Necessary.

How to handle validation errors:

1. The program exited with non-0 error code? There was some error during execution.

• New flag to tfstate: forcedValidation. Must execute this until the validation ends.
• if some error occurs during the validation, save in the output as "Error:" and parse it accordingly.
• Return the error output in the API properly
• Handle error cases in the frontend as well.

For example, should return 404 on reading/deleting a feature that doesn't exists.

Using data pulled from S3 and stored in DynamoDB program will check terraform compliance against TFstate and log results if non-compliant states are found.

This features facilitates the addition of TFSTATES

The application's embeds untrusted data in the generated output with WriteHeader, at line 42... This untrusted data is embedded straight into the output without proper sanitization or encoding, enabling an attacker to inject malicious code in the output.

The attacker would be able to alter the returned web page by saving malicious data in data-store ahead of time. The attacker's modified data is then read from the database by the registerEndpoint method with ReadAll, of this code. This untrusted data then flows through code straight to the output web page without sanitization.

Takes some compexity away, standarizes a way to get results sorted, and results per date as well.
That's because dynamo doesn't keep any order in the records, making this necessary.

• add to restObject TimestampLong() and timestampTime().
• always retrieve id and timestamp on additions
• in get, sort by timestamp.
• remove legacy custom-resource timestamps.