ukhomeoffice / data-transfer Goto Github PK

When running multiple instances of the micro services managing session variables could be cumbersome. To facilitate easier deployment and management we would like to add a configuration file for PM2 that could be used to provide env vars when running the service under PM2.

When the data-transfer app is running in multiple locations, i.e in a chain. There could be issues with files being read before the write is complete. This will differ for each storage type but needs to be catered for to ensure files are not corrupted.

To reduce the risk of moving files that are hosted in the same directory it would be good to add the ability to filter the input list against file extensions required.

• Add env var for ext list
• Add filtering to input list before processing

The Boto3 SDK fails to connect to S3 buckets where the region is using the v4 protocol.

When supplying environment variables on windows, some are expected to be quoted however this causes issues with the log handler if the filename or log level is quoted.

To make it easier for portability suggest adding a quote stripper to ENV's where required in case they are supplied by the user.

To support debugging verbose logging needs to be enhanced.

• Add more logging to startup process and task
• Add enhanced logging to storage class and methods

As an enhancement include an if statement so if the EC2 instance has an IAM role assigned that grants permissions to S3 than use that for auth otherwise fall back to the local config's AWS secret and access id.