jwestgard / aws-archiver Goto Github PK

In order to run against large quantities of content, we need better exception handling, with custom exceptions for the errors that seem likely to occur (e.g. local file not found, etag mismatch, communication error).

It would be nice not to have to hit S3 during development and testing of this tool.

Boto3's stubber might be one solution. https://botocore.amazonaws.com/v1/documentation/api/latest/reference/stubber.html

Currently deposit can be carried out against a single asset or a batch of assets defined by a text file listing their md5 checksums and paths. The task here is to develop a "batch-deposit" subcommand that can be used to launch a series of deposits.

Requirements:

1. Runs against a manifest containing the paths to the manifests of individual batches.
2. In addition to the audit records recorded by the existing batch deposit subcommand, the tool should record summary results of each batch (number of assets successfully transferred, number of failures, total bytes transferred).

The project is designed to preserve "original order" of files relative to the batch by creating an AWS keypath that matches the string representation of the directory path on the local filesystem. This is a neccessary requirement in cases where the files being archived represent a "package" of related files, and also helps prevent path collisions at deposit time. The batch manifest, however, normally contains the full local path to the file being archived.

Currently, the calculation of the path relative to the batch root relies on the user supplying the absolute local path to the batch root with the -r, --root option. It has been observed that the calculation of the relative path can go awry, causing the deposit to fail. Until a more reliable solution can be devised, a workaround is to run the script from the root of the batch, with the default option value ('.').

The task of this ticket is to ensure that the above calculation of relative paths is reliable and that potential errors are handled appropriately.

The built-in help provided by argparse does not seem to categorize the command-line options correctly. Things that are required seem to be listed as optional. This could be because of the subcommand structure.

To see the problem, compare the built-in help text with the "corrected" summary version included in the README file. Particularly, one or the other of options -a or -m is required but the help text does not make this clear.