vmbrasseur / iaupload Goto Github PK

• If --identifier isn't passed but --metadata is, chop the extension off the metadata filename & ask the user whether to use the file basename as the identifier
• If no, either prompt for an identifier or bail w/an error asking them to pass --identifier (pick one implementation)
• Make sure --identifier still isn't a required flag in the case the file basename is used

There are no tests whatsoever. This needs to change.

Right now the files to upload are hard coded in an array in the script.

1. Add a --file flag (repeatable). Can be used in conjunction with the --directory flag ( #5 ).
2. Populate the array from the value(s) in this flag(s)
3. For each file in the array, check that it exists & is readable. Log an error (but do not exit) else.
4. Update documentation.

Currently the location/name of the metadata YAML file is hard coded in mdfile = "/Users/brasseur/Desktop/md.yaml"

1. Add a --metadata flag for the user to define the location/name of the file containing the metadata for the item.
2. Throw an error and exit if the file cannot be found/read/parsed.
3. Update documentation.

Right now the authorization keys are located in the config file. There ought to be a way to specify them on the command line instead.

1. Add --secretkey and --accesskey flags
2. If one is specified, error & exit if other is not also specified
3. If these are specified, they override any values in the config file
4. Defensive programming: if there are no keys set at all (via flag or config), error & exit
5. Update documentation

Currently, the identifier for the IA item is hard coded in identifier = "sfperlmongerslightningtalks2014"

1. Add an --identifier flag so this can be specified on the command line
2. Do a little spelunking to determine whether it's a new item or an existing one
3. Update documentation

I sent the repo link to a Windows user. He was perplexed. Fair 'nuff.

Update the docs to show how to install/use on Windows.

I'm working on uploading the videos from my three talks at Open Source Bridge last week. It occurs so me that it'd be dead handy to have iaupload have a batch mode so it can create/upload to multiple items in succession rather than just one item at a time.

1. Add a function for outputting messages to STDOUT.
2. This function should check these flags. If --verbose, output more. If --quiet, output nothing.
3. Update documentation

The --debug flag will tell the script to perform all operations but NOT actually upload files or create/modify the item. Optionally, it may also output additional information.

This may need to work with the function added in #6.

There currently is no help message for this script. That really, really needs to change.

Don't forget to update the documentation once this flag is added.

Currently the config file (holds auth keys) is hardcoded as config = "./iaupload.yaml".

1. Add a command line flag which will allow the user to define a different config file.
2. Add defensive programming (file exist? can be read? etc.).
3. Update documentation.

This may not be necessary after #10, but it might be nice to be able to type pydoc iaupload and get the contents of the README.md file (parsed/rendered for markdown, if necessary).

The output right now currently looks like this:

Epicurus:iaupload brasseur$ python ./iaupload.py --config config.yaml --metadata ato-ia-api.yaml --identifier ato2015-ia-apis /Users/brasseur/Dropbox/Writing/Presentations/2015/ATO/Internet_Archive-Universal_Access_Open_APIs.pdf
There are 1 files to upload.

Uploading  /Users/brasseur/Dropbox/Writing/Presentations/2015/ATO/Internet_Archive-Universal_Access_Open_APIs.pdf

It would be helpful (to me, at least) if at the end it said something like:

You can view your item at $url

1. Whenever a file is opened, check that it worked. Error/exit else.
2. Same for whenever a file is closed.

trimalchio:iaupload brasseur$ time ./iaupload.py --identifier=mikefriedmanrealworldmongodbaggregationforperl ~/Desktop/friedo2.mov
There are 1 files to upload.

Uploading  /Users/brasseur/Desktop/friedo2.mov


real    646m10.271s
user    1m0.059s
sys 1m37.139s

There are 1 files to upload? Really, Vicky? Subject/verb agreement much?

Yes, it's a silly little bug but it's going to drive me nuts if I see it too often.

#4 adds a --file flag for specifying the files to upload. This is a shortcut for that.

1. Add a --directory flag (not repeatable). May be used in conjunction with --file
2. Populate the files array by traversing this directory (recursively). All files in this directory will be added to the file array and will be uploaded
3. Do a check to confirm the directory is readable
4. Update documentation

From the code:

# TODO: change this to a boolean to avoid the +=1 each iteration of the loop
#       just set it to FALSE at the end of its if block
num = 1

This is just used to determine whether it's the first file uploaded to the item. If it is, it includes the metadata. If not, no metadata is needed.

You will probably need to have a look at the item to determine some of this.

• Item exists?
  • Don't send metadata (would clobber existing metadata)
• Item doesn't exist?
  • Only send metadata with first file uploaded

May be particularly useful in case of --quiet ( #6 ).

1. Add a logging function (check for standard/existing logging libraries before rolling your own).
2. If --log, write to a default log file/location.
3. If --logfile, write to that log file/location instead.
4. Update documentation.

While no metadata fields are officially "required", some are very smart to include.

At the very least, we should make sure that every metadata file contains a title.

This probably should be handled in a new function. Later, if we decide other fields should be verified in some way, we can add them to the function.

It can take a long time for some files to upload. It'd be nice if there were some sort of progress indicator so the user knows that, yes, things are still chugging along.

There may already be a Python library for this. Have a look before rolling your own.