Super new idea: the file voucher about strawberryfield

[dmer]

found my way here via #107
This is a not just a good idea, but might be a functional requirement for some folks.
I was just looking at the S3 multipart upload b/c we're going to have to use that for some of the files in the CAR project and it would be very nice to have an easy-to-use connection to that file for the form.

As I'm writing this there's another thought that is not well-formed, but I'll try - I like the notion that you might be able to choose the voucher from a list of available ones - I'm picturing a possible workflow with a "pending" queue that a user can grab tokens.

Also wondering if this could work w/ AMI? For example could I load all my files to S3, get tokens and then feed them in w/ a spreadsheet?

[pcambra]

This is my free translation with my own comments of the thoughts that @DiegoPino has shared with me via email and slack:

Bottomline as is mentioned here is to be able to upload humongous files before ingesting the content via the webform widget

This not only will serve the needs of creating the ADOs via the Drupal interface (Node/Webform) but also through API. This task can be broken into several parts:

• Upload via the Drupal UI: We need upload by chunks. plupload is an option but there are other solutions based in JS. Comments from Diego include that the S3 dependency can be assumed and this needs to allow [optional] validation of the checksum at the end of the upload.
• API upload. We need an endpoint that requires authentication and allows bulk/chunk uploads.
• Direct upload to storage (complexity ++):
  • Leveraging Minio for S3 so we can use events. Depending on where the file was uploaded and its properties, a new voucher is generated. How do we notify the uploader? email? do we create the user?
  • S3 direct would allow us to use SNS and tunnel this from archipelago.
  • Cron based. This would be basically "brute forcing" by checking every X minutes/hours certain folders (user based?), sensible to date.

Personal note: maybe if we're unsure of what approach to take, we could leave the door open and implement generic processing that then plugins can implement depending on the site options. Another thought I'm having is that maybe the voucher doesn NOT come first to the ADO creation but it is the ADO creation the one specifying "I'm going to use a voucher", then the editor selects how this voucher is going to be generated. Finally, through a simplified edit process, the link is made between ADO and file. This would simplify a lot the process, specially the cron based one.

---

Some preliminary implementation details:

• [Must] Voucher custom entity that implements the logic of the voucher (maybe this could be a handler that can be altered/chosen), forms and permissions
• [Must] A custom widget for uploading the files. Needs to support multipart and resume?
• [Must] Webhook controllers (different depending on the implementation technique, SNS, minio, etc) so notifications are processed.
• [Open question] AMI integration through a service. I would need more information about this.
• [Maybe] Voucher webform element that links the file with the ADO. Maybe it could be an action instead?
• [Misc] Various forms and config

[DiegoPino]

@pcambra I like your approach here. The question of before or at the end will probably have the most complex answer: both. I think this is a good @pcambra + @alliomeria collaboration to find out the different CRUD (create/edit/update/delete) workflows and how/when a Voucher can integrate via the UI/messages and User interaction. This also speaks about maybe different roles users can have in a repo and having a more generalized (as you correctly said) api/services around this may even allow in the future to think of "external systems" adding Vouchers others than the ones we have described here:

My guts tell me we should start with

• Voucher custom entity that implements the logic of the voucher (maybe this could be a handler that can be altered/chosen), forms and permissions. I would add here also: the storage of course and also "the life cycle" as a inherent thing since once used they should In time expire (maybe? Allison may know more.. maybe not immediately thinking of the Batch ingest use case in AMI/Spreadsheet where the same Batch could run a few times while changes/fixes are being made)

What do you think? Does this, more invisible to UI/UX part sounds good? Please feel free to open an ISSUE for that, you get all my help/support/code on this. HUGS!

[DiegoPino]

Yes! I'm actually on that code planning right now and may need more feedback. I had some open ended ideas but happy to hear more workflows and expectations (the simpler for the user the better)

[DiegoPino]

@pcambra I will assign this macro ISSUE to you. It will also entail at least the webform_strawberryfield Module too. As I explained via email it entails more and is a project in itself. I estimate more than 30 days for the whole thing but we can move in steps. You can copy your comments/ideas regarding my email here too. I will also add its own Github Project so we can track this between releases/versions (first time doing this so I may fail). Thanks a lot for taking this one