Cloud computing guide about targets

[wlandau]

The biggest potential difference I see to this (relative to Metaflow's approach to AWS) is that the targets data store will probably live locally. But it's not so bad because drake users want this behavior anyway so they can explore data interactively (example: ropensci/drake#1295). And as @noamross pointed out, aws.s3::s3sync() can upload the data store to an S3 bucket. _targets/ is super light relative to .drake/, so this shouldn't be too painful for most projects.

[wlandau]

If cloudyr packages still work, tar_make_future() can probably already talk to multiple AWS instances: https://gist.github.com/DavisVaughan/5aac4a2757c0947a499d25d28a8ca89b. But the data will still live locally.

[wlandau]

@MilesMcBain, your team uses AWS, right? What's your preferred way to interact with it?

[wlandau]

I read up more on AWS ParallelCluster, AWS Batch, and Metaflow's HPC, and targets' capabilities are not ready for a cloud computing guide yet. But I think development on top of paws can get us much of the way there.

[MilesMcBain]

Hey @wlandau, so far we have preferred to call the AWS CLI directly. This is mainly due to a combination of very simple workflows, and uncertainty about the stability of the AWS-R ecosystem.

[wlandau]

That's helpful, I know cloudyr has had a rough time. What do you think about paws? I'm hoping it can help with ropensci/targets#152 either directly or through HenrikBengtsson/future#415 or mschubert/clustermq#102 (comment).

[wlandau]

Reopening. I plan to write about ropensci/targets#176 at least.

I will keep my eye on R + cloud packages. Looks like aws.s3 has been updated in May, which is a good sign. And paws is under constant development but isn't quite there for the new S3 feature set in targets.

[wlandau-lilly]

Just wrote about S3 integration in the new cloud chapter. Will reopen after ropensci/targets#152.