Comments (8)
The biggest potential difference I see to this (relative to Metaflow's approach to AWS) is that the targets
data store will probably live locally. But it's not so bad because drake
users want this behavior anyway so they can explore data interactively (example: ropensci/drake#1295). And as @noamross pointed out, aws.s3::s3sync()
can upload the data store to an S3 bucket. _targets/
is super light relative to .drake/
, so this shouldn't be too painful for most projects.
from targets.
If cloudyr
packages still work, tar_make_future()
can probably already talk to multiple AWS instances: https://gist.github.com/DavisVaughan/5aac4a2757c0947a499d25d28a8ca89b. But the data will still live locally.
from targets.
@MilesMcBain, your team uses AWS, right? What's your preferred way to interact with it?
from targets.
I read up more on AWS ParallelCluster, AWS Batch, and Metaflow's HPC, and targets
' capabilities are not ready for a cloud computing guide yet. But I think development on top of paws
can get us much of the way there.
from targets.
Hey @wlandau, so far we have preferred to call the AWS CLI directly. This is mainly due to a combination of very simple workflows, and uncertainty about the stability of the AWS-R ecosystem.
from targets.
That's helpful, I know cloudyr
has had a rough time. What do you think about paws
? I'm hoping it can help with ropensci/targets#152 either directly or through HenrikBengtsson/future#415 or mschubert/clustermq#102 (comment).
from targets.
Reopening. I plan to write about ropensci/targets#176 at least.
I will keep my eye on R + cloud packages. Looks like aws.s3
has been updated in May, which is a good sign. And paws
is under constant development but isn't quite there for the new S3 feature set in targets
.
from targets.
Just wrote about S3 integration in the new cloud chapter. Will reopen after ropensci/targets#152.
from targets.
Related Issues (20)
- Set up Discussions HOT 1
- targets-minimal repo doesn't exist HOT 2
- Small typos in drake chapter HOT 2
- Explain that directories as file targets should be nonempty
- Add more mentions of tarchetypes custom invalidation rules? HOT 3
- Clarify when (not) to use tar_read() HOT 2
- Add link to targetopia post in the intro? HOT 1
- Document changes to configuration
- Discuss shortcutting and branch selection in the debugging and dynamic branching chapters
- Load packages in consistent manner in document HOT 3
- Small typos in Advanced topic of HPC chapter HOT 1
- Faulty Hyperlink in User Manual Chapter 2.4
- Chapter on configuration and sub-projects HOT 2
- Maintainer out of office: September 13-20, 2021 HOT 2
- Typo repeated word in Chapter 3 on Debugging
- Debugging stochastic pipelines HOT 1
- explain tar_option_set(error = “null”) in the debugging chapter HOT 2
- explain new seed policy HOT 1
- New chapter on pseudo-random number generation HOT 1
- Incorrect link to functions chapter? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from targets.