Comments (11)
Note this is behavior in #3 currently.
from osfclient.
I think fetch
is doing it correctly. materialized_path
is the path relative to the storage provider root, so you'll want to prefix it with the provider shortname. We don't try to enforce any name uniqueness constraints across providers.
from osfclient.
And one thing to watch out for: GoogleDrive, S3, and osfstorage are not always mappable to filesystems without special handling. All are happy to have a file named foo
and a directory named foo
in the same path. GDrive takes it a step farther and is happy to have two files named foo
in the same directory.
from osfclient.
thanks @felliott
...but let me argue for the other behavior here: it is simply not useful to require people to descend into subdirectories to get access to specific files :).
just to make life more complicated there's all sorts of things that we could do here ... some not necessarily orthogonal options:
- have an explicit policy about what to do in case of collision
- permit a specific osfstorage to be designated as 'root' (and even have that be a default for, say, osfstorage, but make it overridable)
- provide a JSON mapping file that maps between filenames and OSF locations/paths.
I think designating osfstorage
as root is a good default, and then they can fail if they have dared to name something under osfstorage
that is also a storage type, e.g. if they have a dropbox
file in their osfstorage
location. All fun things to test :)
Is there any other remote integration we should be looking to for guidance here, or has this not come up before? I note that there is no way to do a download of an entire project's files as a single zip - it's storage specific.
from osfclient.
OK, well, for now:
% python -m osfclient ls 7g6vu
looking at: testfoo
dropbox/Questionnaire.docx
github/README.md
osfstorage/Recycling Event.png
osfstorage/osf_subdirectory/
osfstorage/osf_subdirectory/dotguide.pdf
from osfclient.
👍 to the "well, for now". As far as having the files in separate directories based on provider, I'd argue that finding files nowadays comes in two styles:
- the rigidly hierarchical (myself)
- the Alfred-loving fuzzy-finders (reasonable people)
I agree that it may not be useful to go directory diving, but most of those folks have moved on to fuzzy-finding, so directories are somewhat irrelevant anyway. For myself, I like distinguishing between what's on my GDrive, what's on my osfstorage, and what's on my Dropbox. I use each of those for different purposes. I especially don't want those mixed into whatever GitHub project I have linked up. If I did that, my flake8 would become self-aware and murder me, and no jury would convict it. (Of course, that could just be an argument for making github/bitbucket/gitlab a special case).
from osfclient.
I like (what a surprise) the behaviour of having subdirectories for each storage. It makes it explicit where a file came from and avoids heuristics to deal with naming collisions. Another argument for sub dirs would be that you are probably keeping different things in the different storage providers (code on github, data on S3, ...) so you want them to be different.
One thing I would like is an option --no-subdir
(needs a better name) in the case where there is only one storage provider active.
from osfclient.
One other thought: osf-cli should be idempotent, i.e.:
osf push foo --create "project title"
osf fetch foo <newproj>
should result in fetch having to do nothing; and likewise
osf fetch foo <newproj>
osf push foo
should do nothing.
from osfclient.
from osfclient.
I'd expect it to show me the subdirectories ... especially as you might have notebooks in multiple storages. I really don't like having files dropped in the root directory because I don't know what a good strategy is for handling conflicts when two or more storages give you the same path.
So I still prefer the --no-subdir
option for the case of only one storage being active, and in the future maybe a .osfconfig
that specifies a mapping from storage provider to local fs path.
from osfclient.
We now have basic config file support. I think adding to the config file a way to specify mappings from "name of the storage on osf.io" to "local path" is the way to go. By default we continue to do what osf
does not (create subdirectories for each storage backend) but we let the (power) user reconfigure it.
from osfclient.
Related Issues (20)
- ConnectionError of osf clone HOT 7
- Incorrect error on large files
- During recursive upload subfolder name is changed if it contains parent folder name as substring HOT 1
- Fetch is extremely slow for large repos
- add size atribute to File class
- Add a special exception for responses with 4xx status
- add token option to client HOT 3
- How to backup wiki-pages via osfclient? HOT 1
- upload directory into remote path will create new directory in root
- upload directory into remote folder will create another new folder in root
- KeyError: 'content-length' when cloning from OSF if Github-Add-on is activated HOT 1
- Show progress when uploading
- Unable to upload file into existing folder in OSF storage HOT 6
- Slow indexing of folder tree HOT 1
- transfer completes but `osfclient.cli.upload` hangs
- Folder indexing returns incomplete list of files HOT 3
- Cannot access private project hosted by an institution HOT 4
- RuntimeError: Response has status code 502 not (200,)
- I have ConnectionError when download large dataset.
- RuntimeError status code 500
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from osfclient.