Comments (15)
oh, very interesting. we'll definitely investigate and have an opinion soon. thanks!
from python-irodsclient.
See WIP here: https://github.com/hechth/fs-irods
from python-irodsclient.
... the repo has become a somewhat standalone resource ...
Just want to confirm that when you say, the repo, you're referring to https://github.com/hechth/fs-irods?
from python-irodsclient.
Cool cool.
Feel free to open new issues here if you encounter problems with the PRC.
from python-irodsclient.
Looks like it may be a good thing on which to build a bilateral,ssh-powered, "rsync-like" file syncher.
from python-irodsclient.
Interesting... could also be a useful abstraction for rsync, ingest, lots of things.
import io
import weakref
from fs.base import FS
from fs.info import Info
import irods.test.helpers as h
from irods.path import iRODSPath
from irods.models import DataObject,Collection
weakrefs = weakref.WeakKeyDictionary()
session = h.make_session()
def norm(*paths):
return str(iRODSPath(*paths))
def is_coll(_abs):
return session.collections.exists(_abs)
# ///////////////////////////////////////////
#
# -- A skeletal and naive PyFilesystem2 implementation.
class iRODS_FS(FS):
def __init__(self, root_path):
super(iRODS_FS,self).__init__()
self.root_path = root_path
def makedir(self,*s,**kw):pass
def removedir(self,*s,**kw):pass
def remove(self,*s,**kw):pass
def setinfo(self,*s,**kw):pass
def openbin(self,path,mode='r',**_):
p = norm(self.root_path, path)
_mode = list(mode)
while _mode.count('b'):
_mode.remove('b')
d = session.data_objects.open(p, ''.join(_mode))
weakrefs [d.raw] = d
return d.raw
def listdir(self, path):
c = session.collections.get(norm(self.root_path, path))
return [o.name for o in c.subcollections + c.data_objects]
def getinfo(self, path, namespaces=None):
p = norm(self.root_path, path)
info = { 'basic':{
'name': p.split('/')[-1],
'is_dir':is_coll(p)
}
}
return Info(info)
# -- ////////////////////////////////////////
if __name__ == '__main__':
home = h.home_collection(session)
try:
session.collections.create(f'{home}/demo/dir1')
session.data_objects.create(f'{home}/demo/emptydata')
UC_enc = '\u1000'.encode('utf8')
with session.data_objects.open(f'{home}/demo/dir1/text.dat','w') as f:
f.write(b'hello'+UC_enc)
# Use iRODS as a PyFilesystem2
ifs = iRODS_FS(home)
# Open binary file as Unicode (default)
contents = ifs.open('/demo/dir1/text.dat').read()
print ('unicode contents length = ',len(contents))
print ('unicode contents = ',contents)
with ifs.open('/demo/dir1/text2.dat','w') as f2:
f2.write(contents + '\u1001')
print ('walking:')
for obj in ifs.walk():
print('\t',obj)
finally:
session.collections.remove(f'{home}/demo',force = True)
from python-irodsclient.
@d-w-moore my plan was to give it a go today to work on this, I might use this as a starting point though I have to say I'm not particularly familiar with iRODS nor the python client.
from python-irodsclient.
@d-w-moore my plan was to give it a go today to work on this, I might use this as a starting point though I have to say I'm not particularly familiar with iRODS nor the python client.
@hechth - You're welcome to mine and play with this code all you want... Meanwhile I'll probably be fleshing it out today into something a little more complete and real. It needs more methods, thread safety and maybe more.
from python-irodsclient.
@d-w-moore We can also schedule acall to coordinate these efforts? I think it will be a bit more complex so it might take time. I started with the skeleton for the extension and checked the options for how to deal with the session, but I!d be very curious to hear how you would implement it. My current approach is to always open a new session for each command so that they are self-contained in that sense. Also implementing unit tests is quite tricky as everything needs to be mocked basically
from python-irodsclient.
@hechth Yes, a call would be good - maybe early next week, Monday (20.11) - Wednesday(22.11) sometime? At any rate, I don't I will be getting as far on it today as I thought.
Regarding the session: that represents state, and my impression is that pyfilesystem2 doesn't deal much with that, beyond storing it in the filesystem object and using _lock to guard against thread conflicts.
The global session
object idea presented in my code snippet above is awkward, as I don't know what is supposed to happen in the case of overlapping access; ie., simultaneous use of irodsfs('/tempZone/home/user')
and irodsfs('/tempZone/home/user/subcollection'
from different threads would seem to demand a shared _lock between the two instances. So maybe better to accept (as a parameter) or generate a new session object for every irodsfs
- ?
from python-irodsclient.
@d-w-moore monday sounds good. Which timezone are you in? I'll continue a bit still today and then I can give an overview on Monday and we can refine the implementation?
from python-irodsclient.
@d-w-moore monday sounds good. Which timezone are you in? I'll continue a bit still today and then I can give an overview on Monday and we can refine the implementation?
Great. I'm in EST time zone
from python-irodsclient.
@hechth changing the openbin
implementation in the above snippet to
def openbin(self,path,mode='r',buffering=-1,**_):
p = norm(self.root_path, path)
_mode = list(mode)
while _mode.count('b'):
_mode.remove('b')
d = session.data_objects.open(p, ''.join(_mode), _buffering = buffering)
if buffering >= 0:
return d
else:
weakrefs[d.raw] = d
return d.raw
And basing on the python-irodsclient from this dev branch:
https://github.com/d-w-moore/python-irodsclient/tree/pyfilesystem2-buffering
we can now run this code After having created /tempZone/home/rods/thing
, via e.g. an iput
):
ifs = iRODS_FS('/tempZone/home/rods')
with ifs.open('thing', 'rb', buffering=2048) as b:
print('buffered: ',b)
b.read(1)
with ifs.open('thing', 'rb', buffering=0) as b:
print('buffered: ',b)
b.read(1)
with ifs.open('thing', 'rb', buffering=-1) as b:
print('buffered: ',b)
b.read(1)
Results are:
buffered: <_io.BufferedReader name='thing'>
buffered: <_io.BufferedReader name='thing'>
buffered: <fs.iotools.RawWrapper object at 0x7fbd86a2c940>
from python-irodsclient.
I think this can be closed as the repo has become a somewhat standalone resource I think and we can move discussions to the issues over there? Thanks for the useful starting points, it has been very helpful :)
from python-irodsclient.
Yes, I am.
from python-irodsclient.
Related Issues (20)
- Renaming a heavy collection fails HOT 2
- Add support for library features API
- Add support for replica truncate API
- Add support for GenQuery2 API HOT 6
- TTL issue for pam_password with v2.0.0 HOT 15
- Document auxiliary test script setupssl.py
- (in iRODS 4.3.2) an iRODSGroup cannot be removed HOT 1
- Prevent creating `iRODSUser` of type `rodsgroup` HOT 1
- Downloading large file to directory throws IsADirectoryError HOT 2
- Make error-passing interface more uniform HOT 1
- Make README clearer with regard to version references
- Trying to add and set non string metadata values raise different errors HOT 2
- Add support for client hints HOT 1
- Investigate interpreter segfault TODO comment for Python 3.6 vs 3.11
- question: is there a way to get events' results that can be captured in server, for example by `Rule`? HOT 4
- Add support for `get_library_features` API HOT 2
- for each replica acls method seems to return available permissions HOT 14
- Return better error on incorrectly constructing `iRODSAccess`
- For enumerating ACLs, PRC should follow ils's pattern and use of specific query
- Simplified acl check for a user
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-irodsclient.