tgbugs / augpathlib

Augmented pathlib. Everything else you could do with a path.

augpathlib

Augmented pathlib. Everything else you could do with a path.

Introduction

Do you like pathlib?
Have you ever wanted to see just how far you can push the path abstraction?
Do you like using the division operator in ways that could potentially cause reading from the network or writing to disk?
Then augpathlib is for you!

Details

augpathlib makes extensive use of the pathlib Path object (and friends) by augmenting the base PosixPath object with additional functionality such as getting and setting xattrs, syncing with other mapped paths etc.

In essence there are 3 ways that a Path object can be used: Local, Cache, and Remote. Local paths return data and metadata that are local the the current computer. Cache paths return local metadata about remote objects (such as their remote id). Remote objects provide an interface to remote data that is associated with a path.

Remote paths should be back by another object which is the representation of the remote according to the remote's APIs.

Remote paths are only intended to provide a 1:1 mapping, so list(local.data) == list(remote.data) should always be true if everything is in sync.

If there is additional metadata that is associated with a local path then that is represented in the layer above this one (currently DatasetData, in the future a validation Stage). That said, it does seem like we need a more formal place that can map between all these things rather than always trying to derive the mappings from data embedded (bound) to the derefereced path object.

Installing on MacOS catalina 10.15.5+

If you are getting a failed build for pxattr, please continue reading the following. When using LLVM (9.0.1+) or another 3rd party library for Clang on MacOS, there was an issue that requires a possible reinstall if you getting a failed build for pxattr. See LLVM Issue for details. To reinstall LLVM on brew or anaconda you can use the following examples.

> brew reinstall llvm 

or

> conda install --force-reinstall llvm 

If the reinstall fails to clear the issue consider uninstalling that library and use the default Clang from Xcode. It should be the following version or higher.

> clang -v
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

augpathlib's Issues

consider __init_subclass__

This might allow us to work around some of the nasty issues/hacks that we pull with new.

EatHelper not populating as expected

Manifests as an AttributError on Path.setxattrs, which leads to the question -- how the heck does anything at all work?!

setting cache meta during data setter is likely broken

augpathlib/augpathlib/core.py

Line 1232 in 82e8f60

cache.meta = cmeta # glories of persisting xattrs :/

The logic in here is bad. I'm 99% sure that we should not be trying to set meta here and should check if cmeta is none or empty.

In nearly all cases EAT metadata (xattrs or ads) should already have been placed on the inode we are about to write if it exists at all, and we check for that. However there are edge cases where meta can be erased during _write_chunks (which is why write chunks exists). When this happened we would try to write meta, but it would fail because there was no setter defined. I'm pretty sure that a setter should never be defined because cmeta = None will cause the same issue. I think that whole branch should likely be an error because it means that somehow we lost the metadata half way through the process, which should never have happened.