advestis / transparentpath Goto Github PK

I'm trying to build complete path filenames from TransparentPath objects as functions I'm trying to use are expecting path strings.

Ideally I would call a method like path.absolute to get "/my/absolute/path/filename" if local or "gs://my-bucket/filename" if GCS.

For the moment I've coded this functionality as :

base_dir = Path("/images/tfrecords")
TRAIN_FILENAMES = base_dir.glob("train*.*")
prefix = "gs://" if FS == 'gcs' else ''
TRAIN_FILENAMES = [prefix + base_dir.path.as_posix() + "/" + el.name for el in TRAIN_FILENAMES]

Maybe there is a way to do it already that I don't know?
Thanks for your repo!

Path cherche en local malgré .set_global_fs(fs="gcs"), pour un chemin incorrecte

Hello! There is a conflict between the versions of fsspec required by the most recent gcsfs and transparentpath. In particular, gcsfs==2022.2.0 depends on fsspec==2022.02.0 and transparentpath depends on fsspec==2021.10.0. Hopefully this is a quick fix. Thanks for this project, by the way, it's super helpful!

I'm getting :

  File "/usr/local/lib/python3.6/dist-packages/transparentpath/gcsutils/transparentpath.py", line 625, in set_global_fs
    raise ValueError("Need to provide a project name!")

I don't see a way to let transparent path know which project I'd like to use other than using a token

dans le même script j'ai set deux token pour Path. Quand je la lance sans la variable GOOGLE_CRED ça ne marche pas sur la lecture des fichiers sand, mais quand je set GOOGLE_CRED avec le token dev, ça marche.

fsspec==2022.2.0 is causing issues with more updated versions of s3fs. Can this dependency please be loosened. Requesting that we do the same for gcsfs==2022.2.0

ERROR: Cannot install -r requirements.txt (line 10), -r requirements.txt (line 5) and transparentpath==1.1.38 because these package versions have conflicting dependencies.

The conflict is caused by:
    dask 2023.9.2 depends on fsspec>=2021.09.0
    s3fs 2023.9.2 depends on fsspec==2023.9.2
    transparentpath 1.1.38 depends on fsspec==2022.2.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

how can I "read bytes" - this seems like he most basic functionality of path pretty much. how have you implemented the other functionalities with out using "read_bytes"?

use glob("*")

Extract clickable link from FinObject and make it a transparentpath attribute