alpernebbi / git-annex-adapter Goto Github PK

View Code? Open in Web Editor NEW

44.0 44.0 10.0 270 KB

Call git-annex commands from Python

License: GNU General Public License v3.0

Python 100.00%

git-annex-adapter's People

Contributors

Stargazers

Watchers

Forkers

neurodebian mannidung doebi fleischkatapult ngenetzky alexginzburg ma27 dotlambda willmerae edwardbetts

git-annex-adapter's Issues

Tests broken

The tests included in the source do not seem to match the to-be-tested code. The use a completely different API ...Repo.make_annex() etc. Are these meant to work?

Test failures against pygit2 1.2.1

Some fairly trivial errors occur when running the tests using pygit2 1.2.1:

ERROR: test_file_tree (tests.test_repo.TestGitAnnexRepoOnThreeFiles) (file='foo')
Flat AnnexedFileTree should be correct.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_repo.py", line 40, in test_file_tree
    self.assertEqual(tree[f].key, k)
AttributeError: '_pygit2.Blob' object has no attribute 'key'

======================================================================
ERROR: test_file_tree (tests.test_repo.TestGitAnnexRepoOnThreeFiles) (file='bar')
Flat AnnexedFileTree should be correct.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_repo.py", line 40, in test_file_tree
    self.assertEqual(tree[f].key, k)
AttributeError: '_pygit2.Blob' object has no attribute 'key'

======================================================================
ERROR: test_file_tree (tests.test_repo.TestGitAnnexRepoOnThreeFiles) (file='baz')
Flat AnnexedFileTree should be correct.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_repo.py", line 40, in test_file_tree
    self.assertEqual(tree[f].key, k)
AttributeError: '_pygit2.Blob' object has no attribute 'key'

due to some API deprecation I assume.

class AnnexedFileTree can not represent the git index

I want to use git-annex-adapter to add multiple files to git-annex and to also set their metadata.

However, the AnnexedFileTree class expects to work with a committed treeish. But git-annex can already run git-annex metadata -s directly after git-annex add but before git commit.

So for now, I need to first commit a file before I can manipulate the metadata with git-annex-adapter.

Incompatibility with distro versions of git-annex

Currently, git-annex 6.20160727 (or later) is supported, but this isn't in most of the distro repositories:

The adapter should be made compatible with these versions until they are upgraded by the maintainers.

A short list of things that I rely on and aren't available in these versions:

metadata --batch --json (6.20160727)
Metadata fields being enclosed in a fields object in JSON output. (6.20160726)

support for git annex find --metadata

I'd need a function to get annexed files as in

git annex find --metadata my-key="*"

Are you still there to merge a patch and release a new version on PyPI?

Mention datalad

So you know, https://github.com/datalad/datalad is a data management system that appears written in python and deeply interconnects between python and git-annex.

I'm looking into your tool because datalad seems kinda big and unwieldy for getting started small, but I was surprised not to see a reference to it.

Debian packaging

Hello,

I'd like to package git-annex-metadata-gui for Debian. Could you tag a stable release, please?

Thanks.

support git-annex-lookupkey

Instead of relying on parsing the key from the blob, just use git-annex.

[feature] Add Dockerfile

Add Dockerfile for easy verification of dependencies and consistent environment for test execution.

[bug] Version parsing does not support "+git" versions

Python 3.7.0 (default, Sep  5 2018, 03:25:31) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import git_annex_adapter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/git_annex_adapter/__init__.py", line 50, in <module>
    git_annex_version = float(version_str.split('-g', 1)[0])
ValueError: could not convert string to float: '6.20181011+git124'
>>> import os
>>> os.system('git-annex version')
git-annex version: 6.20181011+git124-g94aa0e2f6-1~ndall+1
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.19 bloomfilter-2.0.1.0 cryptonite-0.25 DAV-1.3.2 feed-1.0.0.0 ghc-8.2.2 http-client-0.5.12 persistent-sqlite-2.8.1.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar hook external
operating system: linux x86_64
supported repository versions: 3 5 6
upgrade supported from repository versions: 0 1 2 3 4 5
0

ImportError when used with git-annex 8.20210223-1~ndall+1

NeuroDebian have made a new release of git-annex-standalone, 8.20210223-1~ndall+1. This version string breaks the version detection in git_annex_adapter/__init.py

Traceback (most recent call last):
  File ".../git_annex_adapter/__init__.py", line 56, in <module>
    git_annex_version = float(version_str)
ValueError: could not convert string to float: '8.20210223-1~ndall+1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./repo-mirror/git-annex-aware-repo-server.py", line 32, in <module>
    from git_annex_adapter.exceptions import NotAGitAnnexRepoError
  File ".../git_annex_adapter/__init__.py", line 59, in <module>
    msg = fmt.format(git_annnex_version)
NameError: name 'git_annnex_version' is not defined

Pull request imminent

How to run the tests

I'm trying to run the tests with python -m unittest, but I'm getting the following error:

======================================================================
FAIL: test_process_git_status (tests.test_process.TestProcessOnEmptyAnnex)
Process should be able to read whole output
----------------------------------------------------------------------
Traceback (most recent call last):
  File /tmp/nix-build-python3.6-git-annex-adapter-0.2.0.drv-0/git-annex-adapter-v0.2.0-src/tests/test_process.py, line 154, in test_process_git_status
    + 'and use git add to track)',
AssertionError: Lists differ: ['On [16 chars]'', 'No commits yet', '', 'nothing to commit ([42 chars]ck)'] != ['On [16 chars]'', 'Initial commit', '', 'nothing to commit ([42 chars]ck)']

First differing element 2:
'No commits yet'
'Initial commit'

  ['On branch master',
   '',
-  'No commits yet',
+  'Initial commit',
   '',
   'nothing to commit (create/copy files and use git add to track)']

----------------------------------------------------------------------

Could you please document somewhere how to run the tests? Maybe the cause is that the tests are run in a different order?

Parsing of git_annex_version should be more flexible.

Some git-annex packages format the version in a way that is not expected by the check performed in git_annex_adapter/__init__.py.

    >>> import git_annex_adapter
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/src/app/git_annex_adapter/__init__.py", line 50, in <module>
        git_annex_version = float(version_str.split('-g', 1)[0])
    ValueError: could not convert string to float: '6.20180329+gitga5fe62bed-1~ndall+1'

Fetch/download annexed file?

Hi,

I'm looking for a way to automatically fetch/download annexed files on demand if they are not present locally. Is this possible with this package?

Many thanks

Test failures against git-annex 8.20200908

Is git-annex-adapter intended to work against git-annex 8.x?

We get the following test failures

running install tests
.....F.F./nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 493 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 670 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 889 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
..F............/nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 1646 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.
======================================================================
FAIL: test_init_annex_version_five (tests.test_init_annex.TestInitAnnexOnEmptyRepo)
Repository version 5 should be valid.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_init_annex.py", line 58, in test_init_annex_version_five
    self.assertEqual(annex_repo.config['annex.version'], '5')
AssertionError: '8' != '5'
- 8
+ 5


======================================================================
FAIL: test_init_annex_version_six (tests.test_init_annex.TestInitAnnexOnEmptyRepo)
Repository version 7 should be valid.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_init_annex.py", line 63, in test_init_annex_version_six
    self.assertEqual(annex_repo.config['annex.version'], '7')
AssertionError: '8' != '7'
- 8
+ 7

======================================================================
FAIL: test_process_annex_info_batch (tests.test_process.TestProcessOnEmptyAnnex)
Process should be able to read multiple lines
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_process.py", line 229, in test_process_annex_info_batch
    'remote annex size: 0 bytes',
AssertionError: Lists differ: ['trust: semitrusted', 'remote annex keys: 0'] != ['remote annex keys: 0', 'remote annex size: 0 bytes']

First differing element 0:
'trust: semitrusted'
'remote annex keys: 0'

- ['trust: semitrusted', 'remote annex keys: 0']
+ ['remote annex keys: 0', 'remote annex size: 0 bytes']

----------------------------------------------------------------------
Ran 27 tests in 8.325s

FAILED (failures=3)

which worked with earlier versions of git-annex.