fabacab / git-archive-all.sh Goto Github PK

For --format zip, it tries to append submodule files by zip -g command,
as tar --concatenate for --format tar.

However, it just adds the submodule's zip archive file.

Perhaps it is because environment zip command does not have -g option.

Procedure

% git -c protocol.file.allow=always submodule add ../b.git b
% git commit -m "b"
% git archive-all --format zip ../01.zip
  adding: b.zip (stored 0%)

% 7z l ../01.zip

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 793 bytes (1 KiB)

Listing archive: ..\01.zip

--
Path = ..\01.zip
Type = zip
Physical Size = 793
Comment = 506c134f55e7ebc19cb5cf8ac3c15d4ba9a0a458

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2023-03-29 09:12:51 .....           42           42  .gitmodules
2023-03-29 09:12:51 D....            0            0  b
2023-03-29 11:36:59 ....A          260          260  b.zip                      # <-- Just a zip archive is added
------------------- ----- ------------ ------------  ------------------------
2023-03-29 11:36:59                302          302  2 files, 1 folders

% zip --help
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
Zip 3.0 (July 5th 2008). Usage:
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
  The default action is to add or replace zipfile entries from list, which
  can include the special name - to compress standard input.
  If zipfile and list are omitted, zip compresses stdin to stdout.
  -f   freshen: only changed files  -u   update: only changed or new files
  -d   delete entries in zipfile    -m   move into zipfile (delete OS files)
  -r   recurse into directories     -j   junk (don't record) directory names
  -0   store only                   -l   convert LF to CR LF (-ll CR LF to LF)
  -1   compress faster              -9   compress better
  -q   quiet operation              -v   verbose operation/print version info
  -c   add one-line comments        -z   add zipfile comment
  -@   read names from stdin        -o   make zipfile as old as latest entry
  -x   exclude the following names  -i   include only the following names
  -F   fix zipfile (-FF try harder) -D   do not add directory entries
  -A   adjust self-extracting exe   -J   junk zipfile prefix (unzipsfx)
  -T   test zipfile integrity       -X   eXclude eXtra file attributes
  -!   use privileges (if granted) to obtain all aspects of WinNT security
  -$   include volume label         -S   include system and hidden files
  -e   encrypt                      -n   don't compress these suffixes
  -h2  show more help
  

If zip -g doesn't work as expected,
you should create a tar archive, and then, recompress into a zip file.

Maybe a more friendly message could be helpful, or something in the docs, to tell Mac users to run brew install gnu-tar.

I don't know the history of line 107 ([[ $(uname) == "Darwin" ]] && TARCMD=gnutar) (maybe macports?), but when it is installed, it is linked as gtar

I don't understand what the following line (currently at https://github.com/fabacab/git-archive-all.sh/blob/master/git-archive-all.sh#L242 ) attempted to achieve:

TREEISH=$(git submodule | grep "^ .*${path%/} " | cut -d ' ' -f 2)

This is completely

First of all, git submodule only lists the "direct" submodules, not the "transitive" ones. This may be related to #2. Consider using something like git submodule foreach --recursive pwd.

The grep part assumes that the current state of the submodule is clean (the first char is for "clean",+for "changes made", etc.). That's not guaranteed. Indeed,git-archive-all.sh --tree-ish only really makes sense when the given tree-ish is different from HEAD.

The cut part tries to finish the regex matching that should have been done in grep; see grep -o.

It doesn't care anywhere about the original --tree-ish argument at all.

Hi,

we're currently discussing[1] why did you crawl directory to find submodule directory, and not using git submodule foreach command. I would be happy if you give us any comment on that.

git archive -o hoge.tar hoge/
git submodule foreach 'tar -C .. -rvf hoge.tar $path'

[1] vcstools/vcstools#88 (comment)

I run ./git-archive-all.sh --format=zip and the script say's Unrecognized option: --format=zip, am I not doing it right?

Rather than output a file, in my particular case I want to pipe the output of git-archive-all.sh along to another program. It would be convenient if there was an option to output the result to stdout like git archive does.

Script cleans up after the outermost archive, but the archive for the submodule(s) linger.
This prevents back-to-back executions without manual intervention.

Ammendment: This appears to be important, my format is tar.gz. I see now that tar and zip are cleaned up.

Only tar and zip is supported. git archive supports also tar.gz

I've created a simple Bash script reproducibility-test.sh to test the reproducibility of various --format options. I've also compared it to git's built-in archive command.

Here are sample results for a git repo with submodules:

[tadej@tlinux64 genesis]$ ./reproducibility-test.sh 
  adding: genjs.zip (stored 0%)
  adding: genjs.zip (stored 0%)
  adding: genjs.zip (stored 0%)
/tmp/reproducibility_test.WagXHM ~/Genialis/genesis
88baafe24913ccd6ec0945b1b3f4566b  git-archive-all-tar_only-1.tar
88baafe24913ccd6ec0945b1b3f4566b  git-archive-all-tar_only-2.tar
88baafe24913ccd6ec0945b1b3f4566b  git-archive-all-tar_only-3.tar
09f08547cfc2e70bfbae9d570583246e  git-archive-all-tar_with_gzip-1.tar.gz
71bdb10ddf853300f5a98bfe8eaa3b96  git-archive-all-tar_with_gzip-2.tar.gz
3e0d95d0f2dcc4385d5d6deec58b7ef4  git-archive-all-tar_with_gzip-3.tar.gz
06181cf449d7137817d84c3826e64b0b  git-archive-all-zip-1.zip
8681020a6fde7e750419b6e821ad2d9b  git-archive-all-zip-2.zip
c4b9933c33c0e9c4e7bac35cf23f7294  git-archive-all-zip-3.zip
97bafb747b169297f1ebf488dcc9ae5c  git-archive-tar_only-1.tar
97bafb747b169297f1ebf488dcc9ae5c  git-archive-tar_only-2.tar
97bafb747b169297f1ebf488dcc9ae5c  git-archive-tar_only-3.tar
d2c5692100208019da62f26ddb1719a0  git-archive-tar_with_gzip-1.tar.gz
d2c5692100208019da62f26ddb1719a0  git-archive-tar_with_gzip-2.tar.gz
d2c5692100208019da62f26ddb1719a0  git-archive-tar_with_gzip-3.tar.gz
7a1cb6286609909a50b162105ef13eee  git-archive-zip-1.zip
7a1cb6286609909a50b162105ef13eee  git-archive-zip-2.zip
7a1cb6286609909a50b162105ef13eee  git-archive-zip-3.zip
~/Genialis/genesis
[tadej@tlinux64 genesis]$

As can be seen from the results, git's built-in archive command always creates reproducible archives, regardless of the --format option.

On the contrary, git-archive-all.sh only creates reproducible archives with the --format tar option.

If A has submodule B, and B has submodule C, running this script on A misses out C.

If you have two submodules in the same directory that have overlapping names, git-archive-all.sh fails to grab one.

E.g.
libs/test
libs/testing

To fix this, change line #157 from
TREEISH=$(git submodule | grep "^ .${path%/}" | cut -d ' ' -f 2) # git submodule does not list trailing slashes in $path
to
TREEISH=$(git submodule | grep "^ .${path%/} " | cut -d ' ' -f 2) # git submodule does not list trailing slashes in $path

All I am doing is adding a space inside the grep after }.

When creating TMPDIR on OS X it doesn't seem to be unique on each run of the script, especially if using the "zip" type. The fix for me was to initialize TMPDIR as follows:

TMPDIR=`mktemp -d 2>/dev/null || mktemp -d -t '$PROGRAM.XXXXXX'`

This fix is safe for both Linux and OS X.