capnfabs / grouse Goto Github PK

I'm using an out of the box example site with the Google Docsy theme. In each version running "hugo serve" or "hugo" builds properly. When I try to use grouse, I get the following:

ERROR 2019/12/22 18:38:19 Failed to read Git log: fatal: not a git repository (or any of the parent directories): .git

I'm not sure what I could be doing wrong so I was hoping to get some help. Here's my debug log.

❯ grouse --debug b616a47158aaf6459119bafa56ad39e7773f7f0b master
Computing diff between revisions b616a47158aaf6459119bafa56ad39e7773f7f0b (b616a47) and master (637d5ad)
Building revision b616a47158aaf6459119bafa56ad39e7773f7f0b (b616a47)…
2019/12/22 18:38:12 Checking out b616a47158aaf6459119bafa56ad39e7773f7f0b (b616a47) to /tmp/grouse-diff361430417/source/b616a47158aaf6459119bafa56ad39e7773f7f0b…
2019/12/22 18:38:12 Found submodules file, figuring out where to source the submodule contents from
2019/12/22 18:38:12 Found commit for submodule in worktree.
2019/12/22 18:38:12 Found submodules file, figuring out where to source the submodule contents from
2019/12/22 18:38:12 Found commit for submodule in worktree.
2019/12/22 18:38:13 Found commit for submodule in worktree.
2019/12/22 18:38:19 …done checking out.
2019/12/22 18:38:19 Running command
> hugo --destination=/tmp/grouse-diff361430417/output
(from directory /tmp/grouse-diff361430417/source/b616a47158aaf6459119bafa56ad39e7773f7f0b)
Building sites … ERROR 2019/12/22 18:38:19 Failed to read Git log: fatal: not a git repository (or any of the parent directories): .git
Total in 1571 ms
Error: Error building site: logged 1 error(s)
Error: Building at commit b616a47158aaf6459119bafa56ad39e7773f7f0b (b616a47) failed: exit status 255

Hugo installed with snapcraft doesn't work with Grouse, because snapcraft has a super restrictive set of permissions by default that doesn't let Hugo write to / read from existing temporary directories. Instead, snaps have their own private view of /tmp (as per here)

I don't know if I want to bother addressing this just yet -- but we ran into it in this discussion and I wanted to track it.

Some ideas:

• Work out if it's possible to grant permissions / mount directories into a snap's FS layout?
• Symlink / hardlink from the temp directory to the /home/ directory?
• Have a fallback mode that puts all grouse storage in e.g. $HOME/grouse-diff, which would work around this problem. But then, we'd need a strategy for cleaning everything up afterwards (temp files don't have this problem because they're deleted automatically from time-to-time)

With grouse 0.2 installed via homebrew, current versions of homebrew complain on each update:

Warning: Calling bottle :unneeded is deprecated! There is no replacement.
Please report this issue to the capnfabs/tap tap (not Homebrew/brew or Homebrew/core):
/usr/local/Homebrew/Library/Taps/capnfabs/homebrew-tap/grouse.rb:6

I think this may be a result of goreleaser's deprecation of the github section used in the current .goreleaser.yml, which I presume would require an update to the grouse.rb homebrew tap.

That is largely speculation on my part however, though it does seem to be a now fixed bug in goreleaser

i.e. https://medium.com/faun/using-vscode-as-git-mergetool-and-difftool-2e241123abe7 ?

Here's my attempt 😅

% grouse HEAD dev --debug
Computing diff between revisions HEAD (0a7c1e1) and dev (20c9cf1)
Building revision HEAD (0a7c1e1)…
2020/03/13 00:52:58 Checking out HEAD (0a7c1e1) to /var/folders/bj/3d96gjw11tqd9y4ws37mg0sc0000gn/T/grouse-diff963426300/source/0a7c1e1cafe7e81a7e07436b33857e109a52167b…
2020/03/13 00:52:59 …done checking out.
2020/03/13 00:52:59 Running command
> hugo --destination=/var/folders/bj/3d96gjw11tqd9y4ws37mg0sc0000gn/T/grouse-diff963426300/output
(from directory /var/folders/bj/3d96gjw11tqd9y4ws37mg0sc0000gn/T/grouse-diff963426300/source/0a7c1e1cafe7e81a7e07436b33857e109a52167b)
Building sites … ERROR 2020/03/13 00:52:59 Failed to read Git log: fatal: not a git repository (or any of the parent directories): .git
Total in 178 ms
Error: Error building site: logged 1 error(s)
Error: Building at commit HEAD (0a7c1e1) failed: exit status 255

Sounds like #9 might be working on this?

Things that would be nice for the future (roughly ordered, see also #enhancements)

Try to autodetect a few common static site generators (hugo, jekyll, gatsby)

Maybe look into how netlify dev works... I'm sure it's probably just grabbing and running whatever's in netlify.toml?

It's not always sufficient to just run hugo for a build -- many sites have different build processes.

e.g.

• my personal site has a build step after running hugo
• The Docsy theme from Google requires running npm install before the build.

So, I think adding a --build-cmd command-line switch would be valuable. It's probably also worth allowing you to specify it twice to get different builds for different versions (like, if you've moved your build script to a new location between the two commits). This would also allow you to do e.g. --build-cmd=hugo --build-cmd=~/Downloads/hugo to diff between your system hugo and the new version you just downloaded.

I guess this should be run in a login shell, because people are used to thinking of their build command as a shell command, and that means that they'd be able to do things like e.g. use their standard aliases, and chain commands (e.g. npm install && hugo).

I was on the fence about whether I should do this or hand-roll my own checkout code when I first built Grouse. Since then:

• I received a report that v0.1.0 had a bug where it never closed file descriptors (i.e. I got stung by assuming that implementing my own checkout code would be simple, and it's never as simple as you expect) which caused checkouts of large repos to fail
• I've received a report (#2) that Grouse doesn't work with Hugo repos that use GitInfo (because the destination that Grouse's checkout mechanism uses isn't a git repo, it's just a file tree on temp disk storage).

That code was also super fiddly and hard to get right, and is badly tested? So with the benefit of hindsight, I'm inclined to try and replace it with the simpler solution.

Something important to check here: not all git versions support multiple worktrees, and there's a big scary warning in the worktree docs saying:

Multiple checkout in general is still experimental, and the support for submodules is incomplete. It is NOT recommended to make multiple checkouts of a superproject.

I wish the author had written why 😅 But it looks like a few brave souls have figured out exactly what the underlying issues are

So, steps for this task:

• Figure out if / which versions of git support worktree add with submodules.
• Evaluate availability of those versions of git ^ on commonly used OSXes, Linuxes
• Figure out exactly how this interplays with submodules, and what the checkout strategy there should be.

Then, once those investigative tasks are done:

• Actually build a strategy that uses git worktree for checkouts.

I still don't have 100% confidence that I've covered all edge cases for Git handling in my code, so if someone runs into a problem, It'd be nice if they could report that easily.

Having said that, I've received bug reports from users without building this functionality, so I think it's a nice-to-have instead of a necessity.

The git diff command being run by grouse produces results that are different from what I would have expected when files are present in first output but not in second output.

Grouse runs git diff --stat f8f5721544d2442b761763b0ee19c074513076f0 4d84acb03641af028a777c7d86a76b5ab1770b86, which produces:

 index.html        |   1 -
 posts/index.html  |   1 +
 posts/index.xml   | 415 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 sitemap.xml       |  14 ++-
 xpub/rc/index.xml | 101 +++++++++++++++++++
 5 files changed, 527 insertions(+), 5 deletions(-)

whereas running git diff --stat f8f5721544d2442b761763b0ee19c074513076f0 (i.e. without the later commit) produces:

 categories/index.html       | 104 ---------------
 index.html                  |   1 -
 index.xml                   | 459 -----------------------------------------------------------------
 posts/index.html            |   1 +
 feed.xml => posts/index.xml |  85 +++---------
 sitemap.xml                 |  14 +-
 xpub/rc/index.xml           | 101 +++++++++++++++
 7 files changed, 132 insertions(+), 633 deletions(-)

The second result is what I would have expected here.

Ok, this time I think I've finally got something pretty performant with minimal hacks.

Here's a series of bash commands I used to do this on a repo with nested submodules (https://github.com/capnfabs/grouse/blob/master/test-fixtures/nested-submodules-missing-submodules.zip)

~# git clone /app/nested-submodules-missing-submodules clone
Cloning into 'clone'...
done.
~# cd clone/
~/clone# git submodule init themes/bare/
Submodule 'themes/bare' (https://github.com/orf/bare-hugo-theme.git) registered for path 'themes/bare'
~/clone# export REPO=/app/nested-submodules-missing-submodules
~/clone# git -c submodule."themes/bare".url=$REPO/themes/bare submodule update themes/bare
Cloning into 'themes/bare'...
done.
Submodule path 'themes/bare': checked out '6f2f95a1ce89a6e642491072e848eba0b23fac2d'
~/clone# cat .git/modules/themes/bare/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	worktree = ../../../../themes/bare
[remote "origin"]
	url = /app/nested-submodules-missing-submodules/themes/bare
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
~/clone# git submodule update themes/bare
~/clone# # i feel like the remote is supposed to point at the original submodule URL, not this one
~/clone# cd themes/bare/
~/clone/themes/bare# git remote -v
origin	/app/nested-submodules-missing-submodules/themes/bare (fetch)
origin	/app/nested-submodules-missing-submodules/themes/bare (push)
~/clone/themes/bare# git remote remove origin
~/clone/themes/bare# git submodule init assets/css/
bulma/     site.sass
~/clone/themes/bare# git submodule init assets/css/bulma
Submodule 'assets/css/bulma' (https://github.com/jgthms/bulma.git) registered for path 'assets/css/bulma'
~/clone/themes/bare# git -c submodule."assets/css/bulma".url=$REPO/themes/bare/assets/css/bulma submodule update assets/css/bulma
fatal: repository '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' does not exist
fatal: clone of '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' into submodule path 'assets/css/bulma' failed
~/clone/themes/bare# echo $?
128
~/clone/themes/bare# git submodule update --depth 1 assets/css/bulma
Cloning into 'assets/css/bulma'...
remote: Enumerating objects: 898, done.
remote: Counting objects: 100% (898/898), done.
remote: Compressing objects: 100% (870/870), done.
remote: Total 898 (delta 58), reused 545 (delta 22), pack-reused 0
Receiving objects: 100% (898/898), 63.89 MiB | 269.00 KiB/s, done.
Resolving deltas: 100% (58/58), done.
Checking connectivity... done.
fatal: reference is not a tree: 617febbfbf5b95e827d20654a16748c66bb5d647
Unable to checkout '617febbfbf5b95e827d20654a16748c66bb5d647' in submodule path 'assets/css/bulma'
root@0778ade78b9c:~/clone/themes/bare# cd assets/css/bulma/
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# git fetch --depth=1000000
remote: Enumerating objects: 14301, done.
remote: Counting objects: 100% (14301/14301), done.
remote: Compressing objects: 100% (3631/3631), done.
remote: Total 13879 (delta 10016), reused 13404 (delta 9542), pack-reused 0Receiving objects:  99% (1

Receiving objects: 100% (13879/13879), 16.74 MiB | 186.00 KiB/s, done.
Resolving deltas: 100% (10016/10016), completed with 273 local objects.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From https://github.com/jgthms/bulma
 * [new tag]         0.0.1      -> 0.0.1
 * [new tag]         0.0.10     -> 0.0.10
 * [new tag]         0.0.11     -> 0.0.11
 * [new tag]         0.0.12     -> 0.0.12
 * [new tag]         0.0.13     -> 0.0.13
 * [new tag]         0.0.14     -> 0.0.14
 * [new tag]         0.0.15     -> 0.0.15
 * [new tag]         0.0.16     -> 0.0.16
 * [new tag]         0.0.17     -> 0.0.17
 * [new tag]         0.0.18     -> 0.0.18
 * [new tag]         0.0.19     -> 0.0.19
 * [new tag]         0.0.2      -> 0.0.2
 * [new tag]         0.0.20     -> 0.0.20
 * [new tag]         0.0.21     -> 0.0.21
 * [new tag]         0.0.22     -> 0.0.22
 * [new tag]         0.0.23     -> 0.0.23
 * [new tag]         0.0.24     -> 0.0.24
 * [new tag]         0.0.25     -> 0.0.25
 * [new tag]         0.0.26     -> 0.0.26
 * [new tag]         0.0.27     -> 0.0.27
 * [new tag]         0.0.28     -> 0.0.28
 * [new tag]         0.0.3      -> 0.0.3
 * [new tag]         0.0.4      -> 0.0.4
 * [new tag]         0.0.5      -> 0.0.5
 * [new tag]         0.0.7      -> 0.0.7
 * [new tag]         0.0.8      -> 0.0.8
 * [new tag]         0.1.0      -> 0.1.0
 * [new tag]         0.1.1      -> 0.1.1
 * [new tag]         0.1.2      -> 0.1.2
 * [new tag]         0.2.0      -> 0.2.0
 * [new tag]         0.2.1      -> 0.2.1
 * [new tag]         0.3.0      -> 0.3.0
 * [new tag]         0.3.1      -> 0.3.1
 * [new tag]         0.3.2      -> 0.3.2
 * [new tag]         0.4.0      -> 0.4.0
 * [new tag]         0.4.1      -> 0.4.1
 * [new tag]         0.4.2      -> 0.4.2
 * [new tag]         0.4.4      -> 0.4.4
 * [new tag]         0.5.0      -> 0.5.0
 * [new tag]         0.5.1      -> 0.5.1
 * [new tag]         0.5.2      -> 0.5.2
 * [new tag]         0.5.3      -> 0.5.3
 * [new tag]         0.6.0      -> 0.6.0
 * [new tag]         0.6.1      -> 0.6.1
 * [new tag]         0.6.2      -> 0.6.2
 * [new tag]         0.7.0      -> 0.7.0
 * [new tag]         0.7.1      -> 0.7.1
 * [new tag]         0.7.2      -> 0.7.2
 * [new tag]         0.7.3      -> 0.7.3
 * [new tag]         0.7.4      -> 0.7.4
 * [new tag]         0.7.5      -> 0.7.5
 * [new tag]         0.8.0      -> 0.8.0
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# cd -
/root/clone/themes/bare
root@0778ade78b9c:~/clone/themes/bare# git submodule update assets/css/bulma
Submodule path 'assets/css/bulma': checked out '617febbfbf5b95e827d20654a16748c66bb5d647'

So basically, the idea is:

• Clone from the current working tree
• Init each submodule, attempt to clone from the filesystem
• Then, we should probably modify the remote such that 'origin' points at the real remote again
• If cloning from filesystem doesn't work, just use a regular git submodule update.

Something that I don't love about this is -- you can't use git submodule update --depth 1 for nested submodules, because under some circumstances, they point at commits (which you can't fetch directly from remotes). This is just a git problem in general though.

I tested this all on git 2.7.4, which is the current version running on Ubuntu LTS Xenial. I think it would be good to try on git 2.1.4 (Debian Jessie) as well -- but that's probably the oldest stable version we need to support by a long way.

Quick note to performance -- apparently if the cloned repo is on the same disk, then git uses hardlinks so this is mega fast (see Local Protocols in Git On the Server)