panic: runtime error: invalid memory address or nil pointer dereference about fzf

[LangLangBart]

Do you think we should do this?

Have not found a satisfactory explanation that justifies the change beyond "it works", and would like to spend more time examining the source code.

[LangLangBart]

The inability to reproduce the issue on macOS was due to the zsh package receiving an openSuse patch approximately 7 months ago¹. This patch, titled pipe-less-and-signals-handling.patch², was included in revision 103.

If one were to check out revision 102 of the zsh package and try running fzf completion there, no issue would occur.

---

Instructions for Building zsh Revision 102

Using osc³ to work with the Open Build Service and a Docker container.

Create an account on: build.opensuse.org

Create an oscrc file containing your username and password.

[general]
apiurl=https://api.opensuse.org

[https://api.opensuse.org]
user=<HERE YOUR NAME>
pass=<HERE YOUR PASSWORD>
credentials_mgr_class=osc.credentials.PlaintextConfigFileCredentialsManager

FROM registry.opensuse.org/opensuse/tumbleweed:latest
RUN zypper refresh
RUN zypper --verbose --non-interactive dist-upgrade --auto-agree-with-licenses

# some tools
RUN zypper install --no-confirm fzf git vim zsh
# Install OSC and other tools
RUN zypper install --no-confirm osc build obs-service-format_spec_file \
    patterns-devel-base-devel_basis hostname sudo tar zstd diffutils

# Copy the OSC configuration file with username and password
COPY oscrc /root/.config/osc/oscrc
RUN osc -A https://api.opensuse.org checkout openSUSE:Factory/zsh && cd $_ && osc up --revision 102
WORKDIR openSUSE:Factory/zsh

# directory and shell level
ENV PROMPT="%~ %L > "

Start building the Docker image and run it with the --privileged flag.
NOTE: The osc build command cannot be run during docker build … because it would cause a permission error when attempting to mount proc. Therefore, we perform it after executing docker run ….

mount: /var/tmp/build-root/standard-x86_64/.mount/proc: permission denied.
       dmesg(1) may have more information after failed mount system call.

docker build --tag opensuse_osc --file Dockerfile_osc .
docker run --tty --interactive --rm --privileged opensuse_osc:latest /usr/bin/zsh

Build the zsh package for revision 102, start a new session, and notice that the issue no longer occurs.

# process will take ~5min
osc build
# start the `zsh` session
/var/tmp/build-root/standard-x86_64/usr/bin/zsh -f
# test sourcing and completion with fzf, no issue anymore
source <(fzf --zsh)
kill **

---

Note

FYI: The issue doesn't occur on openSuse Leap (predecessor to openSuse Tumbleweed) because Leap runs a different zsh version (5.6). This is also why opensuse/tumbleweed seems to be the only Linux distro to have this issue. Tested on Ubuntu, Alpine, etc.

---

Alternative solutions to a067458

• [openSuse]: 🟡 Contact the maintainer to re-evaluate the changes in revision 103 with the new information provided here.

  • Ping @thesp0nge, who appears to be involved in maintaining the zsh package, based on the revision history.

• [fzf]: 🔴 Using process substitutions could have worked as well, but only with zsh version >=5.6⁴; otherwise, it causes a blockage

  • Ref: #494

Footnotes

1. Revisions of zsh - openSUSE Build Service ↩

2. File pipe-less-and-signals-handling.patch of Package zsh - openSUSE Build Service ↩

3. osc, the Command Line Tool | User Guide ↩

4. zsh/code/3c7489: Improve process group handling in pipelines. ↩

[LangLangBart]

@junegunn Since the patch from openSUSE is based solely on the development version of the zsh¹² source code, I would suggest to keep the change for now, as future zsh releases will likely behave the same way.

Footnotes

1. zsh / Code / Commit [188c5c] ↩

2. zsh / Code / Commit [61610e] ↩

[junegunn]

@LangLangBart Thank you very much for the great analysis.

[LangLangBart]

It's possible the issue has already been fixed in the current master version.

Ref: #3863

If you'd like, you can try downloading the repository, running make, and checking the binary in the target folder.

[michaeltraxler]

Thanks a lot! Yes, after rebuilding fzf the crash is gone.
But the output in zsh with freshly installed .oh-my-zsh (also made a new account for this) is:
kill **input/output error
so, when I press TAB, it just adds input/output error to the current line and reprints
kill ** in a new line.
Ok, this might also be a bug in oh-my-zsh?

[LangLangBart]

Thanks a lot! Yes, after rebuilding fzf the crash is gone.

👍

But the output in zsh with freshly installed .oh-my-zsh (also made a new account for this) is:
kill **input/output error
so, when I press TAB, it just adds input/output error to the current line and reprints
kill ** in a new line.
Ok, this might also be a bug in oh-my-zsh?

Not able to reproduce with a minimal .zshrc file.

# my demo .zshrc

export ZSH="$HOME/.oh-my-zsh"
export PATH="$HOME/Developer/fzf/bin:$PATH"
ZSH_THEME="robbyrussell"

plugins=(fzf)

source $ZSH/oh-my-zsh.sh

What is the output for this command ?

bindkey | grep -F '^I'

Did you source the fzf completion.zsh file differently ?

Any other plugins installed? Try to disable them.

[michaeltraxler]

I made a new user and skipped oh-my-zsh and just did the recommended fzf install method:

git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
~/.fzf/install

copied the newly compiled fzf (without the segmentation fault) to ~/bin
and here I get the same error:
kill **input/output error

[michaeltraxler]

The output for the fzf-only user is:

l10% bindkey | grep -F '^I'
"^I" fzf-completion
"^[^I" self-insert-unmeta

For the oh-my-zsh user it is the same.

By the way: ctrl-r, ctrl-t and alt-c work always very well (also with original binary file).

[LangLangBart]

Can you try a zsh session without reading your .zshrc file ?

# new  session
zsh -f
source ~/.fzf/shell/completion.zsh
kill **
# issue still there ?

EDIT:

The next step would be to try to debug it.

zsh -f
source ~/.fzf/shell/completion.zsh
typeset -ft fzf-completion __fzf_defaults __fzf_comprun __fzf_extract_command _fzf_feed_fifo _fzf_complete _fzf_complete_kill _fzf_complete_kill_post 
kill **
# share verbose output …

[michaeltraxler]

I'm sorry, a part of this is due to my testing procedure...
If I test the following way, coming from the root user:

l10:~ # su - test3
kill **

If I use:

su -P - test3 -c 'zsh'

I get the input/output error.

But this still doesn't solve the original issue with the normal account.

[LangLangBart]

Can you try the debug snippet with the typeset -ft … command to enable xtrace in the functions of the completion.zsh file? It would be interesting to know which command causes the input/output error.

---

If I use:

su -P - test3 -c 'zsh'

The default version of su under under macOS doesn't provide the -P/--pty flag.

# default 'su' man page under macOS
man =(curl https://raw.githubusercontent.com/apple-oss-distributions/shell_cmds/main/su/su.1)

The util-linux formulae has the su command, but isn't supported on macOS¹.

Footnotes

1. util-linux — Homebrew Formulae ↩

[michaeltraxler]

Thanks for your help!

The first thing to do was to log on to the account without residual environment variables, which stay when using just
zsh -f.
So, I used
ssh localhost -t 'zsh -f'
to get a clean environment.

Then source ~/.fzf/shell/completion.zsh didn't work (no ctlr-r for example), I had to use
source ~/.fzf.zsh

The output with typeset ... is here:

l10% kill **+fzf-completion:1> local tokens cmd prefix trigger tail matches lbuf d_cmds
+fzf-completion:2> setopt localoptions noshwordsplit noksh_arrays noposixbuiltins
+fzf-completion:6> tokens=( kill '**' )
+fzf-completion:7> [ 2 -lt 1 ']'
+fzf-completion:12> cmd=+fzf-completion:12> __fzf_extract_command 'kill **'
+__fzf_extract_command:1> local token tokens
+__fzf_extract_command:2> tokens=( kill '**' )
+__fzf_extract_command:3> token=kill
+__fzf_extract_command:4> token=kill
+__fzf_extract_command:5> [[ "$token" -regex-match [[:alnum:]] && ! "$token" -regex-match "=" ]]
+__fzf_extract_command:6> echo kill
+__fzf_extract_command:7> return
+fzf-completion:12> cmd=kill
+fzf-completion:15> trigger='**'
+fzf-completion:16> [ -z '**' -a '*' '=' ' ' ']'
+fzf-completion:19> [[ 'kill **' = *kill\*\* ]]
+fzf-completion:24> lbuf='kill **'
+fzf-completion:25> tail='**'
+fzf-completion:28> [ 2 -gt 1 -a '**' '=' '**' ']'
+fzf-completion:29> d_cmds=( cd pushd rmdir )
+fzf-completion:31> [ -z '**' ']'
+fzf-completion:31> prefix=''
+fzf-completion:32> [[ '' = *\$\(* ]]
+fzf-completion:32> [[ '' = *\<\(* ]]
+fzf-completion:32> [[ '' = *\>\(* ]]
+fzf-completion:32> [[ '' = *:=* ]]
+fzf-completion:32> [[ '' = *`* ]]
+fzf-completion:35> [ -n '**' ']'
+fzf-completion:35> lbuf='kill '
+fzf-completion:37> eval 'type _fzf_complete_kill > /dev/null'
+(eval):1> type _fzf_complete_kill
+fzf-completion:38> prefix='' +fzf-completion:38> eval _fzf_complete_kill 'kill\ '
+(eval):1> _fzf_complete_kill 'kill '
+_fzf_complete_kill:1> _fzf_complete -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -- 'kill '
+_fzf_complete:1> setopt localoptions ksh_arrays
+_fzf_complete:3> local args rest str_arg i sep
+_fzf_complete:4> args=( -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -- 'kill ' )
+_fzf_complete:5> sep=''
+_fzf_complete:6> i=0
+_fzf_complete:7> [[ -m = -- ]]
+_fzf_complete:6> i=1
+_fzf_complete:7> [[ '--header-lines=1' = -- ]]
+_fzf_complete:6> i=2
+_fzf_complete:7> [[ --preview = -- ]]
+_fzf_complete:6> i=3
+_fzf_complete:7> [[ 'echo {}' = -- ]]
+_fzf_complete:6> i=4
+_fzf_complete:7> [[ --preview-window = -- ]]
+_fzf_complete:6> i=5
+_fzf_complete:7> [[ down:3:wrap = -- ]]
+_fzf_complete:6> i=6
+_fzf_complete:7> [[ --min-height = -- ]]
+_fzf_complete:6> i=7
+_fzf_complete:7> [[ 15 = -- ]]
+_fzf_complete:6> i=8
+_fzf_complete:7> [[ -- = -- ]]
+_fzf_complete:8> sep=8
+_fzf_complete:9> break
+_fzf_complete:12> [[ -n 8 ]]
+_fzf_complete:13> str_arg=''
+_fzf_complete:14> rest=( 'kill ' )
+_fzf_complete:15> args=( -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 )
+_fzf_complete:23> local fifo lbuf cmd matches post
+_fzf_complete:24> fifo=/tmp/fzf-complete-fifo-31493
+_fzf_complete:25> lbuf='kill '
+_fzf_complete:26> cmd=+_fzf_complete:26> __fzf_extract_command 'kill '
+__fzf_extract_command:1> local token tokens
+__fzf_extract_command:2> tokens=( kill )
+__fzf_extract_command:3> token=kill
+__fzf_extract_command:4> token=kill
+__fzf_extract_command:5> [[ "$token" -regex-match [[:alnum:]] && ! "$token" -regex-match "=" ]]
+__fzf_extract_command:6> echo kill
+__fzf_extract_command:7> return
+_fzf_complete:26> cmd=kill
+_fzf_complete:27> post=_fzf_complete_kill_post
+_fzf_complete:28> type _fzf_complete_kill_post
+_fzf_complete:30> _fzf_feed_fifo /tmp/fzf-complete-fifo-31493
+_fzf_feed_fifo:1> rm -f /tmp/fzf-complete-fifo-31493
+_fzf_feed_fifo:2> mkfifo /tmp/fzf-complete-fifo-31493
+_fzf_complete:31> matches=+_fzf_feed_fifo:3> cat
+_fzf_complete:31> matches=+_fzf_complete:34> _fzf_complete_kill_post
+_fzf_complete:32> FZF_DEFAULT_OPTS=+_fzf_complete:32> __fzf_defaults --reverse ' '
+__fzf_defaults:3> echo '--height 40% --bind=ctrl-z:ignore --reverse'
+_fzf_complete:31> matches=+_fzf_complete:34> tr '\n' ' '
+_fzf_complete_kill_post:1> awk '{print $2}'
+__fzf_defaults:4> cat ''
+__fzf_defaults:5> echo '  '
+_fzf_complete:32> FZF_DEFAULT_OPTS=$'--height 40% --bind=ctrl-z:ignore --reverse\n  ' FZF_DEFAULT_OPTS_FILE='' +_fzf_complete:32> __fzf_comprun kill -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -q ''
+__fzf_comprun:1> [[ "$(type _fzf_comprun 2>&1)" -regex-match function+__fzf_comprun:1> type _fzf_comprun
+__fzf_comprun:1> [[ "$(type _fzf_comprun 2>&1)" -regex-match function ]]
+__fzf_comprun:3> [ -n '' ']'
+__fzf_comprun:11> shift
+__fzf_comprun:12> fzf -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -q ''
input/output error
+_fzf_complete:31> matches=''
+_fzf_complete:35> [ -n '' ']'
+_fzf_complete:38> rm -f /tmp/fzf-complete-fifo-31493
+fzf-completion:39> zle reset-prompt

EDIT:
If I only use source ~/.fzf/shell/completion.zsh the output is the same...

[LangLangBart]

+__fzf_comprun:3> [ -n '' ']'
+__fzf_comprun:11> shift
+__fzf_comprun:12> fzf -m '--header-lines=1' --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -q ''
input/output error
+_fzf_complete:31> matches=''
+_fzf_complete:35> [ -n '' ']'

The __fzf_comprun function, along with its arguments, is invoked from the _fzf_complete function.

fzf/shell/completion.zsh

Lines 236 to 244 in cc2b214

	_fzf_feed_fifo "$fifo"
	matches=$(
	FZF_DEFAULT_OPTS=$(__fzf_defaults "--reverse" "${FZF_COMPLETION_OPTS-} $str_arg") \
	FZF_DEFAULT_OPTS_FILE='' \
	__fzf_comprun "$cmd" "${args[@]}" -q "${(Q)prefix}" < "$fifo" | $post | tr '\n' ' ')
	if [ -n "$matches" ]; then
	LBUFFER="$lbuf$matches"
	fi

fzf/shell/completion.zsh

Lines 107 to 121 in cc2b214

	__fzf_comprun() {
	if [[ "$(type _fzf_comprun 2>&1)" =~ function ]]; then
	_fzf_comprun "$@"
	elif [ -n "${TMUX_PANE-}" ] && { [ "${FZF_TMUX:-0}" != 0 ] || [ -n "${FZF_TMUX_OPTS-}" ]; }; then
	shift
	if [ -n "${FZF_TMUX_OPTS-}" ]; then
	fzf-tmux ${(Q)${(Z+n+)FZF_TMUX_OPTS}} -- "$@"
	else
	fzf-tmux -d ${FZF_TMUX_HEIGHT:-40%} -- "$@"
	fi
	else
	shift
	fzf "$@"
	fi
	}

This command with the input from < "$fifo" seems to cause the input/output error message, which I am still unable to reproduce on macOS.

FZF_DEFAULT_OPTS='--height 40% --bind=ctrl-z:ignore --reverse' \
  fzf -m --header-lines=1 --preview 'echo {}' --preview-window down:3:wrap --min-height 15 -q ''

Are you aware if the kill ** command worked in previous versions of fzf and only stopped working recently? If you're unsure, could you test this with older versions of the fzf binary to see if the problem persists? If the issue doesn't occur in older versions, it would be helpful to share the oldest working fzf release or even perform a bisect ¹² analysis on the fzf source code.

Footnotes

1. Git - git-bisect Documentation ↩

2. Git - Debugging with Git ↩

[michaeltraxler]

This is all so weird...
Again, some observations:
a new user, install only fzf => kill ** works in bash, but in zsh it completely blocks the shell, ctrl-z, ctrl-c not working. The following processes are running and stuck:

test4     4379  0.0  0.0   5612  4008 pts/82   TN   14:59   0:00 tr \n
test4     4382  0.0  0.0   9076  5820 pts/82   TN   14:59   0:00 awk {print $2}
test4     4384  0.0  0.0 1822044 6272 pts/82   TNl  14:59   0:00 fzf -m --header-lines=1 --preview echo {} --preview-window down:3:wrap --min-height 15 -q

A kill -9 4379 (the "tr" process) releases the shell.

Exactly the same behavior on a different Tumbleweed OS.
If I do the same on a openSUSE Leap 15.5 it works!

So, some external tools (which are newer on Tumbleweed) are doing something different?

[michaeltraxler]

The problem can be easily reproduced with a container:

The following if the build-file

FROM registry.opensuse.org/opensuse/tumbleweed:latest
RUN zypper ref
RUN zypper -vn dup -l
RUN zypper in -y fzf zsh awk 

If you write this to
opensuse_tumbleweed_fzf.txt

then this will make the container:
podman build -t opensuse_fzf -f opensuse_tumbleweed_fzf.txt
Then it can be executed:
podman run --rm -ti opensuse_fzf:latest /usr/bin/zsh
Inside the zsh one can enable fzf completion and keybindings:
source /usr/share/fzf/shell/key-bindings.zsh; source /usr/share/fzf/shell/completion.zsh
Then a
kill ** <TAB>
will show the problem.

[michaeltraxler]

I could find an older executable in a different Tumbleweed, 0.52.0.
This executable causes the same problem on a new tumbleweed.
On the Tumbleweed "Release: 20240507", it works somehow.
Not with "", but with kill<space><TAB>. Then it shows the processes in fzf.
With "" and it only shows all files recursively...

[LangLangBart]

will show the problem.

Thanks for the setup. I used docker and was able to reproduce it inside the container.

Note

I never manged to reproduce the issue on my normal macOS version outside of a Docker container. Even after trying with coreutils tools, the input/output error never occurred for me. It must be a platform-specific bug.

---

Minimal reproduction

Dockerfile

---

FROM registry.opensuse.org/opensuse/tumbleweed:latest
RUN zypper refresh
RUN zypper --verbose --non-interactive dist-upgrade --auto-agree-with-licenses
RUN zypper install --no-confirm  fzf awk git make go vim zsh

# Clone the fzf repository and build the binary
RUN git clone https://github.com/junegunn/fzf.git /fzf && \
    cd /fzf && \
    make && make install

ENV PATH="/fzf/bin:$PATH"

---

brew install --cask docker
docker build --tag opensuse_fzf --file Dockerfile .
docker run --rm --tty  --interactive opensuse_fzf:latest /usr/bin/zsh

run the function below

test_fail() {
  PS4='%B%F{0}+ %D{%T:%3.} %2N:%i%f%b '
  setopt localoptions xtrace verbose
  fifo=$(mktemp -u)
  mkfifo "$fifo"
  (
    echo "Boom" >"$fifo" &
  )
  matches=$(fzf <"$fifo")
  rm -f "$fifo"
}

test_fail

output

+ 03:24:33:752 test_fail:3 fifo=
+ 03:24:33:753 test_fail:3 mktemp -u
+ 03:24:33:752 test_fail:3 fifo=/tmp/tmp.yNSG6LtONw 
+ 03:24:33:756 test_fail:4 mkfifo /tmp/tmp.yNSG6LtONw
+ 03:24:33:761 test_fail:8 matches=
+ 03:24:33:762 test_fail:6 echo Boom
+ 03:24:33:762 test_fail:8 fzf
input/output error
+ 03:24:33:761 test_fail:8 matches='' 
+ 03:24:33:774 test_fail:9 rm -f /tmp/tmp.yNSG6LtONw

Workaround

• remove the sub shell and use a &| (disowned job)

--- a/shell/completion.zsh
+++ b/shell/completion.zsh
@@ -199,9 +199,9 @@ _fzf_dir_completion() {
 }
 
-_fzf_feed_fifo() (
+_fzf_feed_fifo() {
   command rm -f "$1"
   mkfifo "$1"
-  cat <&0 > "$1" &
-)
+  cat <&0 > "$1" &|
+}
 
 _fzf_complete() {

Evolution of the error message by fzf

up to but not including e8405f4

source /fzf/shell/completion.zsh
# Failed to read /dev/tty
kill **22R

since e8405f4 and up to but not including 83b6033

source /fzf/shell/completion.zsh
kill **22R

since 83b6033 and up to but not including 94c33ac

source /fzf/shell/completion.zsh
# panic: runtime error: invalid memory address or nil pointer dereference
# [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x556fc85c0583]
# 
# goroutine 1 [running]:
# main.exit(0x2, {0x0?, 0x0?})
#         /home/abuild/rpmbuild/BUILD/fzf-0.53.0/main.go:43 +0x23
# main.main()
#         /home/abuild/rpmbuild/BUILD/fzf-0.53.0/main.go:100 +0x3ea
kill **;22R

since 94c33ac and up to but not including 2326c74

source /fzf/shell/completion.zsh
kill **22R

since 2326c74

source /fzf/shell/completion.zsh
kill **input/output error

[junegunn]

@LangLangBart Thanks a lot for the repro. Interestingly, the bash version works fine.

[junegunn]

Workaround

• remove the sub shell and use a &| (disowned job)

I can confirm that it fixes the problem. Do you think we should do this?

[junegunn]

That's a reasonable approach.

Possibly related:
https://github.com/wavetermdev/waveterm/pull/608/files / wavetermdev/waveterm#630

But if I recall correctly, the reason I used a subshell in _fzf_feed_fifo was simply to suppress job control messages. Disowning has the same effect, and it fixes this problem, so there's no reason not to use the method. I'll apply the suggested workaround. Thanks.