sourcegraph / lsif-go Goto Github PK

View Code? Open in Web Editor NEW

114.0 63.0 20.0 1.94 MB

Language Server Indexing Format (LSIF) generator for Go

Home Page: https://lsif.dev/

License: MIT License

Go 99.39% Dockerfile 0.10% Shell 0.51%

lsif go lsif-indexer golang

lsif-go's Introduction

DEPRECATED: Use scip-go instead.

lsif-go's People

Stargazers

Watchers

Forkers

blanet krxt jankyboy phanirithvij jeremyquan isabella232 alidn olafurpg seanburke-wf ekmixon colinking jamydev sourcegraph-testing aychtang williamfzc tymofij inful olleman42 arielsrv

lsif-go's Issues

Separate test packages do not correctly link to non-test package

See ./internal/testdata/internal/shouldvisit/tests_separate/pkg_test.go. The foo identifier is not linked to the foo definition in the pkg package.

☂️ Support for generics

Go 1.18 with generics will be released this month (March 2022). We need to add support for generics (e.g. Go to Definition for type parameters in type signatures) so that people can continue using precise code intel when they bump their toolchain version. I'm not entirely sure how much work this will require; if it needs to be broken up, this can be used as an umbrella issue with various sub-issues organized in a TODO list (with - [ ]).

Investigate support for generics in Go 1.18

We should check if generics cause lsif-go to crash by testing it against existing package with generics.

Indexing CGo?

We're running into an issue with v1.7.5 because of CGo:

error: failed to index: failed to load packages: packages.Load: err: exit status 1: stderr: go build github.com/pganalyze/pg_query_go/v2/parser: build constraints exclude all Go files in /go/pkg/mod/github.com/pganalyze/pg_query_go/[email protected]/parser

Looks related to https://github.com/sourcegraph/lsif-go/pull/221/files which sets CGO_ENABLED=0 when loading packages. For now, we can lock to 1.7.4.

Load one dependency at a time for implementations

This would reduce memory usage and obviate the --skip-deps flag #212

Original motivation, OOMing in CI sourcegraph/sourcegraph#27211

Docker images not running go 1.16

Looking at the images on Docker Hub the latest one is f9c2d9cf3bdf which is running go 1.14.5 but before that image was even made the Dockerfile was updated in the repo to 1.16. Shouldn't the latest images on Docker Hub be running 1.16 as well?

`lsif-go --animation` hangs when listing dependencies on sourcegraph/sourcegraph

lsif-go commit: cde8136
sourcegraph/sourcegraph commit: 2d7f5b9a014a81a33df5812ede6f5cfa368b49ee

Delve goroutines

(dlv) goroutines -t 1000
  Goroutine 1 - User: /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:71 github.com/sourcegraph/lsif-go/internal/output.WithProgressParallel.func2 (0x102dd842c) [select]
	 0  0x0000000102c54b3c in runtime.gopark
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	 1  0x0000000102c65aac in runtime.selectgo
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/select.go:327
	 2  0x0000000102dd842c in github.com/sourcegraph/lsif-go/internal/output.WithProgressParallel.func2
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:71
	 3  0x0000000102dd8a08 in github.com/sourcegraph/lsif-go/internal/output.withTitleAnimated.func1
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:114
	 4  0x0000000102dd6754 in github.com/efritz/pentimento.PrintProgress
	     at /Users/chrismwendt/go/pkg/mod/github.com/efritz/[email protected]/printer.go:46
	 5  0x0000000102dd87d8 in github.com/sourcegraph/lsif-go/internal/output.withTitleAnimated
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:109
	 6  0x0000000102dd8504 in github.com/sourcegraph/lsif-go/internal/output.withTitle
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:89
	 7  0x0000000102dd8340 in github.com/sourcegraph/lsif-go/internal/output.WithProgressParallel
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:69
	 8  0x0000000102dd8268 in github.com/sourcegraph/lsif-go/internal/output.WithProgress
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:56
	 9  0x0000000102f080f4 in github.com/sourcegraph/lsif-go/internal/gomod.ListDependencies
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/gomod/dependencies.go:73
	10  0x000000010304b768 in main.mainErr
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/cmd/lsif-go/main.go:59
	11  0x000000010304b3a0 in main.main
	     at /Users/chrismwendt/github.com/sourcegraph/lsif-go/cmd/lsif-go/main.go:21
	12  0x0000000102c546e4 in runtime.main
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:255
	13  0x0000000102c83c64 in runtime.goexit
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 2 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [force gc (idle) 456959h19m52.782832459s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c549c0 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c549c0 in runtime.forcegchelper
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:306
	3  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 3 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC sweep wait]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c402b0 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c402b0 in runtime.bgsweep
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgcsweep.go:182
	3  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 4 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC scavenge wait]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c3e570 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c3e570 in runtime.bgscavenge
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgcscavenge.go:314
	3  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 5 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [finalizer wait 456959h19m52.785584459s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c35028 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c35028 in runtime.runfinq
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mfinal.go:177
	3  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 10 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:56 sync.runtime_Semacquire (0x102c803d8) [semacquire 456959h21m57.201825084s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c66b08 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c66b08 in runtime.semacquire1
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:144
	3  0x0000000102c803d8 in sync.runtime_Semacquire
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:56
	4  0x0000000102c9b624 in sync.(*WaitGroup).Wait
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/sync/waitgroup.go:130
	5  0x0000000102f096cc in github.com/sourcegraph/lsif-go/internal/gomod.resolveImportPaths
	    at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/gomod/dependencies.go:264
	6  0x0000000102f08788 in github.com/sourcegraph/lsif-go/internal/gomod.ListDependencies.func1
	    at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/gomod/dependencies.go:69
	7  0x0000000102dd8044 in github.com/sourcegraph/lsif-go/internal/parallel.Run.func1
	    at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/parallel/parallel.go:24
	8  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 29 - User: /Users/chrismwendt/go/pkg/mod/github.com/efritz/[email protected]/printer.go:112 github.com/efritz/pentimento.(*Printer).Refresh.func1 (0x102dd6da8) [chan receive 456959h21m57.203261084s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c234a4 in runtime.chanrecv
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/chan.go:576
	2  0x0000000102c22e98 in runtime.chanrecv2
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/chan.go:444
	3  0x0000000102dd6da8 in github.com/efritz/pentimento.(*Printer).Refresh.func1
	    at /Users/chrismwendt/go/pkg/mod/github.com/efritz/[email protected]/printer.go:112
	4  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 35 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:56 sync.runtime_Semacquire (0x102c803d8) [semacquire 456959h21m57.204448084s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c66b08 in runtime.goparkunlock
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:372
	2  0x0000000102c66b08 in runtime.semacquire1
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:144
	3  0x0000000102c803d8 in sync.runtime_Semacquire
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/sema.go:56
	4  0x0000000102c9b624 in sync.(*WaitGroup).Wait
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/sync/waitgroup.go:130
	5  0x0000000102dd8488 in github.com/sourcegraph/lsif-go/internal/output.WithProgressParallel.func1
	    at /Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/output/output.go:65
	6  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 36 - User: /Users/chrismwendt/go/pkg/mod/github.com/efritz/[email protected]/printer.go:112 github.com/efritz/pentimento.(*Printer).Refresh.func1 (0x102dd6da8) [chan receive]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c234a4 in runtime.chanrecv
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/chan.go:576
	2  0x0000000102c22e98 in runtime.chanrecv2
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/chan.go:444
	3  0x0000000102dd6da8 in github.com/efritz/pentimento.(*Printer).Refresh.func1
	    at /Users/chrismwendt/go/pkg/mod/github.com/efritz/[email protected]/printer.go:112
	4  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 55 - User: :0 ??? (0x18e1385a0) (thread 6519685) [chan receive 456959h21m52.808257042s]
	0  0x000000018e1385a0 in ???
	    at ?:-1
	1  0x0000000102c83a8c in runtime.asmcgocall
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:935
	    error: protocol error E08 during memory read for packet $mffffffffffc00e20,8
	(truncated)
  Goroutine 82 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle) 456959h21m57.209622084s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 83 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle)]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 84 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle)]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 85 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle) 456959h21m56.325406584s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 98 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle)]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 99 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle)]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 114 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle) 456959h21m56.890885542s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 115 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367 runtime.gopark (0x102c54b3c) [GC worker (idle) 456959h21m55.663103792s]
	0  0x0000000102c54b3c in runtime.gopark
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	1  0x0000000102c38034 in runtime.gcBgMarkWorker
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/mgc.go:1200
	2  0x0000000102c83c64 in runtime.goexit
	    at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
  Goroutine 559 - User: /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/netpoll.go:229 internal/poll.runtime_pollWait (0x102c7e9f4) [IO wait 456959h21m52.815328042s]
	 0  0x0000000102c54b3c in runtime.gopark
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/proc.go:367
	 1  0x0000000102c4d2d8 in runtime.netpollblock
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/netpoll.go:445
	 2  0x0000000102c7e9f4 in internal/poll.runtime_pollWait
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/netpoll.go:229
	 3  0x0000000102ce03a8 in internal/poll.(*pollDesc).wait
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/internal/poll/fd_poll_runtime.go:84
	 4  0x0000000102ce148c in internal/poll.(*pollDesc).waitRead
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/internal/poll/fd_poll_runtime.go:89
	 5  0x0000000102ce148c in internal/poll.(*FD).Read
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/internal/poll/fd_unix.go:167
	 6  0x0000000102ce7524 in os.(*File).read
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/os/file_posix.go:32
	 7  0x0000000102ce7524 in os.(*File).Read
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/os/file.go:119
	 8  0x0000000102d16b24 in bytes.(*Buffer).ReadFrom
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/bytes/buffer.go:204
	 9  0x0000000102cbf5a4 in io.copyBuffer
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/io/io.go:409
	10  0x0000000102db6514 in io.Copy
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/io/io.go:382
	11  0x0000000102db6514 in os/exec.(*Cmd).writerDescriptor.func1
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/os/exec/exec.go:311
	12  0x0000000102db6f58 in os/exec.(*Cmd).Start.func1
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/os/exec/exec.go:441
	13  0x0000000102c83c64 in runtime.goexit
	     at /opt/homebrew/Cellar/go/1.17/libexec/src/runtime/asm_arm64.s:1133
[19 goroutines]

I don't know why there are no goroutines in that block. It appears as though it's in deadlock.

However, I noticed there's a frame in os/exec.(*Cmd).Start.func1 and deeper in io.Copy waiting for stdout from a subprocessed, so I checked pstree for child processes and found:

 |   |       \-+= 29322 chrismwendt /Users/chrismwendt/github.com/sourcegraph/lsif-go/lsif-go
 |   |         \-+- 29730 chrismwendt git ls-remote git+ssh://git.apache.org/thrift
 |   |           \--- 29731 chrismwendt /usr/bin/ssh -o SendEnv=GIT_PROTOCOL git.apache.org git-upload-pack '/thrift'

I ran it myself:

$ git ls-remote git+ssh://git.apache.org/thrift
The authenticity of host 'git.apache.org (52.202.80.70)' can't be established.
ECDSA key fingerprint is SHA256:EzwREkEsDEDLPU4dybKtxgPuXW8Gt/P5Qapo1VheWqw.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

And it hung. When I typed yes, it said "permission denied".

I'm not sure why this is hanging for me when recent lsif-go runs in CI are working https://github.com/sourcegraph/sourcegraph/runs/5252625222?check_suite_focus=true#step:4:5

What about a golang version of LSIF server?

I read some blogs for LSIF protocol recently, but I only found LSIF servers written in typescript.
Is there any plan to write a golang version of server to serve the LSIF dumps?

doc results include //go: pragmas

Documentation in definition results show //go: pragmas. For example, the result for https://sourcegraph.com/github.com/golang/go@master/-/blob/src/runtime/stubs.go#L109:6 includes go:noescape at the end. Package go/ast has removed such lines from CommentGroup nodes since Go 1.15. godoc, gopls, and other tools do not show these directives.

Update to go 1.17

Disable go mod download step

Some users (may be bazel specific but I'm not sure) wrap some of the go tooling in scripts for additional caching to support monorepos. As a pre-indexing step, we shell out and perform go mod download when there is a go.mod file in the directory. We should be able to disable this step.

Fix range for package import strings

Highlights are strange when hovering over a package import string on Sourcegraph. We should ensure that we capture the entire string for highlighting.

bug: no J2D or find-refs of switch-case blocks over an object's type

Code like the following:

switch v := obj.(type) {
  case A: f1(v, v)
  case B: f2(v, v)
}

does not get indexed properly because when processing definitions, the definition for v has a nil associated def object, since there's a different def object for the v in each branch. Then when we encounter each use of v, there is an associated def object but since it's not indexed in our definitions we only output a hover result.

Discussion:

If there's a way to know that this is why the def object is nil during the definitions pass, that would likely be the right way to solve.
What LSIF should we actually output here? IMO:
- Each case branch should have a different hover result cause they have different types.
- The def result is canonical.
- I think the reference result should point to all uses in every branch. Kind of weird to have a reference return results with different types, but I think the experience would be weirder if you hover over one of these and only the symbols in that branch highlight. Also the definition would need to have everything as a reference result anyway.
- Open question: what should go in the hover result when hovering the definition?

Therefore the resulting LSIF structure should be:

vdef -> resultSet1
resultSet1 -> hoverResult1
resultSet1 -> resultSetShared
resultSetShared -> defResult1
resultSetShared -> refResult1
defResult1 -> vdef
refResult1 -> vdef
vref1 -> resultSet2
resultSet2 -> hoverResult2
resultSet2 -> resultSetShared
refResult1 -> vref1
vref2 -> resuiltSet2
refResult1 -> vref2
vref3 -> resultSet3
resultSet3 -> hoverResult3
resultSet3 -> resultSetShared
refResult1 -> vref3
vref4 -> resuiltSet3
refResult1 -> vref4

Thoughts?

Also how does the worker process this graph?

Different index time described in benchmark file and docs

Hi,
I got quite confused when see these two data:

In https://docs.sourcegraph.com/code_intelligence/explanations/precise_code_intelligence,
the index time on aws-sdk-go is 8.20m

, while in https://github.com/sourcegraph/lsif-go/blob/master/BENCHMARK.md,
the index time on monorepo-1 is 23.8s.

Per my understanding.. in these two table, "index time" both indicate processing time of the indexer(which is lisf-go).
According to script that generates source code, monorepo-1 should contain an extra copy of services dir, but these two data seem far too different.

Please let me know if I got it incorrect.
Many thanks,
-John

lsif-go handles `package mypackage` and `import "mypackage"` in unexpected ways

When you click on something like this:

It just takes you to the same location you clicked! Instead of going to the package.

add flags for runtime output control

should be able to disable any sourcegraph extensions
should be able to skip API docs (for example PR branches and what not)
might want to skip implementations

Maybe other fine grained control.

Why are some files missing from `package.Syntax`?

For some reason, some identifiers are present in pkg.TypesInfo, but their position does not appear in pkg.Syntax. Example: https://github.com/yuin/gopher-lua/blob/master/parse/parser.go

Maybe it has something to do with yacc and generated files?

Reported in Slack, introduced in #199, fixed in #210

cc @tjdevries

No LSIF data dumped for test files?

LSIF dumps for test files should be generated and now they are missing.

Maybe Using golang.org/x/tools/go/packages is not the best way to dump LSIF considering generation speed and idempotent. But for now I have no idea how to improve this either. ;-)

Improve handling of repositories with no go.mod file

The indexer currently requires a go.mod file to be present in the repository. It does not need to be at the root (the --moduleRoot flag allows a subdirectory). If there is none, however, dependency resolution will not work.

Although Go packages without a module are increasingly rare, they still occur in the wild. One way to handle such repositories better would be to create a temporary go.mod file during indexing: Using the remote URL and/or canonical import path comments in the source files, the indexer could create a stub go.mod file and populate it (as by go mod tidy) before attempting to index. To avoid polluting a local build, this would probably need to be virtualized through the build package's VFS hooks or maybe a temp directory wrapper.

The first step is to figure out whether this situation happens often enough to be worth fixing.

Provide Releases

For generating lsif, it'd be nice if we could include a binary instead of an entire go install in our CI image.

inVs is null

In the dump of gorilla/mux, there is an edge

{
  id: '12458',
  type: 'edge',
  label: 'contains',
  outV: '96',
  inVs: null
}

this is invalid, inVs should be an empty array or this edge should not be included.

No j2d for defs with no uses

Description

The textDocument/definition vertex (not 100% sure about the type) is not emitted for defs with no uses inside the same package.

Examples:

Observation

Actual

Expected

Incorrect Go to Definition for io.Writer from sourcegraph fork

At https://sourcegraph.com/github.com/C-EO/sourcegraph@f6e8b61465c2e0474011b62819898179d2f36a90/-/blob/cmd/server/shared/copy.go?L73, the code intel popover for Writer in io.Writer says that the code intel is "Precise". However, if I click Go to Definition, it takes me to https://sourcegraph.com/github.com/golang/go@
63b968f4f86f4c23ce92b7ac2feda4fc7ca17c8e/-/blob/src/io/multi_test.go?L84:3&subtree=true, which is not the definition of Go's io.Writer.

error: failed to infer module name: unrecognized import path ""

Hi, I am a new user to lsif-go, I want to use code intelligence in sourcegraph instead of search-based code navigation.

So I am here and run 'lsif-go' in my git repository, but it reports an error:

error: failed to infer module name: unrecognized import path ""

When I read the code, apparently this error is caused by golang.org/x/tools/go/vcs.vcs.RepoRootForImportPath(...).

I used private git platform, and the repo's module is like git.code.xxx.com/group/repo, function vcs.RepoRootForImportPath(...) reports error: unrecognized import path "git.code.xxx.com/xxxx/yyyy".

So what should I do to work around this?

Unable to index dependancies for kubevirt

Got the following errors:

Indexing project 
---------------------
✔ Loading packages... Done (10.28s)

error: index: loadPackages: packages.Load: go [-e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 2: go: downloading k8s.io/client-go v0.16.4
go: downloading k8s.io/apimachinery v0.16.4
go: downloading k8s.io/api v0.16.4

Indexing project vendor/gopkg.in/yaml.v2
---------------------

error: failed to list modules: exit status 1
go: updates to go.sum needed, disabled by -mod=readonly

Processor failing during CI/CD

Customer reported as well, filed separately here: https://github.com/sourcegraph/customer/issues/216

The processor is failing during CI/CD; log are linked in the customer issue for privacy. Reverting to 1.2 resolves the issue. Top error:

panic: interface conversion: types.Type is *types.Struct, not *types.Named

@Strum355 reported that he found a public repo demonstrating the issue as well.

Unrecognized import path

We're seeing errors similar to error: unrecognized import path "phabricator.example.com/source/project" when running over code with non-public dependency. Currently waiting for additional customer input so we can find out the common way to reproduce and fix.

Uncommitted API docs changes when running tests

@slimsag Is this for golden tests? Checking generated code into the repo adds a bit of overhead while developing other features. Perhaps the API docs tests could generate in a tmp directory and perform assertions on that?

remote implementations are not linked for method calls

I'll probably work on this next, but it is not a blocking issue to ship implementations, IMO

Some example tests:

	t.Run("remote_implementations: finuds implementations in signature", func(t *testing.T) {
		r := mustRange(t, w, "file://"+filepath.Join(projectRoot, "implementations_remote.go"), 10, 38)

		monikers := findMonikersByRangeOrReferenceResultID(w, r.ID)
		t.Fatalf("%+v\n", monikers)
	})

	t.Run("remote_implementations: finds implementations on method call", func(t *testing.T) {
		r := mustRange(t, w, "file://"+filepath.Join(projectRoot, "implementations_remote.go"), 11, 12)

		// assertRanges(
		// 	t,
		// 	w,
		// 	findImplementationRangesByRangeOrResultSetID(w, r.ID),
		// 	[]string{"4:5-4:21"},
		// 	"ResponseWriter Implementation (from signature)",
		// )

		monikers := findMonikersByRangeOrReferenceResultID(w, r.ID)
		t.Fatalf("%+v\n", monikers)
	})

No LSIF info for uses found before defs

Description

Missing textDocument/references vertex for uses (aka references) in a file found before the file contains that definition.

This regression is introduced in ba053fa.

e.g.

Route is used in mux.go but its definition is only found in the file route.go which is processed after mux.go

Package: mux
	...
	File: .../mux.go
	...
	File: .../route.go
...

Observation

Actual

Expected

Emit 'Deprecated' Diagnostic messages

The Go wiki outlines a standard for marking constructs as Deprecated. We should emit these as LSIF Diagnostics to support querying for usages of deprecated APIs.

Follow-on: what else can/should we emit? Are there existing tools we can utilize (cursed raw text output aside)? gopls/govet/staticcheck/golanglint-ci

Buildkite job

Add a buildkite job that runs the unit tests for this repository.

lsif-go fails for mysterious reasons on sourcegraph/resource-estimator

What currently happens:

$ git clone https://github.com/sourcegraph/resource-estimator
$ cd resource-estimator
$ lsif-go
error: fetching dependencies: exit status 1

cc @sourcegraph/code-intel

Cannot run lsif-go because of dependency issue with etcd-io/bbolt

I am following the instructions listed here to run this job:
https://docs.gitlab.com/ee/user/project/code_intelligence.html

go get works fine but running the binary runs into problems because it looks like lsif-go downloads a few more modules.

go: github.com/coreos/etcd/etcdctl/ctlv2/command imports
github.com/coreos/bbolt: github.com/coreos/[email protected]: parsing go.mod:
module declares its path as: go.etcd.io/bbolt
but was required as: github.com/coreos/bbolt

there is a mismatch in the path of bbolt in the gomod and how it is imported in code.

Relevant issue here:
etcd-io/bbolt#224

I imagine when they fix it, lsif-go will also need to be updated to reference the latest etcd, which imports bbolt.

go get -u github.com/sourcegraph/lsif-go/cmd/lsif-go
go: downloading github.com/sourcegraph/lsif-go v0.8.0
go: found github.com/sourcegraph/lsif-go/cmd/lsif-go in github.com/sourcegraph/lsif-go v0.8.0
go: downloading github.com/alecthomas/kingpin v2.2.6+incompatible
go: downloading github.com/slimsag/godocmd v0.0.0-20161025000126-a1005ad29fe3
go: downloading golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: golang.org/x/tools upgrade => v0.0.0-20200713235242-6acd2ab80ede
go: github.com/alecthomas/template upgrade => v0.0.0-20190718012654-fb15b899a751
go: github.com/alecthomas/units upgrade => v0.0.0-20190924025748-f65c72e2690d
go: downloading golang.org/x/tools v0.0.0-20200713235242-6acd2ab80ede
go: downloading golang.org/x/mod v0.3.0
go: golang.org/x/xerrors upgrade => v0.0.0-20191204190536-9bdfabe68543
go: golang.org/x/mod upgrade => v0.3.0
$ $GOPATH/bin/lsif-go
.
error: index: load packages: go [-e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 1: go: finding module for package github.com/hashicorp/consul/api
go: finding module for package google.golang.org/grpc/peer
go: finding module for package github.com/jonboulle/clockwork
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package golang.org/x/time/rate
go: finding module for package github.com/afex/hystrix-go/hystrix
go: finding module for package github.com/coreos/bbolt
go: finding module for package gopkg.in/tomb.v2
go: finding module for package github.com/urfave/cli
go: finding module for package github.com/casbin/casbin
go: downloading github.com/jonboulle/clockwork v0.2.0
go: downloading github.com/coreos/bbolt v1.3.5
go: downloading github.com/casbin/casbin v1.9.1
go: downloading github.com/urfave/cli v1.22.4
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/hashicorp/consul/api v1.5.0
go: downloading golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
go: downloading github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5
go: downloading github.com/hashicorp/consul v1.8.0
go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
go: downloading google.golang.org/grpc v1.30.0
go: finding module for package github.com/VividCortex/gohistogram
go: finding module for package google.golang.org/grpc/metadata
go: finding module for package github.com/openzipkin/zipkin-go
go: finding module for package github.com/streadway/handy/breaker
go: finding module for package github.com/opentracing/opentracing-go
go: finding module for package google.golang.org/grpc/health
go: finding module for package go.etcd.io/etcd/clientv3
go: downloading github.com/VividCortex/gohistogram v1.0.0
go: downloading github.com/openzipkin/zipkin-go v0.2.2
go: finding module for package github.com/hudl/fargo
go: downloading github.com/opentracing/opentracing-go v1.2.0
go: downloading github.com/streadway/handy v0.0.0-20200128134331-0f66f006fb2e
go: finding module for package github.com/openzipkin-contrib/zipkin-go-opentracing
go: downloading go.etcd.io/etcd v0.5.0-alpha.5
go: downloading github.com/hudl/fargo v1.3.0
go: finding module for package go.etcd.io/etcd/pkg/transport
go: finding module for package github.com/gorilla/mux
go: finding module for package google.golang.org/grpc/health/grpc_health_v1
go: downloading github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5
go: finding module for package google.golang.org/grpc/codes
go: finding module for package github.com/shurcooL/httpfs/vfsutil
go: downloading github.com/gorilla/mux v1.7.4
go: finding module for package google.golang.org/grpc
go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
go: finding module for package go.opencensus.io/trace/propagation
go: finding module for package cloud.google.com/go/pubsub
go: downloading go.etcd.io/etcd v3.3.22+incompatible
go: finding module for package github.com/juju/ansiterm
go: downloading cloud.google.com/go v0.61.0
go: finding module for package go.uber.org/zap
go: downloading go.opencensus.io v0.22.4
go: downloading cloud.google.com/go/pubsub v1.5.0
go: downloading go.uber.org/zap v1.15.0
go: downloading github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a
go: finding module for package cloud.google.com/go/datastore
go: downloading cloud.google.com/go/datastore v1.2.0
go: finding module for package cloud.google.com/go/bigquery
go: downloading cloud.google.com/go/bigquery v1.9.0
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/runtime
go: finding module for package github.com/coreos/pkg/capnslog
go: finding module for package go.opencensus.io/plugin/ochttp
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.14.6
go: downloading github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.2.0
go: finding module for package go.opencensus.io/plugin/ochttp/propagation/b3
go: finding module for package github.com/performancecopilot/speed
go: downloading github.com/performancecopilot/speed v1.0.0
go: finding module for package github.com/godbus/dbus
go: downloading github.com/godbus/dbus v4.1.0+incompatible
go: downloading github.com/performancecopilot/speed v3.0.0+incompatible
go: finding module for package sourcegraph.com/sourcegraph/appdash
go: finding module for package github.com/samuel/go-zookeeper/zk
go: downloading sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
go: downloading github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da
go: finding module for package github.com/streadway/amqp
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/utilities
go: downloading github.com/streadway/amqp v1.0.0
go: finding module for package google.golang.org/grpc/transport
go: finding module for package github.com/aws/aws-sdk-go-v2/service/cloudwatch/cloudwatchiface
go: downloading github.com/aws/aws-sdk-go-v2 v0.23.0
go: finding module for package github.com/aws/aws-sdk-go-v2/service/cloudwatch
go: finding module for package google.golang.org/grpc/keepalive
go: finding module for package google.golang.org/api/googleapi
go: downloading google.golang.org/api v0.29.0
go: finding module for package github.com/andreiavrammsd/validator
go: downloading github.com/andreiavrammsd/validator v9.31.0+incompatible
go: finding module for package github.com/xiang90/probing
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: finding module for package github.com/nats-io/go-nats
go: downloading github.com/nats-io/go-nats v1.8.1
go: finding module for package github.com/opentracing/opentracing-go/mocktracer
go: finding module for package github.com/aws/aws-sdk-go/service/cloudwatch
go: downloading github.com/aws/aws-sdk-go v1.33.5
go: finding module for package github.com/sony/gobreaker
go: finding module for package github.com/tmc/grpc-websocket-proxy/wsproxy
go: downloading github.com/sony/gobreaker v0.4.1
go: finding module for package github.com/apache/thrift/lib/go/thrift
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966
go: downloading github.com/apache/thrift v0.13.0
go: finding module for package github.com/lightstep/lightstep-tracer-go
go: downloading github.com/lightstep/lightstep-tracer-go v0.20.0
go: finding module for package gopkg.in/cheggaaa/pb.v1
go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.28
go: finding module for package gopkg.in/ini.v1
go: finding module for package github.com/soheilhy/cmux
go: finding module for package github.com/codegangsta/cli
go: downloading gopkg.in/ini.v1 v1.57.0
go: downloading github.com/soheilhy/cmux v0.1.4
go: downloading github.com/codegangsta/cli v1.22.4
go: finding module for package github.com/shurcooL/go/importgraphutil
go: finding module for package github.com/pborman/getopt/v2
go: downloading github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636
go: finding module for package google.golang.org/grpc/credentials
go: downloading github.com/pborman/getopt v0.0.0-20190409184431-ee0cd42419d3
go: finding module for package github.com/olekukonko/tablewriter
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: finding module for package github.com/google/btree
go: downloading github.com/google/btree v1.0.0
go: finding module for package github.com/neelance/sourcemap
go: downloading github.com/neelance/sourcemap v0.0.0-20200213170602-2833bce08e4c
go: finding module for package github.com/goji/param
go: downloading github.com/goji/param v0.0.0-20160927210335-d7f49fd7d1ed
go: finding module for package google.golang.org/grpc/grpclog
go: finding module for package go.opencensus.io/trace
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/badgerodon/collections
go: finding module for package github.com/pact-foundation/pact-go/dsl
go: finding module for package github.com/pborman/uuid
go: downloading github.com/pact-foundation/pact-go v1.4.3
go: downloading github.com/pborman/uuid v1.2.0
go: downloading github.com/badgerodon/collections v0.0.0-20130729185459-604e922904d3
go: finding module for package google.golang.org/api/option
go: finding module for package go.etcd.io/etcd/client
go: finding module for package github.com/pquerna/otp/totp
go: finding module for package github.com/aws/aws-sdk-go-v2/aws
go: downloading github.com/pquerna/otp v1.2.0
go: finding module for package github.com/influxdata/influxdb/client/v2
go: downloading github.com/influxdata/influxdb v1.8.1
go: finding module for package google.golang.org/grpc/status
go: finding module for package github.com/neelance/astrewrite
go: downloading github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
go: finding module for package github.com/nats-io/gnatsd/server
go: downloading github.com/nats-io/gnatsd v1.4.1
go: finding module for package github.com/juju/retry
go: downloading github.com/juju/retry v0.0.0-20180821225755-9058e192b216
go: finding module for package github.com/ghodss/yaml
go: downloading github.com/ghodss/yaml v1.0.0
go: finding module for package github.com/juju/clock
go: downloading github.com/juju/clock v0.0.0-20190205081909-9c5c9712527c
go: finding module for package github.com/openzipkin/zipkin-go/reporter/http
go: finding module for package cloud.google.com/go/bigtable
go: downloading cloud.google.com/go/bigtable v1.4.0
go: finding module for package github.com/opentracing/opentracing-go/ext
go: finding module for package github.com/mcuadros/go-gin-prometheus
go: downloading github.com/mcuadros/go-gin-prometheus v0.1.0
go: finding module for package github.com/aws/aws-sdk-go/service/cloudwatch/cloudwatchiface
go: finding module for package github.com/juju/utils
go: downloading github.com/juju/utils v0.0.0-20200604140309-9d78121a29e0
go: finding module for package github.com/golang/groupcache/lru
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: finding module for package github.com/bgentry/speakeasy
go: finding module for package go.opencensus.io/plugin/ochttp/propagation/tracecontext
go: downloading github.com/bgentry/speakeasy v0.1.0
go: finding module for package github.com/aws/aws-sdk-go/aws
go: finding module for package github.com/juju/version
go: downloading github.com/juju/version v0.0.0-20191219164919-81c1be00b9a6
go: finding module for package github.com/oklog/oklog/pkg/group
go: downloading github.com/oklog/oklog v0.3.2
go: finding module for package sourcegraph.com/sourcegraph/appdash/opentracing
go: finding module for package github.com/casbin/casbin/persist/file-adapter
go: finding module for package github.com/openzipkin/zipkin-go/model
go: finding module for package github.com/ansel1/merry
go: downloading github.com/ansel1/merry v1.5.1
go: finding module for package github.com/coreos/pkg/dlopen
go: finding module for package github.com/openzipkin/zipkin-go/propagation/b3
go: finding module for package github.com/openzipkin/zipkin-go/reporter/recorder
go: found github.com/coreos/pkg/capnslog in github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f
go: found google.golang.org/grpc/credentials in google.golang.org/grpc v1.30.0
go: found github.com/ghodss/yaml in github.com/ghodss/yaml v1.0.0
go: found github.com/jonboulle/clockwork in github.com/jonboulle/clockwork v0.2.0
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/grpc-ecosystem/grpc-gateway/runtime in github.com/grpc-ecosystem/grpc-gateway v1.14.6
go: found github.com/soheilhy/cmux in github.com/soheilhy/cmux v0.1.4
go: found github.com/tmc/grpc-websocket-proxy/wsproxy in github.com/tmc/grpc-websocket-proxy v0.0.0-20200427203606-3cfed13b9966
go: found github.com/urfave/cli in github.com/urfave/cli v1.22.4
go: found github.com/bgentry/speakeasy in github.com/bgentry/speakeasy v0.1.0
go: found github.com/coreos/bbolt in github.com/coreos/bbolt v1.3.5
go: found github.com/olekukonko/tablewriter in github.com/olekukonko/tablewriter v0.0.4
go: found golang.org/x/time/rate in golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
go: found gopkg.in/cheggaaa/pb.v1 in gopkg.in/cheggaaa/pb.v1 v1.0.28
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.2.0
go: found go.uber.org/zap in go.uber.org/zap v1.15.0
go: found github.com/google/btree in github.com/google/btree v1.0.0
go: found github.com/golang/groupcache/lru in github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: found github.com/xiang90/probing in github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: found github.com/godbus/dbus in github.com/godbus/dbus v4.1.0+incompatible
go: found gopkg.in/tomb.v2 in gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
go: found github.com/casbin/casbin in github.com/casbin/casbin v1.9.1
go: found github.com/afex/hystrix-go/hystrix in github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5
go: found github.com/sony/gobreaker in github.com/sony/gobreaker v0.4.1
go: found github.com/streadway/handy/breaker in github.com/streadway/handy v0.0.0-20200128134331-0f66f006fb2e
go: found github.com/apache/thrift/lib/go/thrift in github.com/apache/thrift v0.13.0
go: found github.com/lightstep/lightstep-tracer-go in github.com/lightstep/lightstep-tracer-go v0.20.0
go: found github.com/opentracing/opentracing-go in github.com/opentracing/opentracing-go v1.2.0
go: found github.com/openzipkin-contrib/zipkin-go-opentracing in github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5
go: found github.com/openzipkin/zipkin-go in github.com/openzipkin/zipkin-go v0.2.2
go: found sourcegraph.com/sourcegraph/appdash in sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
go: found github.com/oklog/oklog/pkg/group in github.com/oklog/oklog v0.3.2
go: found github.com/gorilla/mux in github.com/gorilla/mux v1.7.4
go: found github.com/hashicorp/consul/api in github.com/hashicorp/consul/api v1.5.0
go: found github.com/pborman/uuid in github.com/pborman/uuid v1.2.0
go: found github.com/nats-io/go-nats in github.com/nats-io/go-nats v1.8.1
go: found github.com/aws/aws-sdk-go/aws in github.com/aws/aws-sdk-go v1.33.5
go: found github.com/aws/aws-sdk-go-v2/aws in github.com/aws/aws-sdk-go-v2 v0.23.0
go: found github.com/VividCortex/gohistogram in github.com/VividCortex/gohistogram v1.0.0
go: found github.com/influxdata/influxdb/client/v2 in github.com/influxdata/influxdb v1.8.1
go: found github.com/performancecopilot/speed in github.com/performancecopilot/speed v3.0.0+incompatible
go: found go.etcd.io/etcd/client in go.etcd.io/etcd v3.3.22+incompatible
go: found github.com/hudl/fargo in github.com/hudl/fargo v1.3.0
go: found github.com/samuel/go-zookeeper/zk in github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da
go: found go.opencensus.io/plugin/ochttp in go.opencensus.io v0.22.4
go: found github.com/streadway/amqp in github.com/streadway/amqp v1.0.0
go: found github.com/badgerodon/collections in github.com/badgerodon/collections v0.0.0-20130729185459-604e922904d3
go: found github.com/neelance/sourcemap in github.com/neelance/sourcemap v0.0.0-20200213170602-2833bce08e4c
go: found github.com/shurcooL/httpfs/vfsutil in github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
go: found github.com/neelance/astrewrite in github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
go: found gopkg.in/ini.v1 in gopkg.in/ini.v1 v1.57.0
go: found github.com/juju/ansiterm in github.com/juju/ansiterm v0.0.0-20180109212912-720a0952cc2a
go: found github.com/juju/clock in github.com/juju/clock v0.0.0-20190205081909-9c5c9712527c
go: found github.com/juju/retry in github.com/juju/retry v0.0.0-20180821225755-9058e192b216
go: found github.com/juju/utils in github.com/juju/utils v0.0.0-20200604140309-9d78121a29e0
go: found github.com/juju/version in github.com/juju/version v0.0.0-20191219164919-81c1be00b9a6
go: found github.com/pquerna/otp/totp in github.com/pquerna/otp v1.2.0
go: found google.golang.org/api/option in google.golang.org/api v0.29.0
go: found cloud.google.com/go/bigquery in cloud.google.com/go/bigquery v1.9.0
go: found cloud.google.com/go/bigtable in cloud.google.com/go/bigtable v1.4.0
go: found cloud.google.com/go/datastore in cloud.google.com/go/datastore v1.2.0
go: found cloud.google.com/go/pubsub in cloud.google.com/go/pubsub v1.5.0
go: found github.com/ansel1/merry in github.com/ansel1/merry v1.5.1
go: found github.com/pborman/getopt/v2 in github.com/pborman/getopt v0.0.0-20190409184431-ee0cd42419d3
go: found github.com/codegangsta/cli in github.com/codegangsta/cli v1.22.4
go: found github.com/goji/param in github.com/goji/param v0.0.0-20160927210335-d7f49fd7d1ed
go: found github.com/mcuadros/go-gin-prometheus in github.com/mcuadros/go-gin-prometheus v0.1.0
go: found github.com/andreiavrammsd/validator in github.com/andreiavrammsd/validator v9.31.0+incompatible
go: found github.com/pact-foundation/pact-go/dsl in github.com/pact-foundation/pact-go v1.4.3
go: found github.com/nats-io/gnatsd/server in github.com/nats-io/gnatsd v1.4.1
go: found github.com/shurcooL/go/importgraphutil in github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636
go: github.com/coreos/etcd/etcdctl/ctlv2/command imports
	github.com/coreos/bbolt: github.com/coreos/[email protected]: parsing go.mod:
	module declares its path as: go.etcd.io/bbolt
	        but was required as: github.com/coreos/bbolt

Removing CGO_ENABLED=0 causes a panic accessing documents

On sourcegraph/sourcegraph/lib cd lib/ at 2d7f5b9a014a81a33df5812ede6f5cfa368b49ee

lsif-go f7ea242 with the following patch:

diff --git a/internal/indexer/indexer.go b/internal/indexer/indexer.go
index 3546c8d..f0a3ef2 100644
--- a/internal/indexer/indexer.go
+++ b/internal/indexer/indexer.go
@@ -9,7 +9,6 @@ import (
 	"go/types"
 	"log"
 	"math"
-	"os"
 	"path"
 	"strings"
 	"sync"
@@ -279,7 +278,7 @@ func (i *Indexer) loadPackage(deduplicate bool, patterns ...string) ([]*packages
 
 		// CGO_ENABLED=0 makes sure that we can handle files with assembly
 		// and other problems that may occur (unsure of exact reasons at time of writing)
-		Env: append(os.Environ(), "CGO_ENABLED=0"),
+		// Env: append(os.Environ(), "CGO_ENABLED=0"),
 	}
 
 	// Make sure we only load packages once per execution.

✔ Resolving module name... Done
✔ Listing dependencies... Done
✔ Loading packages... Done
✔ Emitting documents... Done
⠦ Emitting import references and definitions... panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1027d27f8]

goroutine 5105 [running]:
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportMoniker(0x140001266c0, 0x566, 0x140007ecf00, {0x10296bd30, 0x14026c19560}, 0x0)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/moniker.go:69 +0x1b8
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportMonikerReference(0x140001266c0, 0x140007ecf00, 0x1400013d7c0, 0x140109982a0)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/indexer.go:447 +0x220
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportsForPackage(0x140001266c0, 0x140007ecf00)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/indexer.go:416 +0x12c
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).visitEachPackage.func1.1()
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/visit.go:65 +0xb4
github.com/sourcegraph/lsif-go/internal/parallel.Run.func1(0x1402655a1a0, 0x14026701080, 0x1402655a190)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/parallel/parallel.go:24 +0x74
created by github.com/sourcegraph/lsif-go/internal/parallel.Run
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/parallel/parallel.go:20 +0x98

Lots of duplication

If you search for type Router struct in the output LSIF for gorilla/mux, you can see that the same hover result is duplicated 33 times, all referenced from the same result set.

Here's a visualisation:

The center is a single resultSet vertix, with many textDocument/hover edges that each point to duplicated hover result vertixes with the same content.

cross-package type alias definitions have self-referencing semantic definition results

Go To Definition on sfnt.Font in the definition of opentype.Font at https://sourcegraph.com/github.com/golang/image@35266b937fa6/-/blob/font/opentype/opentype.go#L67:6 goes to opentype.Font. I expected it to instead take me to the definition of sfnt.Font, which is in a different package in the same module.

failure to cross-reference internal/buildcfg with internal/goexperiment

In golang/go, the packages internal/buildcfg and internal/goexperiment both seem to be indexed, but references from the former to the latter are search-based: https://sourcegraph.com/github.com/golang/go/-/blob/src/internal/buildcfg/exp.go?L24:59#tab=def

Possibly related to #190. Both packages were added in Go 1.17, and both have similar special handling in the go tool as packages under GOROOT.

Flag to skip the emitting of textDocument/hover for import monikers

Add flag to point to go.mod file

Some projects do not use go.mod to delineate projects. We have a customer with a global dependency declaration in the root, then many projects that share the same go.mod. In order to index these separately, they need to link or copy the go.mod file into the index directory. Instead, we should accept a flag to point to a go.mod file that's in a different directory than what's being indexed.

no semantic results for cmd/...

Similar to std, the Go tool treats paths that start with cmd as special, referring to $GOROOT/src/cmd. lsif-go understands to treat std specially, but it does not do so for cmd:

lsif-go/internal/gomod/dependencies.go

Lines 236 to 244 in 6860e46

    
           func resolveRepoRootForImportPath(name string) (string, bool) { 
        
           	// When indexing golang/go, there are some references to the package "std" itself. 
        
           	//    Generally, "std/" is not referenced directly (it is just assumed when you have "fmt" or similar 
        
           	//    in your imports), but inside of golang/go, it is directly referenced. 
        
           	// 
        
           	//    In that case, we just return it directly, there is no other resolving to do. 
        
           	if name == "std" { 
        
           		return name, true 
        
           	}

The result is that there are no semantic results for code in some packages inside golang/go/src/cmd. E.g., Sourcegraph gives the wrong result for types.AlgType here: https://sourcegraph.com/github.com/golang/go@master/-/blob/src/cmd/compile/internal/reflectdata/alg.go?L22

Packages added under cmd in Go 1.17 do seem to work, according to @Strum355.

Broken Markdown formatting of lists in comments

Example: https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/frontend/graphqlbackend/search_pagination.go?L197-220

First line of each list is displayed normally, but then the rest of the lines seem to be wrapped in a code block. Image:

The same text seems to be fine when using search-based results:

Add more verbose output

Have a flag that will display which packages are found to be indexed. We should also exit 1 and print an error if there were no packages to index at all.

Minor typo

dependecy -> dependency

lsif-go/internal/gomod/dependencies.go

Line 95 in f8617a7

    
           return nil, fmt.Errorf("failed to list dependecy packages: %v\n%s", err, output)

Missing reference (calls of searchCommitLogInRepo)

This is not urgent, just flagging for visibility. Feel free to reassign or unassign. Looks like we may miss some references.

I was surprised that find reference of searchCommitLogInRepo didn't turn up anything. Find references shows me this:

When I search for searchCommitLogInRepo, it shows a caller in the function searchCommitLogInRepos (note the s there). The caller is in the same file:

Not sure why that call is not picked up as a reference. The Go-to-definition on that call works, so it's almost like the references edge is missing.

Update changelog and cut new release

We have a bunch of cool new features, need to update and release (make sure docker image is up-to-date as well? since I think that's how we use lsif-go in cloud)

Update Dependencies (Renovate Bot)

This master issue contains a list of Renovate updates and their statuses.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update golang.org/x/tools commit hash to e377ae9

Segfaults when running over go-ethereum

go-ethereum on  master via 🐹 v1.16.6 on 🐳 v20.10.8
[I] ➜ lsif-go-impl --no-animation
Resolving module name
Listing dependencies
Loading packages
Emitting documents
Emitting import references and definitions
panic: assignment to entry in nil map

goroutine 13582 [running]:
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).ensureRangeFor(0xc00000c1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xa60a08,
5d0b7aa0, 0x0, 0x0)
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/indexer/indexer.go:1355 +0x3bb
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportMonikerReference(0xc00000c1e0, 0xc00129c140, 0xc00061
 0xc01d7a4270)
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/indexer/indexer.go:417 +0x233
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportsForPackage(0xc00000c1e0, 0xc00129c140)
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/indexer/indexer.go:387 +0x13b
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).visitEachPackage.func1.1()
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/indexer/visit.go:65 +0x47
github.com/sourcegraph/lsif-go/internal/parallel.Run.func1(0xc05b93a730, 0xc02ad7b0e0, 0xc05b93a720)
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/parallel/parallel.go:24 +0x67
created by github.com/sourcegraph/lsif-go/internal/parallel.Run
        /home/tjdevries/sourcegraph/lsif-go.git/implementations/internal/parallel/parallel.go:20 +0xa6

Removing CGO_ENABLED=0 causes a panic accessing ranges

On sourcegraph/sourcegraph/lib cd lib/ at 2d7f5b9a014a81a33df5812ede6f5cfa368b49ee

lsif-go cde8136

✔ Listing dependencies... Done
✔ Loading packages... Done
✔ Emitting documents... Done
⠸ Emitting import references and definitions... panic: assignment to entry in nil map

goroutine 5079 [running]:
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).ensureRangeFor(0x1400009a6c0, {{0x0, 0x0}, 0x0, 0x0, 0x0}, {0x104e97dd0, 0x14027d22d80})
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/indexer.go:902 +0x2b0
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportMonikerReference(0x1400009a6c0, 0x14000b27180, 0x1400041a3c0, 0x1400f9c0570)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/indexer.go:446 +0x1e4
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).emitImportsForPackage(0x1400009a6c0, 0x14000b27180)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/indexer.go:416 +0x12c
github.com/sourcegraph/lsif-go/internal/indexer.(*Indexer).visitEachPackage.func1.1()
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/indexer/visit.go:65 +0xb4
github.com/sourcegraph/lsif-go/internal/parallel.Run.func1(0x14027745720, 0x140279b9f80, 0x14027745710)
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/parallel/parallel.go:24 +0x74
created by github.com/sourcegraph/lsif-go/internal/parallel.Run
	/Users/chrismwendt/github.com/sourcegraph/lsif-go/internal/parallel/parallel.go:20 +0x98

Support multi-field embedded structs

For a definition like the following, we should emit multiple monikers for each of the leaf fields:

type Example struct {
    a, b, c struct {
      x int
      y int
      z int
    }
}

We currently emit a.{x,y,z} but none of {b,c}.{x,y,z}.

	func resolveRepoRootForImportPath(name string) (string, bool) {
	// When indexing golang/go, there are some references to the package "std" itself.
	// Generally, "std/" is not referenced directly (it is just assumed when you have "fmt" or similar
	// in your imports), but inside of golang/go, it is directly referenced.
	//
	// In that case, we just return it directly, there is no other resolving to do.
	if name == "std" {
	return name, true
	}

sourcegraph / lsif-go Goto Github PK

lsif-go's Introduction

lsif-go's People

Stargazers

Watchers

Forkers

lsif-go's Issues

Description

Observation

Description

Observation

Open

Recommend Projects

Recommend Topics

Recommend Org