sourcegraph / lsif-py

Language Server Indexing Format (LSIF) generator for Python

(Deprecated) Python LSIF Indexer

This project is no longer maintained. Please use scip-python instead.

See below for the unmaintained documentation on how to use this project.

🚨 This implementation is in its infancy and conforms to the 0.4.0 draft of the LSIF spec.

Language Server Index Format

The purpose of the Language Server Index Format (LSIF) is to define a standard format for language servers and related tools to dump their knowledge about a workspace. The dump is basically a pre-computed set of responses that a language server would send back about a particular range of source code.

Basic Usage

To run the indexer, simply run the provided shell script wrapper and provide a workspace directory to be indexed.

$ ./lsif-py lsif_indexer
Indexing file lsif_indexer/analysis.py
Indexing file lsif_indexer/index.py
Indexing file lsif_indexer/__init__.py
Indexing file lsif_indexer/consts.py
Indexing file lsif_indexer/script.py
Indexing file lsif_indexer/emitter.py

Processed in 2834.89ms

Verbose logging can be enabled with -v. The dump file is data.lsif by default, but can be changed via the -o <filename> flag.

Installation Notes

The indexer requires Python 3.x. To install dependencies, run:

pip install -r requirements.txt

If you don't want to install dependencies globally, use virtualenv:

$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install -r requirements.txt
(venv) $ ./lsif-py lsif_indexer
# ...
(venv) $ deactivate
$

Installation on macOS

On macOS, where the system Python on macOS is still 2.7.x, you can install Python 3 via Homebrew:

brew install python@3

You may need to write pip3 instead of pip to get the correct version.

lsif-py's Issues

Why the status of this indexer marked as Unstable?

Can't run on python3.8.4

I don't quite know which python version this starts happening but lsif-py doesn't run:

  File "./lsif-py", line 6, in <module>
    main()
  File "/home/jwon/lsif-py/lsif_indexer/script.py", line 13, in main
    index(args.workspace, f, args.verbose, args.exclude_content)
  File "/home/jwon/lsif-py/lsif_indexer/index.py", line 285, in index
    FileIndexer(
  File "/home/jwon/lsif-py/lsif_indexer/index.py", line 61, in index
    self._index(source)
  File "/home/jwon/lsif-py/lsif_indexer/index.py", line 80, in _index
    self._export_definition(name)
  File "/home/jwon/lsif-py/lsif_indexer/index.py", line 116, in _export_definition
    docstring = name.docstring
  File "/home/jwon/lsif-py/lsif_indexer/analysis.py", line 41, in docstring
    return self.definition.docstring(raw=True, fast=False)
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/api/classes.py", line 246, in docstring
    doc = self._get_docstring()
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/api/classes.py", line 257, in _get_docstring
    return self._name.py__doc__()
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/names.py", line 340, in py__doc__
    names = self.goto()
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/names.py", line 155, in goto
    module_names = goto_import(context, name)
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/cache.py", line 43, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/imports.py", line 76, in goto_import
    _prepare_infer_import(module_context, tree_name)
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/imports.py", line 114, in _prepare_infer_import
    importer = Importer(module_context.inference_state, tuple(import_path),
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/imports.py", line 168, in __init__
    debug.speed('import %s %s' % (import_path, module_context))
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/context.py", line 210, in __repr__
    return '%s(%s)' % (self.__class__.__name__, self._value)
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/value/module.py", line 263, in __repr__
    self.__class__.__name__, self._string_name,
  File "/home/jwon/.pyenv/versions/lsif-py-3.8.4/lib/python3.8/site-packages/jedi/inference/value/module.py", line 125, in _string_name
    r = re.search(r'([^%s]*?)(%s__init__)?(\.pyi?|\.so)?$' % sep, self._path)
  File "/home/jwon/.pyenv/versions/3.8.4/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

I believe this is because in Jedi self._path starts referring to a Path object rather than a str/bytes one, so the regex fails.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): pin mcr.microsoft.com/vscode/devcontainers/python docker tag to b90ac67

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update actions/checkout action to v3
• chore(deps): update actions/setup-go action to v3

Detected dependencies

dockerfile

.devcontainer/Dockerfile

• mcr.microsoft.com/vscode/devcontainers/python 0-3.9

github-actions

.github/workflows/pr-auditor.yml

• actions/checkout v2
• actions/setup-go v2

pip_requirements

requirements.txt

---

• Check this box to trigger a request for Renovate to run again on this repository

Publishing to PyPi

Is there a reason that this package hasn't yet been published to pypi? If not, I can open a PR setting up a GH action to do it, though will need a maintainer to set up the credentials secret in the repo

Support for incremental updates to the index

Re-building the entire index from scratch on every code change to a project is hardly ideal, especially for large projects. Is it feasible to implement support for incremental updates to the index? The incremental indexer could take as input a previously generated language graph and the diff of changes to the codebase, and use that to output the delta that should be applied to the old graph to get the new one.

Add pip packaging

Add setup.py so that the indexer does not need to be run in-tree.

KeyError when running lsif-py

~/lsif-py/lsif-py .

Loading files (93.61s)
Generating LSIF data (93.61s)
Traceback (most recent call last):
File "/Users/william.chau/lsif-py/lsif-py", line 6, in
lsif_py()
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/Users/william.chau/lsif-py/lsif_indexer/indexer.py", line 611, in lsif_py
for element in project.index():
File "/Users/william.chau/lsif-py/lsif_indexer/indexer.py", line 504, in index
self.load_files()
File "/Users/william.chau/lsif-py/lsif_indexer/indexer.py", line 580, in load_files
document = Document(script=script)
File "", line 9, in init
File "/Users/william.chau/lsif-py/lsif_indexer/indexer.py", line 331, in post_init
self.definitions = self.script.get_names(
File "/usr/local/lib/python3.10/site-packages/jedi/api/init.py", line 544, in get_names
names = self._names(**kwargs)
File "/usr/local/lib/python3.10/site-packages/jedi/api/init.py", line 558, in _names
defs = [
File "/usr/local/lib/python3.10/site-packages/jedi/api/init.py", line 559, in
module_context.create_name(name)
File "/usr/local/lib/python3.10/site-packages/jedi/inference/context.py", line 294, in create_name
func = self.create_value(funcdef)
File "/usr/local/lib/python3.10/site-packages/jedi/inference/context.py", line 236, in create_value
func = value.FunctionValue.from_context(parent_context, node)
File "/usr/local/lib/python3.10/site-packages/jedi/inference/value/function.py", line 145, in from_context
overloaded_funcs = list(_find_overload_functions(context, tree_node))
File "/usr/local/lib/python3.10/site-packages/jedi/inference/value/function.py", line 443, in _find_overload_functions
filter = ParserTreeFilter(
File "/usr/local/lib/python3.10/site-packages/jedi/inference/filters.py", line 138, in init
super().init(parent_context, node_context)
File "/usr/local/lib/python3.10/site-packages/jedi/inference/filters.py", line 100, in init
self._parso_cache_node = get_parso_cache_node(
File "/usr/local/lib/python3.10/site-packages/jedi/parser_utils.py", line 287, in get_parso_cache_node
return parser_cache[grammar._hashed][path]
KeyError: PosixPath(...)

LSIF-PY: Error in Upload and Code intel not able to link sub packages in a monorepo

Description

• SRC CLI version: 3.38.0
• Sourcegraph version: 136196_2022-03-10_bd60b4b0391c (we realize this is accidentally insiders for now, but mentioning in case it matters for this issue)
• Submitted from a customer with information redacted. ticket link

Reported that when trying to use LSIF-PY, they received an error (below) and code intelligence is not able to link sub
packages in the monorepo. The error happened when running the lsif-py tool on the python codebases (and ran into this in every one of the python repos) Example:

For example @bfly/ui resolves locally to ’../libraries/ui but Souregraph is not seeing that The same is true for imports like in other repos, Sourcegraph seems to understand which dependencies are part of the graph (or is guessing) but not where to resolve them strictly.

Stack trace with name info redacted:

% ./lsif-py /Users/redactedname/src/redacted-link/redacted-link
 
 
Loaded 58 files (3.47s)
 
Indexing (9.28s)
Generating LSIF data (12.76s)
Traceback (most recent call last):
  File "/Users/redactedname/src/lsif-py/./lsif-py", line 6, in <module>
    lsif_py()
  File "/Users/redactedname/.pyenv/versions/global/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/redactedname/.pyenv/versions/global/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/redactedname/.pyenv/versions/global/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/redactedname/.pyenv/versions/global/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 611, in lsif_py
    for element in project.index():
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 521, in index
    yield from self.toposort_index(to_index)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 562, in toposort_index
    yield from try_to_index(document)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 555, in try_to_index
    yield from try_to_index(dep)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 555, in try_to_index
    yield from try_to_index(dep)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 555, in try_to_index
    yield from try_to_index(dep)
  File "/Users/redactedname/src/lsif-py/lsif_indexer/indexer.py", line 553, in try_to_index
    raise CircularDependencyError(stack, dep.filename)
lsif_indexer.indexer.CircularDependencyError: Circular dependency detected: /Users/redactedname/src/redacted-link/redacted-link/dicom_client.py -> /Users/redactedname/src/redacted-link/redacted-link/dicom.py -> /Users/redactedname/src/redacted-link/redacted-link/utils/client.py -> /Users/redactedname/src/redacted-link/redacted-link/models.py -> /Users/redactedname/src/redacted-link/redacted-link/dicom.py

Questions:

1. What could be causing this error? Is the behavior described in the screen shot and behavior expected given that the Python LSIF Indexer status is 'in development'?
2. I had trouble figuring out if this is a potential compatibility issue with either Sourcegraph, SRC CLI, Python, etc. What would help me narrow this down in the future?