tqchen / ffi-navigator

FFI Navigator

Most modern IDEs support find function definition within the same language(e.g. python or c++). However, it is very hard to do that for cross-language FFI calls. While solving this general problem can be very technically challenging, we can get around it by building a project-specific analyzer that matches the FFI registration code patterns and recovers the necessary information.

This project is an example of that. Currently, it supports the PackedFunc FFI in the Apache TVM project. It is implemented as a language server that provides getDefinition function for FFI calls and returns the location of the corresponding C++ API in the TVM project. It complements the IDE tools that support navigation within the same language. We also have preliminary support for MXNet, DGL, and PyTorch, so we can do goto-definition from Python to C++ in these projects too.

Installation

Install python package

pip install --user ffi-navigator

VSCode

See vscode-extension

Emacs

Install lsp-mode

Add the following configuration

(lsp-register-client
 (make-lsp-client
  :new-connection (lsp-stdio-connection '("python3" "-m" "ffi_navigator.langserver"))
  :major-modes '(python-mode c++-mode)
  :server-id 'ffi-navigator
  :add-on? t))

• Use commands like M-x lsp-find-definition and M-x lsp-find-references

If you use eglot instead, check out this PR. eglot does not support multiple servers per language at the moment, the PR above contains a workaround.

Other editors/IDEs

It should be straightforward to add support for other editors or IDEs that have a LSP client implementation. Please refer to this site for the availability of clients.

Since ffi-navigator is intended to be used with other general purpose servers such as pyls, your LSP client needs to be able to talk to multiple servers per language. If your client does not have such feature, check out this PR and the discussion there for a workaround. This branch has fallback to pyls and it is always kept up to date with the master branch.

Features

TVM FFI

• Find definition/references of FFI objects(e.g. PackedFunc in TVM) on the python and c++ side.
  • Jump from a python PackedFunc into TVM_REGISTER_GLOBAL, @register_func
• Find definition/references of FFI objects on the python and c++ side.
  • move cursor to the object class name on the python side.
  • move cursor to the _type_key string on the c++ side.

PyTorch

• Jump to C10 registered ops. In python they corresponds to functions under torch.ops namespace.
  • Example: torch.ops.quantized.conv2d (py) -> c10::RegisterOperators().op("quantized::conv2d", ...) (cpp)
• Jump to cpp functions wrapped by pybind.
  • Example: torch._C._jit_script_class_compile (py) -> m.def( "_jit_script_class_compile", ...) (cpp)

Development

For developing the python package locally, we can just make sure ffi_navigator is in your python path in bashrc.

export PYTHONPATH=${PYTHONPATH}:/path/to/ffi-navigator/python

Project Structure

• python/ffi_navigator The analysis code and language server
• python/ffi_navigator/dialect Per project dialects
• vscode-extension language server extension for vscode

Adding Support for New FFI Patterns

Add your FFI convention to dialect namespace.

Demo

VSCode

See vscode-extension

Emacs

Goto definition from Python to C++

Goto definition from C++ to Python

Find reference across Python and C++

Goto definition in PyTorch

ffi-navigator's Issues

Migrate to pyright from pyls?

pyls we depend on has not been active https://github.com/palantir/python-language-server
Emacs lsp-mode also says pyls is deprecated https://emacs-lsp.github.io/lsp-mode/page/lsp-pyls/

Apparently, pyright is a popular lang server right now. Probably we should support this server.

TypeError: startswith first arg must be str or a tuple of str, not NoneType

TypeError: startswith first arg must be str or a tuple of str, not NoneType

Error: The connected server(s) does not support method textDocument/definition.

HI, When I use M-x lsp-find-definition, it came the error:

The connected server(s) does not support method textDocument/definition.                                                                                                        
To find out what capabilities support your server use ‘M-x lsp-describe-session’                                                                                                
and expand the capabilities section 

and then I use M-x lsp-describe-session, it create a new buffer, the new buffer contains
nothing. So what's wrong with it, can anyone help me? Thanks!

I am using emacs-26.1, and config is :

   1 ;;; init-local.el --- ffi_navigator setup -*- lexical-binding: t -*-                                                                                                       
   2 ;;; Commentary:                                                                                                                                                            
   3 ;;; Code:                                                                                                                                                                  
   4                                                                                                                                                                            
   5 (require 'lsp)                                                                                                                                                             
   6 (lsp-register-client                                                                                                                                                       
   7  (make-lsp-client                                                                                                                                                          
   8   :new-connection (lsp-stdio-connection '("python" "-m" "ffi_navigator.langserver"))                                                                                       
   9   :major-modes '(python-mode c++-mode)                                                                                                                                     
  10   :server-id 'ffi-navigator                                                                                                                                                
  11   :add-on? t))                                                                                                                                                             
  12                                                                                                                                                                            
  13 (provide 'init-local)    
  14 ;;; init-local.el ends here  

Any plan to make this package PyPi available?

I think it would be convenient because we can let vs code extension pip install this package when it is missing. Although we should be able to let vs code extension clone this package from Github and run setup.py, installing from pip is more reliable in my opinion.

cannot jump in VS Code

Hi, I installed the extension in VS Code and it turned out that it cannot jump between Python and C++ code within TVM codebase. VS Code gives the prompt "ffi_navigator is not installed. Please run "pip3 install ffi_nagivator" and reload the window" as showed below.

Then I installed ffi_navigator with pip (I supposed "ffi_nagivator" was typo). However, the extension seems still not working (VS Code warns that "No definition found for ...").
Did I miss something to use this extension properly? Thanks for any feedback.

Server initialization failed

Hi folks,
I got issue which seems to be related with server initialization when I try to using vscode extention. Here's my setup:

Setup

1. Completed all the required steps in README, including npm complie and install ffi-navigator vscode extention on my remote ssh machine.
2. I got conda on my remote machine and there is a env called "tvm". I installed ffi pip package in this env and selected /home/ruotongw/.conda/envs/tvm/bin/python as interpreter on vscode. Also, I set "ffi_navigator.pythonpath": "/home/ruotongw/.conda/envs/tvm/bin/python" in the extention settings.
3. Seems the pip package version is 0.5.2 and vscode extention version is 0.7

Issue

When I try to reload the window, it pops up in the output section followings:

[2021-10-26 17:16:06,085] Initialize {'processId': 199364, 'rootPath': '/home/ruotongw/Workspace/tvm', 'rootUri': 'file:///home/ruotongw/Workspace/tvm', 'capabilities': {'workspace': {'applyEdit': True, 'workspaceEdit': {'documentChanges': True, 'resourceOperations': ['create', 'rename', 'delete'], 'failureHandling': 'textOnlyTransactional'}, 'didChangeConfiguration': {'dynamicRegistration': True}, 'didChangeWatchedFiles': {'dynamicRegistration': True}, 'symbol': {'dynamicRegistration': True, 'symbolKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]}}, 'executeCommand': {'dynamicRegistration': True}, 'configuration': True, 'workspaceFolders': True}, 'textDocument': {'publishDiagnostics': {'relatedInformation': True}, 'synchronization': {'dynamicRegistration': True, 'willSave': True, 'willSaveWaitUntil': True, 'didSave': True}, 'completion': {'dynamicRegistration': True, 'contextSupport': True, 'completionItem': {'snippetSupport': True, 'commitCharactersSupport': True, 'documentationFormat': ['markdown', 'plaintext'], 'deprecatedSupport': True, 'preselectSupport': True}, 'completionItemKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]}}, 'hover': {'dynamicRegistration': True, 'contentFormat': ['markdown', 'plaintext']}, 'signatureHelp': {'dynamicRegistration': True, 'signatureInformation': {'documentationFormat': ['markdown', 'plaintext'], 'parameterInformation': {'labelOffsetSupport': True}}}, 'definition': {'dynamicRegistration': True, 'linkSupport': True}, 'references': {'dynamicRegistration': True}, 'documentHighlight': {'dynamicRegistration': True}, 'documentSymbol': {'dynamicRegistration': True, 'symbolKind': {'valueSet': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]}, 'hierarchicalDocumentSymbolSupport': True}, 'codeAction': {'dynamicRegistration': True, 'codeActionLiteralSupport': {'codeActionKind': {'valueSet': ['', 'quickfix', 'refactor', 'refactor.extract', 'refactor.inline', 'refactor.rewrite', 'source', 'source.organizeImports']}}}, 'codeLens': {'dynamicRegistration': True}, 'formatting': {'dynamicRegistration': True}, 'rangeFormatting': {'dynamicRegistration': True}, 'onTypeFormatting': {'dynamicRegistration': True}, 'rename': {'dynamicRegistration': True, 'prepareSupport': True}, 'documentLink': {'dynamicRegistration': True}, 'typeDefinition': {'dynamicRegistration': True, 'linkSupport': True}, 'implementation': {'dynamicRegistration': True, 'linkSupport': True}, 'colorProvider': {'dynamicRegistration': True}, 'foldingRange': {'dynamicRegistration': True, 'rangeLimit': 5000, 'lineFoldingOnly': True}, 'declaration': {'dynamicRegistration': True, 'linkSupport': True}}}, 'trace': 'off', 'workspaceFolders': [{'uri': 'file:///home/ruotongw/Workspace/tvm', 'name': 'tvm'}]}
[2021-10-26 17:16:06,086] root_path: /home/ruotongw/Workspace/tvm
[2021-10-26 17:16:06,087] Workspace.update_dir /home/ruotongw/Workspace/tvm/src start
[2021-10-26 17:16:06,094] Failed to handle request 0
Traceback (most recent call last):
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 113, in consume
    self._handle_request(message['id'], message['method'], message.get('params'))
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/pyls_jsonrpc/endpoint.py", line 182, in _handle_request
    handler_result = handler(params)
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/pyls_jsonrpc/dispatchers.py", line 23, in handler
    return method(**(params or {}))
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/langserver.py", line 47, in m_initialize
    self.ws.initialize(root_path)
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/workspace.py", line 36, in initialize
    self._reload()
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/workspace.py", line 51, in _reload
    self.update_dir(dirname)
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/workspace.py", line 73, in update_dir
    self.update_doc(path, open(path).readlines())
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/workspace.py", line 84, in update_doc
    for pt in provider.extract(path, source):
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/dialect/base_provider.py", line 57, in extract
    return self._py_extract(path, source, begin, end)
  File "/home/ruotongw/.conda/envs/tvm/lib/python3.8/site-packages/ffi_navigator/dialect/tvm.py", line 97, in _py_extract
    if path.startswith(self._pypath_api_internal):
TypeError: startswith first arg must be str or a tuple of str, not NoneType
[Error - 5:16:06 PM] Server initialization failed.
  Message: TypeError: startswith first arg must be str or a tuple of str, not NoneType
  Code: -32602 
[object Object]

How should I deal with this problem? Thanks!

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4622: ordinal not in range(128)

Hi,

When I installed FFI Navigator vscode extension and pip install ffi-navigator, then restart vscode. This error comes out. Is there anyone to meet this before and how can I fix this?

Environment is ubuntu 16.04 docker. Thanks.

[2022-05-01 08:05:00,635] Failed to handle request 0 Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pyls_jsonrpc/endpoint.py", line 113, in consume self._handle_request(message['id'], message['method'], message.get('params')) File "/usr/lib/python3.6/site-packages/pyls_jsonrpc/endpoint.py", line 182, in _handle_request handler_result = handler(params) File "/usr/lib/python3.6/site-packages/pyls_jsonrpc/dispatchers.py", line 23, in handler return method(**(params or {})) File "/usr/lib/python3.6/site-packages/ffi_navigator/langserver.py", line 47, in m_initialize self.ws.initialize(root_path) File "/usr/lib/python3.6/site-packages/ffi_navigator/workspace.py", line 36, in initialize self._reload() File "/usr/lib/python3.6/site-packages/ffi_navigator/workspace.py", line 51, in _reload self.update_dir(dirname) File "/usr/lib/python3.6/site-packages/ffi_navigator/workspace.py", line 77, in update_dir self.update_doc(path, open(path).readlines()) File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4622: ordinal not in range(128) [Error - 8:05:00 AM] Server initialization failed.

TypeError

TypeError: startswith first arg must be str or a tuple of str, not NoneType

"FileNotFoundError" when use vscode in windows

error log:
Traceback (most recent call last):
File "C:\Users\Admin\AppData\Roaming\Python\Python39\site-packages\pyls_jsonrpc\endpoint.py", line 113, in consume
self._handle_request(message['id'], message['method'], message.get('params'))
File "C:\Users\Admin\AppData\Roaming\Python\Python39\site-packages\pyls_jsonrpc\endpoint.py", line 182, in _handle_request
handler_result = handler(params)
File "C:\Users\Admin\AppData\Roaming\Python\Python39\site-packages\pyls_jsonrpc\dispatchers.py", line 23, in handler
return method(**(params or {}))
File "C:\Users\Admin\AppData\Roaming\Python\Python39\site-packages\ffi_navigator\langserver.py", line 62, in m_text_document__definition
source = open(path).readlines()
FileNotFoundError: [Errno 2] No such file or directory: 'c%3A\work\file\technicial_file_record\TVM\incubator-tvm-main\tutorials\language\schedule_primitives.py'

Error when URI Contains Encoded Characters

Just like below log shown, path "/usr/include/c++/7/tr1/shared_ptr.h" is encoded by VS Code as "file:///usr/include/c%2B%2B/7/tr1/shared_ptr.h", we need decoded it before use.

[2020-10-19 15:54:57,256] textDocument/definition {'textDocument': {'uri': 'file:///usr/include/c%2B%2B/7/tr1/shared_ptr.h'}, 'position': {'line': 571, 'character': 19}}
[2020-10-19 15:54:57,256] Failed to handle request 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyls_jsonrpc/endpoint.py", line 113, in consume
    self._handle_request(message['id'], message['method'], message.get('params'))
  File "/usr/local/lib/python3.6/dist-packages/pyls_jsonrpc/endpoint.py", line 182, in _handle_request
    handler_result = handler(params)
  File "/usr/local/lib/python3.6/dist-packages/pyls_jsonrpc/dispatchers.py", line 23, in handler
    return method(**(params or {}))
  File "/usr/local/lib/python3.6/dist-packages/ffi_navigator/langserver.py", line 62, in m_text_document__definition
    source = open(path).readlines()
FileNotFoundError: [Errno 2] No such file or directory: '/usr/include/c%2B%2B/7/tr1/shared_ptr.h'
[Error - 3:54:57 PM] Request textDocument/definition failed.
  Message: FileNotFoundError: [Errno 2] No such file or directory: '/usr/include/c%2B%2B/7/tr1/shared_ptr.h'
  Code: -32602 
[object Object]