farishijazi / localcopilot

A self-hosted github copilot guide using oobabooga webui

localCopilot

donate here

This project allows you to host your own GitHubCopilot-like model locally while using the official GitHubCopilot VSCode extension.

Installation:

1. Download and install the oobabooga backend

2. Download a model

   open the oobabooga UI, go to the models tab and download a code completion model. I'm using: Deci/DeciCoder-1b, paste that name, then click download, then click load once complete

   Which model should I choose? Use smaller models for faster predictions, especially if you have a weaker PC. I tested DeciCoder-1b

   size	speed	model name
   125M	superfast	flax-community/gpt-neo-125M-code-clippy-dedup-2048
   1B	fast	Deci/DeciCoder-1b
   3B	medium	TheBloke/stablecode-instruct-alpha-3b-GGML
   7B	slow	mlabonne/codellama-2-7b
   15B	slow	TheBloke/WizardCoder-15B-1.0-GGML

   

Optional testing

A. (optional) Test the backend using curl:

```sh
curl -X 'POST'   'http://localhost:5000/v1/engines/codegen/completions'   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '{"prompt":"def hello_w","suffix":"","max_tokens":500,"temperature":0.4,"top_p":1,"n":10,"stop":["\ndef ","\nclass ","\nif ","\n\n#"],"logprobs":2,"stream":true}'
```

B. (optional) Test that the model is working by going to the "chat" tab and clicking "generate".

3. Go to VSCode and modify the settings and add the following:

   "github.copilot.advanced": {
       "debug.overrideEngine": "codegen",
       "debug.testOverrideProxyUrl": "http://localhost:8000", // address:port of middleware
       "debug.overrideProxyUrl": "http://localhost:8000",
   },

4. (optional for authentication) Update ~/.vscode/extensions/github.copilot-*/dist/extension.js with the following:

   • Replace https://api.github.com/copilot_internal with http://127.0.0.1:8000/copilot_internal
   • replace https://copilot-proxy.githubusercontent.com with http://127.0.0.1:8000

5. Run the proxy:

   pip install git+https://github.com/FarisHijazi/localCopilot
   localCopilot --port 7000

   If you have oobabooga running on a separate server use the --backend argument {hostname:port}

   pip install git+https://github.com/FarisHijazi/localCopilot
   localCopilot --port 8000 --backend http://10.0.0.1:5002

(Optional): testing the middleware

curl -X 'POST'   'http://localhost:8000/v1/engines/codegen/completions'   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '{"prompt":"def hello_w","suffix":"","max_tokens":500,"temperature":0.4,"top_p":1,"n":2,"stop":["\ndef ","\nclass ","\nif ","\n\n#"],"logprobs":2,"stream":true}'

expected output

data: {"id": "conv-1692741316942825472", "object": "text_completion.chunk", "created": 1692741316, "model": "Deci_DeciCoder-1b", "choices": [{"index": 0, "finish_reason": "stop", "text": "", "logprobs": {"top_logprobs": [{"<|endoftext|>": -0.4215908944606781, "<fim_middle>": -1.2965909242630005, "\n": -3.0741329193115234}]}}], "usage": {"prompt_tokens": 4, "completion_tokens": 13, "total_tokens": 17}}

data: [DONE]

6. install the official GitHub copilot extension

7. HAPPY CODING!

   To test that the copilot extension is working, either type some code and hope for a completion or use the command pallet (Ctrl+Shift+P) and search for GitHub Copilot: Open Completions Panel

---

How this works

This is done using a single script: localCopilot/middleware.py (only 90 lines of code), which is a compatibility layer between the official GitHub copilot VSCode extension and oobabooga as a backend.

Credit: I learned about the traffic redirecting from the Fauxpilot project here.

Cloud
Self-hosted

Advanced experimental hacks

The tokenizers used by Copilot are not the same, so you can overwrite them. However, I'm not sure how useful this actually is as I don't notice any change in performance

COPILOTPATH=$HOME/.vscode/extensions/github.copilot-1.105.353
MODELPATH=$HOME/Projects/oobabooga_linux/text-generation-webui/models/Deci_DeciCoder-1b

mv $COPILOTPATH/dist/resources $COPILOTPATH/dist/resources.backup
mkdir -p $COPILOTPATH/dist/resources/cushman001
mkdir -p $COPILOTPATH/dist/resources/cushman002

cp $MODELPATH/tokenizer.json $COPILOTPATH/dist/resources/cushman001/tokenizer_cushman001.json
cp $MODELPATH/merges.txt     $COPILOTPATH/dist/resources/cushman001/vocab_cushman001.bpe

cp $MODELPATH/tokenizer.json $COPILOTPATH/dist/resources/cushman002/tokenizer_cushman002.json
cp $MODELPATH/merges.txt     $COPILOTPATH/dist/resources/cushman002/vocab_cushman002.bpe

And to revert your changes, just uninstall and reinstall the extension.

OR:

rm -rf $COPILOTPATH/dist/resources
mv $COPILOTPATH/dist/resources.backup $COPILOTPATH/dist/resources

😁 Benefits of self-hosting

1. 🔏 Privacy: No more sending your code to the cloud! This is the main benefit especially for enterprise. No code is sent to the cloud when self-hosting since everything runs on your machine(s).
2. 🌐 Works without internet: use it on the plane! ✈️
3. 💰 Free: No need to pay for your monthly subscription

😞 Limitations

1. GitHub copilot looks at multiple files for context. The current hack only looks at the current file
2. Quality Open source models might not have suggestions as good as copilot, but still as good most of the time
3. GitHub copilot gives 10 suggestions, while this hack gives only 1 suggestion per completion
4. 🐛 There's a bug where the first space in autocompletion is skipped, this is due to the oobabooga backend, not the model

Why is this project setup this way?

There are many other projects for having an open source alternative for copilot, but they all need so much maintenance, I tried to use an existing large project that is well maintained: oobabooga, since it supports almost all open source LLMs and is commonly used, and is well maintained

I know that the middleware method might not be optimal, but this is a minimal hack that's easy to run, and this repository should be really easy to maintain.

Once oobabooga supports multiple requests in a single call, then the middleware should no longer be needed.

Other works

Here are some helpful open source projects I found while doing my research:

Project URL	description	actively maintained (as of Aug 2023)
https://github.com/CodedotAl/gpt-code-clippy	Frontend + models	❌
https://github.com/Venthe/vscode-fauxpilot	this is a FauxPilot frontend	✅
https://github.com/hieunc229/copilot-clone	frontend which uses Google/StackOverflow search as a backend	✅
https://github.com/fauxpilot/fauxpilot	FauxPilot backend	✅
https://github.com/ravenscroftj/turbopilot	A backend that runs models	✅

localcopilot's Issues

No large responses? (max 16 tokens)

Could be an environment issue but I've tried different sizes of model and my suggestions are always ~16 tokens long (when if i don't override the api and use regular co-piolet i would get much longer responses). if i paste my code into the ooba notbook i get larger responses with the same model so its something to do with how co-pilot is passing the request to ooba.

In the sample gif i can see you get a full function response, is there something i need to change in ooba or another setting i need to override in the extension? any ideas?

VSCode Remote

I use VSCode exclusively for remote dev, so if I want to do everything remotely, is it possible to run the localCopilot on the same server that's running my oobabooga backend? And then link my copilot to that remote proxy server instead of my localhost?

GitHub authentication

Greetings Faris ,

mid there a way to bypass the authentication process for GitHub Copilot, because the plug-in that is in vscode is not fully offline , but needs authentication in order to make your middleware works.

thanks a lot

[Alert] You need to have access to the original Copilot initially

After the VSCode reloads, you might encounter the message "GitHub Copilot could not establish a connection to the server" along with "No access to GitHub Copilot detected." If you come across this message, it appears to be a roadblock, even though we do not actually intend to establish a connection with GitHub.

To complete the setup and direct traffic to your locally configured LLM, it must initially authenticate with the authentic GitHub Copilot service.

So if we have any of the scenarios below this repo wouldn't work.

Ollama support

Ollama is a very popular backend for running local models with a large library of supported models. It would be great to see ollama support

Cant get it installed.

followed the instructions and got the following error:

Collecting git+https://github.com/FarisHijazi/PrivateGitHubCopilot
  Cloning https://github.com/FarisHijazi/PrivateGitHubCopilot to c:\users\lvana\appdata\local\temp\pip-req-build-p48_4g0k
  Running command git clone --filter=blob:none --quiet https://github.com/FarisHijazi/PrivateGitHubCopilot 'C:\Users\lvana\AppData\Local\Temp\pip-req-build-p48_4g0k'
  Resolved https://github.com/FarisHijazi/PrivateGitHubCopilot to commit ce5f2a5fca48d3ad894cacf6937439cb6fb2c893
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.      
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\lvana\AppData\Local\Temp\pip-req-build-p48_4g0k\setup.py", line 8, in <module>
          long_description = fh.read().replace('](', '](https://raw.githubusercontent.com/FarisHijazi/PrivateGitHubCopilot/master/')
        File "C:\Users\lvana\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
          return codecs.charmap_decode(input,self.errors,decoding_table)[0]
      UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 955: character maps to <undefined>
      [end of output]

figured maybe you weren't on windows and it was just an encoding problem but when i forked the repo and made a new setup that included utf 8 encoding. The install seems to work but the middleware module is not being found when i try the run command

PrivateGitHubCopilot --port 8000

I get

Traceback (most recent call last):
  File "C:\Users\lvana\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\lvana\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\lvana\AppData\Local\Programs\Python\Python310\Scripts\PrivateGitHubCopilot.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'middleware'

too sleepy now to figure it out so posting to bring it to your attention to see if you know how I'm being silly.

thanks for making this!