Comments (7)
i am also try to run this project successfully, cheer up and thx for your guys‘ issue, it gives me hope to do it!thx!
If you're interested, here's our implementation of gt retrieval (without filtering the unfinished part): https://github.com/code-rag-bench/code-rag-bench/tree/main?tab=readme-ov-file#retrieval.
from codet.
We are truly sorry the generated content cannot be restored for now. We would love to help you reproduce the results though.
from codet.
Thanks for being willing to help! Just want to ask about the retrieval setting. When you do retrieval, do you filter out the file containing the code to complete?
If not, the model is possible to retrieve the target of code generation as the context, which does not make much sense to me -- if you want to use a model to help you complete the code, by the time you call the model, the target code does not exist in the repo yet.
If you do, is there an efficient way to do that?
from codet.
Of course, we need to filter out the target file to avoid leakage. However, we did not filter out all the content in the target file. We keep the content in the front of the target file that is not covered by the context provided to the LM. For example, file A has 100 lines, we have line 20-80 as the unfinished code, and line 81 as the completion hole. During retrieval, we also retrieve line 1-19 as useful supplementary Information for the completion.
The code related to this matter is
CodeT/RepoCoder/search_code.py
Line 44 in 35f54d6
The context_start_lineno is metadata we stored for each completion case.
from codet.
Thanks. The setting makes sense to me!
I retrieved the GT context for the "function" split by adapting your code (window_size=50, slice_size=5). Then I filtered out the unfinished part using your logic (line 44) and run code generation using ChatGPT (2k tokens for GT context, 2k for infile context). I only got Pass@1=0.2895, and you reported 0.4263.
The full evaluation results are:
{
"EM": 0.10723860589812333,
"ES": 0.48067297674081083,
"Pass@1": 0.289544235924933,
}
Here's the GT context I got:
repoeval-function-4k-gt-top5-filter.jsonl.txt
Can you run your code to produce the GT context file for RepoEval-function, so I can compare the difference? Thanks!!!
from codet.
i am also try to run this project successfully, cheer up and thx for your guys‘ issue, it gives me hope to do it!thx!
from codet.
from codet.
Related Issues (20)
- This repo is missing important files
- The wrong constant rg was used in RepoCoder
- Random subsets for reproduction HOT 1
- File Not Found Error HOT 2
- [RepoCoder] Benchmark test file not found: datasets/random-api-completion.test.jsonl HOT 1
- Missing code for inference HOT 4
- No such file or directory: 'cache/window/random_api/gt/huggingface_diffusers_ws20.pkl HOT 3
- Clarification on Datasets and Hyperparameters Used in Experiments HOT 1
- The growth of a fish is often modeled by the von Bertalanffy growth model
- Issue with Handling Input Prompt Files in codegen-inference.py HOT 7
- No such file or directory: 'prompts/rg-one-gram-ws-20-ss-2.jsonl' HOT 5
- Some questions about APPS on CodeT HOT 1
- repocoder missing repo vectors, the error still exists
- RepoCoder: The input file is missing run_pipline.py run
- A bug related to regular expression in extracting equations in GSM8K in DIVERSE project
- A problem about the splitting "&&" in eval_sequences DIVERSE run_ner.py
- Issue with token length of input prompt in codegen-inference.py HOT 1
- Inquiry About Retrieving and Generating Different Lengths of Datasets HOT 2
- please check out branch zfj/RepoCoder for the latest version of RepoCoder
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codet.