xlang-ai / binder Goto Github PK
View Code? Open in Web Editor NEW[ICLR 2023] Code for the paper "Binding Language Models in Symbolic Languages"
License: Apache License 2.0
[ICLR 2023] Code for the paper "Binding Language Models in Symbolic Languages"
License: Apache License 2.0
Hi! It's me again.๐คฃ
When I run the annotate script on wikitq dataset (Here are my cmd and args)
python -u scripts/annotate_binder_program.py --dataset wikitq
--dataset_split test
--prompt_file templates/prompts/wikitq_binder.txt
--n_parallel_prompts 1
--n_processes 2
--max_generation_tokens 512
--temperature 0.4
--sampling_n 20
-v
Error generation error: list index out of range
occurred in 76 samples. I took the first 5 samples (wtqid#nu-30, 208, 263, 279, 367) for debug, and found that the input still exceeded the length limit even when n_shot
was reduced to 0.
scripts/annotate_binder_program.py
The list few_shot_prompt_list
is empty and this step will throw the exception.
Will the missing results of these 76 samples have any effect on the execution stage? How should I solve this problem?
Thanks!
Hi there,
I was wondering where I can find "binder_program_wikitq_test_exec.json" file, mentioned in the following command.
Thanks a lot.
=======================
"wikitq nsql execution command"
os.system(fr"""{TOKENIZER_FALSE}python {ROOT_DIR}/scripts/execute_binder_program.py --dataset wikitq
--dataset_split test
--qa_retrieve_pool_file templates/qa_retrieve_pool/qa_retrieve_pool.json
--input_program_file binder_program_wikitq_test.json
--output_program_execution_file binder_program_wikitq_test_exec.json
--vote_method simple
""")
Would it be possible to share the result files (both gpt-3.5
and codex
maybe?) for the WTQ dataset (test set)? I'm planning on sampling a few data points for testing and would appreciate having the same subset for a comparison. Thank you.
@Timothyxxx @BlankCheng Thanks for sharing the code!
I noticed that the default option in the code is to list out all rows for the testing table. However, in the paper, it seems you select the first 3 rows. I wonder what is the setting that corresponds to the results reported in the paper? Thanks!
I tried to create a proper env using
conda env create -f py3.7binder.yaml
pip install records==0.5.3
however, the conda
command fails.
I tried it over Mac OS, Unbuntu ec2 machine, Windows. Could you please verify and update the yaml file?
Thanks for the nice work!
Also are we still expecting updated results rerun on GPT-3.5-turbo? Looking forward to it.
Hi there,
When I try to run python run.py
, I encountered the error /scripts/annotate_binder_program.py
.
After I change the third line of run.py
to ROOT_DIR = './'
, the error message becomes:
File ".//scripts/annotate_binder_program.py", line 15, in
from generation.generator import Generator
ModuleNotFoundError: No module named 'generation'
Would you please take a look for me? Thanks.
Hi, when I run the "wikitq nsql annotation command" with following commands,
cd Binder
export TOKENIZERS_PARALLELISM=false
nohup python scripts/annotate_binder_program.py --dataset wikitq \
--dataset_split test \
--prompt_file templates/prompts/wikitq_binder.txt \
--n_parallel_prompts 1 \
--max_generation_tokens 512 \
--temperature 0.4 \
--sampling_n 20 \
-v \
> run-wikitq-annotation.out 2>&1 &
tail -fn 5000 run-wikitq-annotation.out
(I didn't run run.py
directly, because it reports an error: python: can't open file '/scripts/annotate_binder_program.py': [Errno 2] No such file or directory
)
the console outputs the following message:
I located the code that threw the error:
self.records_conn = self.db.get_connection()
at line 100, nsql/database.py
I haven't completed the test yet, but I'm wondering if this whole bunch of 'Database' object has no attribute 'get_connection'
errors is normal? How should I fix it?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.