guidance-ai / guidance Goto Github PK
View Code? Open in Web Editor NEWA guidance language for controlling large language models.
License: MIT License
A guidance language for controlling large language models.
License: MIT License
I tried to toy around with the examples in the README but couldn't get a program to execute
pip install guidance
to get 0.0.39import guidance
due to unspecified dependency.Python 3.11.3 | packaged by conda-forge | (main, Apr 6 2023, 08:58:31) [Clang 14.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.13.2 -- An enhanced Interactive Python. Type '?' for help.
[ins] In [1]: import guidance
...: # Set the default llm. Could also pass a different one as argument to guidance(), with guidance(llm=...)
...: guidance.llm = guidance.llms.OpenAI("text-davinci-003")
...: program = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')
...: prompt = program()
...: prompt
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 1
----> 1 import guidance
2 # Set the default llm. Could also pass a different one as argument to guidance(), with guidance(llm=...)
3 guidance.llm = guidance.llms.OpenAI("text-davinci-003")
File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/__init__.py:12
10 from ._utils import load, chain
11 from . import selectors
---> 12 import nest_asyncio
14 # allows us to start inner event loops within jupyter notebooks
15 nest_asyncio.apply()
ModuleNotFoundError: No module named 'nest_asyncio'
guidance.llm
attribute.[ins] In [1]: import guidance
...:
...: # set the default language model used to execute guidance programs
...: guidance.llm = guidance.llms.OpenAI("text-davinci-003")
...:
...: # define a guidance program that adapts a proverb
...: program = guidance("""Tweak this proverb to apply to model instructions instead.
...:
...: {{proverb}}
...: - {{book}} {{chapter}}:{{verse}}
...:
...: UPDATED
...: Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
...: - GPT {{gen 'chapter'}}:{{gen 'verse'}}""")
...:
...: # execute the program on a specific proverb
...: executed_program = program(
...: proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
...: book="Proverbs",
...: chapter=11,
...: verse=14
...: )
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[9], line 7
4 guidance.llm = guidance.llms.OpenAI("text-davinci-003")
6 # define a guidance program that adapts a proverb
----> 7 program = guidance("""Tweak this proverb to apply to model instructions instead.
8
9 {{proverb}}
10 - {{book}} {{chapter}}:{{verse}}
11
12 UPDATED
13 Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
14 - GPT {{gen 'chapter'}}:{{gen 'verse'}}""")
16 # execute the program on a specific proverb
17 executed_program = program(
18 proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
19 book="Proverbs",
20 chapter=11,
21 verse=14
22 )
File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/__init__.py:22, in Guidance.__call__(self, template, llm, cache_seed, logprobs, silent, async_mode, stream, caching, await_missing, **kwargs)
21 def __call__(self, template, llm=None, cache_seed=0, logprobs=None, silent='auto', async_mode=False, stream='auto', caching=None, await_missing=False, **kwargs):
---> 22 return Program(template, llm=llm, cache_seed=cache_seed, logprobs=logprobs, silent=silent, async_mode=async_mode, stream=stream, caching=caching, await_missing=await_missing, **kwargs)
File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/_program.py:85, in Program.__init__(self, text, llm, cache_seed, logprobs, silent, async_mode, stream, caching, await_missing, **kwargs)
83 # save the given parameters
84 self._text = text
---> 85 self.llm = llm or guidance.llm
86 self.cache_seed = cache_seed
87 self.caching = caching
AttributeError: module 'guidance' has no attribute 'llm'```
Can an example of streaming output as a generator be added? My use case is replacing langchain in production for a QA system.
The bug
AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.
...
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.
The error goes away after seting stop=False as suggested in the error message but I think this should be added to the readme. I can do the pull request if you agree.
To Reproduce
I copy pasted the example Agents with geneach
given in the readme.
import guidance
guidance.llm = guidance.llms.OpenAI('gpt-3.5-turbo')
guidance.llm.cache.clear()
prompt = guidance(
'''{{#system~}}
You are a helpful assistant
{{~/system}}
{{~#geneach 'conversation'}}
{{#user~}}
{{set 'this.user_text' (await 'user_text')}}
{{~/user}}
{{#assistant~}}
{{gen 'this.ai_text' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''')
prompt= prompt(user_text ='hi there')
prompt
System info (please complete the following information):
Is your feature request related to a problem? Please describe.
For non-executed programs storing and loading from text files works fine. But for partially executed programs with custom variables an ugly workaround was necessary. Otherwise I would get the error
Error in program: Can't set a property of a non-existing variable: conversation[-1].user_text
when trying to continue the chat after loading the text file and converting it to a program.
Describe the solution you'd like
Ideally we can just program.save(filename)
and program.load(filename)
and it just works without worrying about storing and loading the program.variables()
separately.
Describe alternatives you've considered
I was trying to do this:
import guidance
guidance.llm = guidance.llms.OpenAI('gpt-3.5-turbo')
guidance.llm.cache.clear()
prompt = guidance(
'''{{#system~}}
You are a helpful assistant
{{~/system}}
{{~#geneach 'conversation' stop=False~}}
{{#user~}}
{{set 'this.user_text' (await 'user_text')}}
{{~/user}}
{{#assistant~}}
{{gen 'this.ai_text' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''', stream=False, silent=True)
prompt = prompt(user_text ='Hello there :)')
with open('prompt.txt', 'w') as f:
f.write(str(prompt))
with open('prompt.txt', 'r') as f:
prompt = f.read()
prompt = guidance(prompt)
prompt = prompt(user_text ='I want to travel to the moon')
But got the error mentioned above.
My solution was:
def save_prompt(prompt, filename):
variables = prompt.variables()
del variables['llm']
to_store = {'text': str(prompt), 'variables': variables}
with open(filename, 'w') as f:
json.dump(to_store, f)
def load_prompt(filename):
with open(filename, 'r') as f:
loaded = json.load(f)
prompt = guidance(loaded['text'], **loaded['variables'])
return prompt
Note that the del variables['llm']
is necessary because the llm object is not serializable.
how do i call guidance but laod the models as int8 so i can fit them on even an 80Gb GPU?
installed on windows 10 with Anaconda Python 3.10.9 openai-0.27.6
ran pip install guidance
notebook tutorial.ipynb worked until section 21
Traceback
Traceback (most recent call last):
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py", line 143, in geneach
gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 310, in call
out = self.llm.caller(**call_args)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 186, in _library_call
kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 24, in prompt_to_messages
assert prompt.endswith("<|im_start|>assistant\n"), "When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting."
AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 94, in run
await self.visit(self.parse_tree)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 390, in visit
command_output = await command_function(*positional_args, **named_args)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py", line 145, in geneach
raise Exception(f"Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.")
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.
AssertionError Traceback (most recent call last)
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py:143, in geneach(list_name, stop, max_iterations, min_iterations, num_iterations, hidden, join, single_call, single_call_temperature, single_call_max_tokens, single_call_top_p, _parser_context)
142 try:
--> 143 gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
144 except Exception:
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:310, in OpenAISession.call(self, prompt, stop, stop_regex, temperature, n, max_tokens, logprobs, top_p, echo, logit_bias, token_healing, pattern, stream, cache_seed, caching)
309 call_args["logit_bias"] = {str(k): v for k,v in logit_bias.items()} # convert keys to strings since that's the open ai api's format
--> 310 out = self.llm.caller(**call_args)
312 except openai.error.RateLimitError:
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:186, in OpenAI._library_call(self, **kwargs)
185 if self.chat_mode:
--> 186 kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
187 del kwargs['prompt']
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:24, in prompt_to_messages(prompt)
21 # if len(start_tags) != len(end_tags):
22 # raise MalformedPromptException("Malformed prompt: start and end tags are not properly paired")
---> 24 assert prompt.endswith("<|im_start|>assistant\n"), "When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting."
26 pattern = r'<|im_start|>(\w+)(.*?)(?=<|im_end|>|$)'
AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
Cell In[21], line 13
1 prompt = guidance(
2 '''{{#system~}}
3 You are a helpful assistant
(...)
11 {{/assistant}}/geneach}}''')
12 {{
---> 13 prompt= prompt(user_text ='hi there')
14 prompt
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:225, in Program.call(self, **kwargs)
223 loop = asyncio.new_event_loop()
224 loop.create_task(new_program.update_display.run()) # start the display updater
--> 225 loop.run_until_complete(new_program.execute())
227 return new_program
File C:\anaconda3\lib\site-packages\nest_asyncio.py:90, in _patch_loop..run_until_complete(self, future)
87 if not f.done():
88 raise RuntimeError(
89 'Event loop stopped before Future completed.')
---> 90 return f.result()
File C:\anaconda3\lib\asyncio\futures.py:201, in Future.result(self)
199 self.__log_traceback = False
200 if self._exception is not None:
--> 201 raise self._exception.with_traceback(self._exception_tb)
202 return self._result
File C:\anaconda3\lib\asyncio\tasks.py:232, in Task.__step(failed resolving arguments)
228 try:
229 if exc is None:
230 # We use the send
method directly, because coroutines
231 # don't have __iter__
and __next__
methods.
--> 232 result = coro.send(None)
233 else:
234 result = coro.throw(exc)
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:311, in Program.execute(self)
309 else:
310 with self.llm.session(asynchronous=True) as llm_session:
--> 311 await self._executor.run(llm_session)
312 self._text = self._executor.prefix
314 # delete the executor and so mark the program as not executing
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:98, in ProgramExecutor.run(self, llm_session)
96 print(traceback.format_exc())
97 print("Error in program: ", e)
---> 98 raise e
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:94, in ProgramExecutor.run(self, llm_session)
88 self.llm_session = llm_session
89 try:
90 # first parse all the whitespace control
91 # self.whitespace_control_visit(self.parse_tree)
92
93 # now execute the program
---> 94 await self.visit(self.parse_tree)
95 except Exception as e:
96 print(traceback.format_exc())
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:423, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
421 else:
422 inner_prev_node = prev_node
--> 423 visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
424 # visited_children = [self.visit(child) for child in node.children]
426 if len(visited_children) == 1:
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:423, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
421 else:
422 inner_prev_node = prev_node
--> 423 visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
424 # visited_children = [self.visit(child) for child in node.children]
426 if len(visited_children) == 1:
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:390, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
388 # call the optionally asyncronous command
389 if inspect.iscoroutinefunction(command_function):
--> 390 command_output = await command_function(*positional_args, **named_args)
391 else:
392 command_output = command_function(*positional_args, **named_args)
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py:145, in geneach(list_name, stop, max_iterations, min_iterations, num_iterations, hidden, join, single_call, single_call_temperature, single_call_max_tokens, single_call_top_p, _parser_context)
143 gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
144 except Exception:
--> 145 raise Exception(f"Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.")
146 if gen_obj["choices"][0]["finish_reason"] == "stop":
147 break
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.
Pattern guides/regex patterns don't seem to have quite the impact one would expect
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Neko-Institute-of-Science/LLaMA-7B-HF")
model = AutoModelForCausalLM.from_pretrained("Neko-Institute-of-Science/LLaMA-7B-HF")
llama = guidance.llms.Transformers(model=model, tokenizer=tokenizer, device=5)
statement_gen = guidance("""
Today we want to say that our new tech company with the name {{gen 'companyname' max_tokens=10 pattern='[a-zA-Z]{8}'}} went public under the ticker {{gen 'ticker' pattern='[A-Z]{4}' temperature=0.3}}
""")
statement_gen(llm=llama)
I would expect this to output
Today we want to say that our new tech company with the name NameOfCompany went public under the ticker TCKR
where NameOfCompany
matches [a-zA-Z]{8}
, i.e. consisting of letters, no whitespace and exactly eight characters.
Actual output:
Today we want to say that our new tech company with the name ofTechno 2000 is going went public under the ticker TKOO
, i.e. companyname = ofTechno 2000 is going
(bad) and ticker = TKOO
(good)
Do I misunderstand pattern guides or is this an issue?
Just walking through the tutorial.
guidance.llm = guidance.llms.OpenAI("text-curie-001")
p = guidance('''Is the following sentence offensive? Yes, no, or maybe.
Sentence: {{example}}
Answer: {{#select 'answer' logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''')
p = p(example='You are such an asshole!')
What did I miss?
ps: text-davinci-003
failed as well.
When using {{select}}
, the same response is returned twice. I have tested it with text-davinci-003
, so not sure if it applies to other local models.
Here,
Expected output: Anachronism: Yes
Actual output: Anachronism: Yes Yes
Here,
Expected output: armor: leather
Actual output: armor: leatherleather
Expected output: weapon: sword
Actual output: weapon: swordsword
Does not work
Assistant: {{#select "answer"}} Yes{{or}} No{{/select}}
Error: ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)
Works
Assistant: {{#select "answer"}}Yes{{or}}No{{/select}}
Originally posted by @PenutChen in #40 (comment)
The bug
KeyError: "Command/variable 'first_question' not found!
To Reproduce
import guidance
import re
guidance.llm = guidance.llms.OpenAI("gpt-4")
role_simulator = guidance('''
{{#system~}}
You are a helpful assistant
{{~/system}}
{{#user~}}
You will answer the user as {{role}} in the following conversation. At every step, I will provide you with the user input, as well as a comment reminding you of your instructions. Never talk about the fact that you are an AI, even if the user asks you. Always answer as {{role}}.
{{#if first_question}}You can also start the conversation.{{/if}}
{{~/user}}
{{~! The assistant either starts the conversation or not, depending on if this is the first or second agent }}
{{#assistant~}}
Ok, I will follow these instructions.
{{#if first_question}}Let me start the conversation now:
{{role}}: {{first_question}}{{/if}}
{{~/assistant}}
{{~! Then the conversation unrolls }}
{{~#geneach 'conversation'}}
{{#user~}}
User: {{set 'this.input' (await 'input')}}
Comment: Remember, answer as a {{role}}. Start your utterance with {{role}}:
{{~/user}}
{{#assistant~}}
{{gen 'this.response' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''')
republican = role_simulator(role='Republican')
democrat = role_simulator(role='Democrat')
first_question = '''What do you think is the best way to stop inflation?'''
republican = republican(input=first_question, first_question=None)
democrat = democrat(input=republican["conversation"][-2]["response"].strip('Republican: '), first_question=first_question)
for i in range(2):
republican = republican(input=democrat["conversation"][-2]["response"].replace('Democrat: ', ''))
democrat = democrat(input=republican["conversation"][-2]["response"].replace('Republican: ', ''))
print('Democrat: ' + first_question)
for x in democrat['conversation'][:-1]:
print('Republican:', x['input'])
print()
print(x['response'])
System info (please complete the following information):
I'm very new to Python and tooling surrounding it, and making use of the library without documentation online is difficult.
A see that the repo contains /docs
folder with docs that are supposed to be build with sphinx
, but I can not get it to work.
I installed sphinx-build
and a bunch of libraries it said it can not import, but stuck at
PandocMissing in example_notebooks\anachronism.ipynb:
Pandoc wasn't found.
Please check that pandoc is installed:
https://pandoc.org/installing.html
even though I did both installed pandoc executable and did pip install pandoc
.
Would love to know what is the process here, and the steps to follow.
OS: Windows
Lib verions 0.0.48
Is your feature request related to a problem? Please describe.
JavaScript support is currently not available.
Describe the solution you'd like
I would like to request JavaScript support.
Describe alternatives you've considered
I have considered using alternative libraries or frameworks to achieve similar functionality in JavaScript like langchainjs and my own version. However, the Guidance library stands out due to its specific features and capabilities.
The bug
This is based on #19 which has been resolved. When I run the same code as that issue I get the error "No module named 'guidance'". This is despite having done pip install guidance
To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
"""Run with `streamlit run app.py`"""
import guidance
import streamlit as st
#import asyncio
#loop = asyncio.new_event_loop()
#asyncio.set_event_loop(loop)
st.title("Test Guidance")
guidance.llm = guidance.llms.OpenAI("text-davinci-003")
prompt = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')
st.write(prompt())
System info (please complete the following information):
MacOS
guidance.__version__
):This may be a stupid question, please forgive if so
the openAI interface obviously relies an on idenpendently existing server for gpt-3.5 and gpt-4
the Transformers interface, though, assumes guidance will load the model internally. Loading models in Transformers takes forever, even when already cached.
Is there a way to point to an existing 'guidance' server to handle guidance prompts, so I don't have to wait an entire model startup cycle every prompt test when using Transformer models like Wizard-13B?
Issue: Library fails to import on Windows 10 because termios
has no Windows package.
Possible Solutions: Instead of termios
, we must import msvcrt
. For the pseudoterminals (pty
), one could perhaps look at using andfoy/pywinpty.
Stacktrace:
File [{...}\lib\site-packages\guidance\library\__init__.py:1]({...}/lib/site-packages/guidance/library/__init__.py:1)
----> 1 from ._shell import shell
2 from ._gen import gen
3 from ._await import await_
File [{...}\lib\site-packages\guidance\library\_shell.py:7]({...}/lib/site-packages/guidance/library/_shell.py:7)
5 import os
6 import subprocess
----> 7 import pty
8 import asyncio
11 def shell(command, partial_output, safe=True):
File [{...}\lib\pty.py:12]({...}/lib/pty.py:12)
10 import os
11 import sys
---> 12 import tty
14 __all__ = ["openpty","fork","spawn"]
16 STDIN_FILENO = 0
File [{...}\lib\tty.py:5]({...}/lib/tty.py:5)
1 """Terminal utilities."""
3 # Author: Steen Lumholt.
----> 5 from termios import *
7 __all__ = ["setraw", "setcbreak"]
9 # Indexes for termios list.
ModuleNotFoundError: No module named 'termios'
Code reference: https://github.com/microsoft/guidance/blob/ff31b31177ee9ad036ef5130daada5008444c734/guidance/library/_shell.py#LL7C11-L7C11
I'm on Python 3.11
pip3 install guidance==0.0.42
Then a simple script with
import guidance
Gets error
Traceback (most recent call last):
File "scrubbed...", line 1, in <module>
import guidance
File "/opt/homebrew/lib/python3.11/site-packages/guidance/__init__.py", line 7, in <module>
from ._program import Program
File "/opt/homebrew/lib/python3.11/site-packages/guidance/_program.py", line 19, in <module>
from . import library
File "/opt/homebrew/lib/python3.11/site-packages/guidance/library/__init__.py", line 11, in <module>
from ._geneach import geneach
File "/opt/homebrew/lib/python3.11/site-packages/guidance/library/_geneach.py", line 112
partial_output( block_content = parser_context
^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
I run pip3 install guidance==0.0.41
and import works.
A minor bug is causing the AzureOpenAI llm __init__
to fail.
Steps to reproduce:
import guidance
guidance.llms.AzureOpenAI("gpt-3.5-turbo")
Error message:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[17], line 3
1 import guidance
----> 3 guidance.llms.AzureOpenAI("gpt-3.5-turbo")
File [~/medical-writing/.venv/lib/python3.10/site-packages/guidance/llms/_azure_openai.py:34](https://file+.vscode-resource.vscode-cdn.net/Users/kevinblissett/medical-writing/~/medical-writing/.venv/lib/python3.10/site-packages/guidance/llms/_azure_openai.py:34), in AzureOpenAI.__init__(self, model, client_id, authority, caching, max_retries, max_calls_per_min, token, endpoint, scopes, temperature, chat_mode)
31 if os.path.exists(self._token_cache_path):
32 self._token_cache.deserialize(open(self._token_cache_path, 'r').read())
---> 34 self._rest_headers["X-ModelType"] = self.model_name
AttributeError: 'AzureOpenAI' object has no attribute '_rest_headers'
installed on windows 10 with Anaconda Python 3.10.9 openai-0.27.6
ran pip install guidance
notebook tutorial.ipynb worked until "Using Tools" section
IncompleteParseError Traceback (most recent call last)
Cell In[8], line 87
35 program = guidance('''
36 {{#system~}}
37 You are a helpful assistant.
(...)
82 {{~/assistant}}
83 ''')
85 query = "What is Facebook's stock price right now?"
---> 87 program = program(
88 user_query=query,
89 search=search,
90 is_search=is_search,
91 practice=practice_round
92 )
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:204, in Program.call(self, **kwargs)
195 new_program = Program(
196 text=self.marked_text,
197
(...)
200 {{k: v if callable(v) else copy.deepcopy(v) for k,v in self._variables.items()}, **kwargs}
201 )
203 # create an executor for the new program (this also marks the program as executing)
--> 204 new_program._executor = ProgramExecutor(new_program)
206 # if we are in async mode schedule the program in the current event loop
207 if new_program.async_mode:
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:50, in ProgramExecutor.init(self, program)
48 except parsimonious.exceptions.ParseError as e:
49 self._check_for_simple_error(text)
---> 50 raise e
File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:47, in ProgramExecutor.init(self, program)
45 # parse the program text
46 try:
---> 47 self.parse_tree = grammar.parse(text)
48 except parsimonious.exceptions.ParseError as e:
49 self._check_for_simple_error(text)
File ~\AppData\Roaming\Python\Python310\site-packages\parsimonious\grammar.py:112, in Grammar.parse(self, text, pos)
106 """Parse some text with the :term:default rule
.
107
108 :arg pos: The index at which to start parsing
109
110 """
111 self._check_default_rule()
--> 112 return self.default_rule.parse(text, pos=pos)
File ~\AppData\Roaming\Python\Python310\site-packages\parsimonious\expressions.py:143, in Expression.parse(self, text, pos)
141 node = self.match(text, pos=pos)
142 if node.end < len(text):
--> 143 raise IncompleteParseError(text, node.end, self)
144 return node
IncompleteParseError: Rule 'template' matched in its entirety, but it didn't consume all the text. The non-matching portion of the text begins with '{{#each practice}}
{' (line 14, column 1).
I'm not sure if this is already possible, if so maybe an example could be included in the README? I was not able to find functionality that enables this.
It would be nice if all inferences in {{each}} blocks are available as output variables. Currently only the last generation is available as a named variable.
For example say the following guidance program is executed:
choices = ['odd', 'correct']
program = guidance("""
You are a helpful and terse assistant.
You have to check each of the following sentences for odd grammar and give a judgement.
{{#each sentences}}
Sentence {{@index}}: ```{{this}}```: Judgement: (Respond with 'correct' or 'odd' only!) {{select 'judgement' options=choices}}
{{/each}}""")
outputs = program(
sentences=a_list_of_sentences,
choices=choices
)().variables
print(outputs)
Will only output the last generated 'judgement':
{'llm': <guidance.llms._openai.OpenAI object at 0x0000025BA1A44B90>, 'sentences': ['sentence1', 'sentence2', 'sentence3'], 'choices': ['odd', 'correct'], 'judgement': 'odd'}
A much better result would be:
{'llm': <guidance.llms._openai.OpenAI object at 0x0000025BA1A44B90>, 'sentences': ['sentence1', 'sentence2', 'sentence3'], 'choices': ['odd', 'correct'], 'judgement': ['odd', 'correct', 'odd']}
I'm trying to use a transformers model like so
model = guidance.llms.Transformers('stabilityai/stablelm-base-alpha-3b', device=0)
but getting the following error when I try to call the guidance program:
AssertionError: You must provide an OpenAI API key to use the OpenAI LLM. Either pass it in the constructor, set the OPENAI_API_KEY environment variable, or create the file ~/.openai_api_key with your key in it.
Message continues: Please post a GitHub issue since this should not happen :)
So here I am.
Raised from:
File "[..]\guidance\_program_executor.py", line 94, in run
await self.visit(self.parse_tree)
File "[..]\guidance\_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "[..]\guidance\_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "[..]\guidance\_program_executor.py", line 390, in visit
command_output = await command_function(*positional_args, **named_args)
File "[..]\guidance\library\_select.py", line 138
Running the following from tutorial.ipynb
:
import guidance
guidance.llm = guidance.llms.OpenAI("text-davinci-003")
prompt = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: {{example}}
Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''')
prompt = prompt(example="I hate pineapple on pizza, so that I am ready to kill because of it") //not my actual opinion, was trying to get it to reply "Yes".
print(prompt)
option_logprobs
is:
{' Yes': -1000, ' No': -1000, ' Maybe': -1000}
Hello,
I've been using your role_simulator function and encountered a KeyError for the 'first_question' variable. The error message suggests that 'first_question' should have been passed as an argument when calling the program, or it should have been set as a default value when creating the program.
Here's the code that's causing the error:
role_simulator = guidance('''
...
{{#if first_question}}You can also start the conversation.{{/if}}
...
{{#if first_question}}Let me start the conversation now:
{{role}}: {{first_question}}{{/if}}
...
''')
republican = role_simulator(role='Republican')
democrat = role_simulator(role='Democrat')
first_question = '''What do you think is the best way to stop inflation?'''
...
The error occurs because 'first_question' is not defined at the time when the role_simulator function is called.
What's the best way to take advantage of the streaming capabilities of hugging face transformers in this library? I see that streaming is all done internally but it's unclear how its exposed to the library user (me)
I figured out a few hacky methods. The only promising one so far is using list_append with a List-like object. Code needs a lot of work. append is called once at the beginning, and then __setitem__
is repeatedly called with -1, so the whole thing works more like a callback itself (output down below):
class dynlist:
def __init__(self, callback):
self.data = list()
self.callback = callback
def append(self, item):
self.data.append(item)
self.callback(self.data)
def __setitem__(self, key, val):
self.data.__setitem__(key, val)
self.callback(self.data)
def update_x(session_id, data: List[str]):
....
prompt = guidance("...{{~gen "response" list_append=True temperature=0.4 top_p=0.9}}")
my_session_id = ...
response = dynlist(functools.partial(update_x, my_session_id))
await prompt("...", llm=llm, stream=True, async_mode=True, response=response)
list-append:
set-item: -1 I
set-item: -1 I like
set-item: -1 I like hanging
set-item: -1 I like hanging out
set-item: -1 I like hanging out with
set-item: -1 I like hanging out with you
set-item: -1 I like hanging out with you.
set-item: -1 I like hanging out with you.
set-item: -1 I like hanging out with you.
Steps to reproduce:
gpt-4
to gpt-3.5-turbo
Error
IndexError Traceback (most recent call last)
Cell In[14], line 45
3 return options[best]
5 create_plan = guidance('''{{#system~}}
6 You are a helpful assistant.
7 {{[~/system](https://file+.vscode-resource.vscode-cdn.net/Users/user/guidance/notebooks/~/system)}}
(...)
42 {{gen 'plan' max_tokens=500}}
43 {{[~/assistant](https://file+.vscode-resource.vscode-cdn.net/Users/user/guidance/notebooks/~/assistant)}}''')
---> 45 out = create_plan(goal='read more books', parse_best=parse_best)
46 out
File [/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/guidance/_program.py:216](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/guidance/_program.py:216), in Program.__call__(self, **kwargs)
214 loop = asyncio.new_event_loop()
215 loop.create_task(new_program.update_display.run()) # start the display updater
--> 216 loop.run_until_complete(new_program.execute())
218 return new_program
File [/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nest_asyncio.py:90](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
87 if not f.done():
88 raise RuntimeError(
89 'Event loop stopped before Future completed.')
---> 90 return f.result()
...
1 def parse_best(prosandcons, options):
----> 2 best = int(re.findall(r'Best=(\d+)', prosandcons)[0])
3 return options[best]
IndexError: list index out of range
Hello,
Like #11 , so I suppose the resolution will be similar. Creating this for future reference for other searches.
Pure intuition, Streamlit spawns worker threads to run the Python Script in, which may be decorrelated from any Asyncio loop you would start on initialization on the main thread through nest_asyncio
. Though just creating/setting an asyncio loop in the script also doesn't register correctly :/
Using guidance==0.0.44 and streamlit==1.22.0
"""Run with `streamlit run app.py`"""
import guidance
import streamlit as st
#import asyncio
#loop = asyncio.new_event_loop()
#asyncio.set_event_loop(loop)
st.title("Test Guidance")
guidance.llm = guidance.llms.OpenAI("text-davinci-003")
prompt = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')
st.write(prompt())
File "guidance\__init__.py", line 15, in <module>
nest_asyncio.apply()
File "nest_asyncio.py", line 16, in apply
loop = loop or asyncio.get_event_loop()
File "nest_asyncio.py", line 45, in _get_event_loop
loop = events.get_event_loop_policy().get_event_loop()
File "lib\asyncio\events.py", line 656, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'ScriptRunner.scriptThread'.
Have a nice day
Thanks ! Love the token control.
Just wish your example was Keynes v Hayek.
This project is seriously embarrassing. The sooner someone in management comes to their senses and gets this deleted the better off you will all be.
This library from HF is pretty great and I get use out of it in production settings for LLMs. Would love to figure out how to integrate a system like this for LLM safety with it so I can use HF models, get dynamic batching, and be able to stream tokens with the guidance library!
The bug
Although producing individual fields as numbers seems to work fine (although invariably the actual parsed variable is still a string as seen below), and arrays of strings also seem to work fine, arrays of numbers do not seem work properly.
I'd love to be able to fully understand why this is happening and am also very happy to attempt to fix this if someone is able to point me in the right direction.
To Reproduce
import guidance
import os
os.environ["OPENAI_API_KEY"] = "sk-..."
llm = guidance.llms.OpenAI("text-davinci-003")
def example():
# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
# define the prompt
program = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
"description": "{{description}}",
"name": "{{gen 'name'}}",
"age": {{gen 'age' stop=','}},
"armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
"weapon": "{{select 'weapon' options=valid_weapons}}",
"class": "{{gen 'class'}}",
"mantra": "{{gen 'mantra'}}",
"strength": {{gen 'strength' stop=','}},
"item_ids": [{{#geneach 'numbers' num_iterations=3}}
{{gen 'this'}},{{/geneach}}
]
}```""")
# execute the prompt
result = program(
description="A quick and nimble fighter.", valid_weapons=valid_weapons, llm=llm
)
print(result)
print(result.variables())
return result
if __name__ == "__main__":
example()
The outputs of the print statements are as follows:
print(result)
The following is a character profile for an RPG game in JSON format.
```json
{
"description": "A quick and nimble fighter.",
"name": "Fighter",
"age": 25,
"armor": "leather",
"weapon": "sword",
"class": "warrior",
"mantra": "Strength in numbers.",
"strength": 8,
"item_ids": [
1,
2,
3
]
},
]
}```
print(result.variables())
{
'llm': <guidance.llms._openai.OpenAI object at 0x101205df0>,
'description': 'A quick and nimble fighter.',
'valid_weapons': ['sword', 'axe', 'mace', 'spear', 'bow', 'crossbow'],
'name': 'Fighter',
'age': ' 25',
'armor': 'leather',
'weapon': 'sword',
'class': 'warrior',
'mantra': 'Strength in numbers.',
'strength': ' 8',
'numbers': [' 1', ' 2', ' 3\n ]\n}']
}
Most of the problem here seems to be that the generation of the final number in the array goes a bit wonky - not sure if I could tweak my prompt, or if it's a deeper issue.
System info
I suspect that the select
statement is broken whenever some the options are longer than 1 token, due to a misunderstanding of how top_logprobs
work. The crucial property is that the i
th entry of top_logprobs
contains the top logprobs assuming that the i-1
first tokens are as in the returned completion.
To exemplify the problem, say that your select has two options YesNo
and NoYes
for strange reasons. These are both 2 tokens and consist of tokens Yes
and No
, so those are the only ones biased to be output by the model. They're both weird, though, and it's much more likely that the model returns e.g. YesYes
. So let's say it does.
What your select statement then should be computing is:
logprob(YesNo)
: The logprob of first generating Yes
plus then generating No
.logprob(NoYes)
: The logprob of first generating No
plus then generating Yes
.Your code gets the first one right. But for point two, what you're computing is rather:
X
: The logprob of first generating No
, plus the logprob of generating Yes
after first generating Yes
.It's unclear how X
will compare to the true logprob(NoYes)
, but the end result is that what gets selected has little relation to what the model truly finds most probably.
Note that the problem compounds the longer the options are.
There is an issue with this notebook:
The one-shot example for the simple equation is incorrect:
For example, if the solution is x=2, say SOLUTION = [3].
I believe the numbers there are meant to align to each other. Not sure if it would impact the evaluation results, but worth being accurate.
Hi,
I'm trying to run guidance within a Gradio app. I'm not sure about the async framework behind Gradio but it seems to be AnyIO and is using some worker threads.
The problem I am experiencing is that when I attempt to run a program I get an error that there is no current event loop in the thread. I am not trying to use the program asynchronously (the flag passed in).
File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 194, in __call__
new_program = Program(
File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 110, in __init__
self._execute_complete = asyncio.Event() # fires when the program is done executing to resolve __await__
File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/locks.py", line 177, in __init__
self._loop = events.get_event_loop()
File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/nest_asyncio.py", line 45, in _get_event_loop
loop = events.get_event_loop_policy().get_event_loop()
File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.
Any help is welcome.
I skimmed the documentation and notebooks, and I found some similarities between guidance and langchain. Can you please explain the reasons one should use guidance over langchain? More importantly, some of the features (e.g., token healing) seem not to be available for GPT-4 (or closed-source models in general). Is that true?
Error was raised running the following:
prompt = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "Nein", or "Vielleicht".
Sentence: {{example}}
Answer:{{#select "answer" logprobs='logprobs'}} Ja{{or}} Nein{{or}} Vielleicht{{/select}}''')
prompt = prompt(example='I hate tacos.')
prompt
I tried multiple variations. When the answer format specified at the beginning of the prompt and the options inside the select tag vary then very often this error happens.
When importing guidance from a read-only filesystem, there are a couple of issues:
Opening the log.txt file (in https://github.com/microsoft/guidance/blob/main/guidance/_utils.py#L10) will always raise an exception as it can't be opened for writing. Can this be switched to use the standard python logging
module instead?
Setting caching=False
still tries to create the cache directory. Can be worked around by setting os.environ['XDG_CACHE_HOME'] = '/tmp'
(or somewhere that is definitely writeable) before importing, but isn't ideal.
System info:
I think that the output for hidden generation on README.me is wrong. Can someone double check it?
I'm trying to generate some text using a modified version of the main example and am running into an issue where the model is picking a token containing a non-numerical character for the 'chapter' variable despite that not matching the pattern. Here's my code:
prompt = '''
Tweak this proverb to apply to machine learning model instructions instead.
{{proverb}}
- {{book}} {{chapter}}:{{verse}}
UPDATED
Where there is no guidance{{gen 'rewrite' stop='- '}}
- GPT {{gen 'chapter' pattern='[0-9]' max_tokens=1}}:{{gen 'verse' pattern='[0-9]+' stop='\\n'}}
'''[1:-1]
model = guidance.llms.Transformers('stabilityai/stablelm-base-alpha-3b', device=0)
program = guidance(prompt, llm = model)
executed_program = program(
proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
book="Proverbs",
chapter=11,
verse=14,
)
print(executed_program)
print(executed_program["chapter"])
and the generated output:
Tweak this proverb to apply to machine learning model instructions instead.
Where there is no guidance, a people falls,
but in an abundance of counselors there is safety.
- Proverbs 11:14
UPDATED
Where there is no guidance, a people falls, but in an abundance of counselors there is safety.
- GPT 2::2
2:
I should also add that I saw a similar issue earlier today with pattern not working as intended so I made sure I was on the latest version just before running this just now.
Is your feature request related to a problem? Please describe.
Seeing that guidance has some ability to generate syntax specific outputs, I'm curious to know if the team has thoughts on generating more complex programming language grammars.
Describe the solution you'd like
When the points to a reference syntax, the llm output would be syntactically correct specific to the given language grammar.
Describe alternatives you've considered
None
Additional context
I have a bunch of domain specific languages I've built and I've been thinking that if I can basically validate each token that comes out of the stream against a language grammar, it should basically ensure that it works correctly all the time. It's kind of an extension of the valid options you have in the examples.
I'm happy to help build this but I'd need some help is outlining this.
Issue: we can't import ggml (ex: llama-cpp) models.
The bug
ValueError: No valid option generated in #select
To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
import os
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
import guidance
d = '''
User: Question: Which is the best MCU film?
Assistant: Yes. I need to search the web for the best MCU film.
User: Question: {{question}}
Assistant {{#select "answer"}} Yes{{or}} No{{/select}}
'''
chatgpt = guidance.llms.OpenAI("text-davinci-003")
prompt = guidance(d)
prompt = prompt(llm = chatgpt, question = "What is Google's Headquarter address?")
Traceback (most recent call last):
File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 94, in run
await self.visit(self.parse_tree)
File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 390, in visit
command_output = await command_function(*positional_args, **named_args)
File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/library/_select.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/library/_select.py)", line 138, in select
raise ValueError("No valid option generated in #select! Please post a GitHub issue since this should not happen :)")
ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)
Error in program: No valid option generated in #select! Please post a GitHub issue since this should not happen :)
Output exceeds the [size limit](command:workbench.action.openSettings?%5B%22notebook.output.textLineLimit%22%5D). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?3f937783-9df6-4db8-8cc8-185d951c685d)---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[7], line 20
16 chatgpt = guidance.llms.OpenAI("text-davinci-003")
18 prompt = guidance(d)
---> 20 prompt = prompt(llm = chatgpt, question = "What is Google's Headquarter address?")
File [~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program.py:225](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program.py:225), in Program.__call__(self, **kwargs)
223 loop = asyncio.new_event_loop()
224 loop.create_task(new_program.update_display.run()) # start the display updater
--> 225 loop.run_until_complete(new_program.execute())
227 return new_program
File [~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/nest_asyncio.py:90](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
87 if not f.done():
88 raise RuntimeError(
89 'Event loop stopped before Future completed.')
---> 90 return f.result()
File [~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/futures.py:201](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/futures.py:201), in Future.result(self)
199 self.__log_traceback = False
200 if self._exception is not None:
--> 201 raise self._exception
202 return self._result
...
137 if max(option_logprobs.values()) <= -1000:
--> 138 raise ValueError("No valid option generated in #select! Please post a GitHub issue since this should not happen :)")
140 partial_output(selected_option)
ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)
System info (please complete the following information):
guidance.__version__
): guidance-0.0.47My results with LLaMA-7B don't match the tutorial, please see some examples below.
import guidance
guidance.llm = guidance.llms.Transformers('Neko-Institute-of-Science/LLaMA-7B-HF', device=0)
# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
# define the prompt
character_maker = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
"id": "{{id}}",
"description": "{{description}}",
"name": "{{gen 'name'}}",
"age": {{gen 'age' pattern='[0-9]+' stop=','}},
"armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
"weapon": "{{select 'weapon' options=valid_weapons}}",
"class": "{{gen 'class'}}",
"mantra": "{{gen 'mantra' temperature=0.7}}",
"strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
"items": [{{#geneach 'items' num_iterations=5 join=', '}}"{{gen 'this' temperature=0.7}}"{{/geneach}}]
}```""")
# generate a character
character_maker(
id="e1f491f7-7ab8-4dac-8c20-c92b5e7d883d",
description="A quick and nimble fighter.",
valid_weapons=valid_weapons
)
Result:
The following is a character profile for an RPG game in JSON format.
{ "id": "e1f491f7-7ab8-4dac-8c20-c92b5e7d883d", "description": "A quick and nimble fighter.", "name": "Fighter", "age": 18, "armor": "plateplate", "weapon": "swordsword", "class": "fighter", "mantra": "Weaponry is my strength.", "strength": 10, "items": ["sword", "armor", "helmet", "sword", "fishing pole"] }
select
for the armor and weapon duplicates the output.
```python prompt = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe". Sentence: {{example}} Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''') prompt = prompt(example='I hate tacos') prompt ```
Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: I hate tacos
Answer: Yes Yes
prompt = guidance('''Generate a response to the following email:
{{email}}.
Response:{{gen "response"}}
Is the response above offensive in any way? Please answer with a single word, either "Yes" or "No".
Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{/select}}''')
prompt = prompt(email='I hate tacos')
prompt
Generate a response to the following email:
I hate tacos.
Response: I hate tacos too.
I hate tacos.
Response: I hate tacos too. I hate tacos too.
I hate tacos too.
Response: I hate tacos too. I hate tacos too. I hate tacos too.
I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hateIs the response above offensive in any way? Please answer with a single word, either "Yes" or "No".
Answer: Yes Yes
select
duplicates the Yes
The bug
Bump into this error:
File "/home/lucas/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/library/_role.py", line 12, in role
partial_output(parser.program.llm.role_start(name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Transformers' object has no attribute 'role_start'
Error in program: 'Transformers' object has no attribute 'role_start'
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[5], line 21
1 program = guidance('''
2 {{#system}}You are an expert unix systems admin.{{/system}}
3
(...)
19 {{[~/assistant](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/assistant)}}
20 ''', llm=chat_llm)
---> 21 out = program(os="Linux", unique=lambda x: list(set(x)), caching=False)
File [~/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/_program.py:225](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/_program.py:225), in Program.__call__(self, **kwargs)
223 loop = asyncio.new_event_loop()
224 loop.create_task(new_program.update_display.run()) # start the display updater
--> 225 loop.run_until_complete(new_program.execute())
227 return new_program
File [~/miniconda3/envs/seed/lib/python3.11/site-packages/nest_asyncio.py:90](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
87 if not f.done():
88 raise RuntimeError(
89 'Event loop stopped before Future completed.')
---> 90 return f.result()
File [~/miniconda3/envs/seed/lib/python3.11/asyncio/futures.py:203](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/asyncio/futures.py:203), in Future.result(self)
201 self.__log_traceback = False
...
18 next_next_node=_parser_context["next_next_node"]
19 )
21 # send the role-end special tokens
AttributeError: 'Transformers' object has no attribute 'role_start'
Run
To Reproduce(direct copy from use_clear_syntax.ipynb
)
chat_llm = guidance.llms.Transformers("stabilityai/stablelm-tuned-alpha-3b", device=1)
program = guidance('''
{{#system}}You are an expert unix systems admin.{{/system}}
{{#user~}}
What are the most common commands used in the {{os}} operating system?
{{~/user}}
{{#assistant~}}
{{#block hidden=True~}}
Here is a common command: "{{gen 'commands' stop='"' n=10 max_tokens=20 temperature=0.7}}"
{{~/block~}}
{{#each (unique commands)}}
{{@index}}. {{this}}
{{~/each}}
Perhaps the most useful command from that list is: "{{gen 'cool_command'}}", because{{gen 'cool_command_desc' max_tokens=100 stop="\\n"}}
On a scale of 1-10, it has a coolness factor of: {{gen 'coolness' pattern="[0-9]+"}}.
{{~/assistant}}
''', llm=chat_llm)
out = program(os="Linux", unique=lambda x: list(set(x)), caching=False)
System info (please complete the following information):
guidance.__version__
): 0.0.47Can there be a way to load a transformer from huggingface in bits and bytes. That could make the model loading easier. I might add this after work but it would be nice to have.
I'm getting an error I think on trying to read the response from the llm. I'm using OpenAI via self.guide = guidance.llms.OpenAI(model='gpt-3.5-turbo')
, and certain questions trigger the traceback below - not all questions, it seems to be fairly arbitrary. Looking at the code, it possibly needs a guard against empty string responses?
agent | File "/usr/local/lib/python3.9/site-packages/guidance/library/_gen.py", line 133, in gen
agent | gen_obj = await parser.llm_session(
agent | File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 310, in __call__
agent | out = self.llm.caller(**call_args)
agent | File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 192, in _library_call
agent | out = add_text_to_chat_mode(out)
agent | File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 57, in add_text_to_chat_mode
agent | c['text'] = c['message']['content']
agent | File "/usr/local/lib/python3.9/site-packages/openai/openai_object.py", line 71, in __setitem__
agent | raise ValueError(
agent | ValueError: You cannot set text to an empty string. We interpret empty strings as None in requests.You may set {
agent | "finish_reason": "stop",
agent | "index": 0,
agent | "message": {
agent | "content": "",
agent | "role": "assistant"
agent | }
agent | }.text = None to delete the property
Hi,
Firstly, thanks for sharing the idea of prompt templating.
I'm playing with use_clear_syntax.ipynb in notebooks/art_of_prompt_design/use_clear_syntax.ipynb
,
and trying to understand how it works with openai API using the following code:
https://gist.github.com/wooparadog/5765952394c33c5c9091f2835467eeec
But I got an exception:
Traceback (most recent call last):
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 94, in run
await self.visit(self.parse_tree)
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 434, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 434, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 394, in visit
command_output = await command_function(*positional_args, **named_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/library/_system.py", line 13, in system
return await role(name="system", hidden=hidden, _parser_context=_parser_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/library/_role.py", line 4, in role
block_content = _parser_context['block_content']
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
Is there anything wrong with my test case? Thanks again
Is your feature request related to a problem? Please describe.
I know this lib is new. It would be nice to see more LLMs to be added, especially Anthropic Claudes
Describe the solution you'd like
More LLMs support
Describe alternatives you've considered
N/A
Additional context
Claudes-100k is super useful.
I would like to understand how Guidance compares to LMQL when reading the docs. Can we gather a list of additional features?
The bug
A clear and concise description of what the bug is.
ValueError: Fail to create device flow. Err: {
"error": "invalid_request",
"error_codes": [
900144
],
....
}
To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
# put your code snippet here
System info (please complete the following information):
guidance.__version__
):A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.