guidance-ai / guidance Goto Github PK

I'm trying to use a transformers model like so
model = guidance.llms.Transformers('stabilityai/stablelm-base-alpha-3b', device=0)
but getting the following error when I try to call the guidance program:
AssertionError: You must provide an OpenAI API key to use the OpenAI LLM. Either pass it in the constructor, set the OPENAI_API_KEY environment variable, or create the file ~/.openai_api_key with your key in it.

I'm not sure if this is already possible, if so maybe an example could be included in the README? I was not able to find functionality that enables this.

It would be nice if all inferences in {{each}} blocks are available as output variables. Currently only the last generation is available as a named variable.

For example say the following guidance program is executed:

choices = ['odd', 'correct']

program = guidance("""
You are a helpful and terse assistant.

You have to check each of the following sentences for odd grammar and give a judgement.

{{#each sentences}}
Sentence {{@index}}: ```{{this}}```: Judgement: (Respond with 'correct' or 'odd' only!) {{select 'judgement' options=choices}}
{{/each}}""")

outputs = program(
    sentences=a_list_of_sentences,
    choices=choices
)().variables

print(outputs)

Will only output the last generated 'judgement':

{'llm': <guidance.llms._openai.OpenAI object at 0x0000025BA1A44B90>, 'sentences': ['sentence1', 'sentence2', 'sentence3'], 'choices': ['odd', 'correct'], 'judgement': 'odd'}

A much better result would be:

{'llm': <guidance.llms._openai.OpenAI object at 0x0000025BA1A44B90>, 'sentences': ['sentence1', 'sentence2', 'sentence3'], 'choices': ['odd', 'correct'], 'judgement': ['odd', 'correct', 'odd']}

The bug
KeyError: "Command/variable 'first_question' not found!

To Reproduce

import guidance
import re
guidance.llm = guidance.llms.OpenAI("gpt-4")
role_simulator = guidance('''
{{#system~}}
You are a helpful assistant
{{~/system}}
                                           
{{#user~}}
You will answer the user as {{role}} in the following conversation. At every step, I will provide you with the user input, as well as a comment reminding you of your instructions. Never talk about the fact that you are an AI, even if the user asks you. Always answer as {{role}}.
{{#if first_question}}You can also start the conversation.{{/if}}
{{~/user}}
                                           
{{~! The assistant either starts the conversation or not, depending on if this is the first or second agent }}
{{#assistant~}}
Ok, I will follow these instructions.
{{#if first_question}}Let me start the conversation now:
{{role}}: {{first_question}}{{/if}}
{{~/assistant}}

{{~! Then the conversation unrolls }}
{{~#geneach 'conversation'}}
{{#user~}}
User: {{set 'this.input' (await 'input')}}
Comment: Remember, answer as a {{role}}. Start your utterance with {{role}}:
{{~/user}}

{{#assistant~}}
{{gen 'this.response' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''')
                          
republican = role_simulator(role='Republican')
democrat = role_simulator(role='Democrat')

first_question = '''What do you think is the best way to stop inflation?'''
republican = republican(input=first_question, first_question=None)
democrat = democrat(input=republican["conversation"][-2]["response"].strip('Republican: '), first_question=first_question)

for i in range(2):
    republican = republican(input=democrat["conversation"][-2]["response"].replace('Democrat: ', ''))
    democrat = democrat(input=republican["conversation"][-2]["response"].replace('Republican: ', ''))
print('Democrat: ' + first_question)
for x in democrat['conversation'][:-1]:
    print('Republican:', x['input'])
    print()
    print(x['response'])

System info (please complete the following information):

• OS: Ubuntu
• Guidance Version: 0.0.47

When using {{select}}, the same response is returned twice. I have tested it with text-davinci-003, so not sure if it applies to other local models.

Example 1:

Here,
Expected output: Anachronism: Yes
Actual output: Anachronism: Yes Yes

Example 2:

Here,

1. Expected output: armor: leather
   Actual output: armor: leatherleather

2. Expected output: weapon: sword
   Actual output: weapon: swordsword

The bug
A clear and concise description of what the bug is.

ValueError: Fail to create device flow. Err: {
"error": "invalid_request",
"error_codes": [
900144
],
....
}

To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

# put your code snippet here

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.):
• Guidance Version (guidance.__version__):

I tried to toy around with the examples in the README but couldn't get a program to execute

• Fresh Python 3.11 virtual env (via mamba, i.e., conda) on macOS
• pip install guidance to get 0.0.39

Unable to import guidance due to unspecified dependency.

Python 3.11.3 | packaged by conda-forge | (main, Apr  6 2023, 08:58:31) [Clang 14.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.13.2 -- An enhanced Interactive Python. Type '?' for help.

[ins] In [1]: import guidance
         ...: # Set the default llm. Could also pass a different one as argument to guidance(), with guidance(llm=...)
         ...: guidance.llm = guidance.llms.OpenAI("text-davinci-003")
         ...: program = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')
         ...: prompt = program()
         ...: prompt
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 import guidance
      2 # Set the default llm. Could also pass a different one as argument to guidance(), with guidance(llm=...)
      3 guidance.llm = guidance.llms.OpenAI("text-davinci-003")

File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/__init__.py:12
     10 from ._utils import load, chain
     11 from . import selectors
---> 12 import nest_asyncio
     14 # allows us to start inner event loops within jupyter notebooks
     15 nest_asyncio.apply()

ModuleNotFoundError: No module named 'nest_asyncio'

Unable to execute program due to missing guidance.llm attribute.

[ins] In [1]: import guidance
         ...:
         ...: # set the default language model used to execute guidance programs
         ...: guidance.llm = guidance.llms.OpenAI("text-davinci-003")
         ...:
         ...: # define a guidance program that adapts a proverb
         ...: program = guidance("""Tweak this proverb to apply to model instructions instead.
         ...:
         ...: {{proverb}}
         ...: - {{book}} {{chapter}}:{{verse}}
         ...:
         ...: UPDATED
         ...: Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
         ...: - GPT {{gen 'chapter'}}:{{gen 'verse'}}""")
         ...:
         ...: # execute the program on a specific proverb
         ...: executed_program = program(
         ...:     proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
         ...:     book="Proverbs",
         ...:     chapter=11,
         ...:     verse=14
         ...: )
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 7
      4 guidance.llm = guidance.llms.OpenAI("text-davinci-003")
      6 # define a guidance program that adapts a proverb
----> 7 program = guidance("""Tweak this proverb to apply to model instructions instead.
      8
      9 {{proverb}}
     10 - {{book}} {{chapter}}:{{verse}}
     11
     12 UPDATED
     13 Where there is no guidance{{gen 'rewrite' stop="\\n-"}}
     14 - GPT {{gen 'chapter'}}:{{gen 'verse'}}""")
     16 # execute the program on a specific proverb
     17 executed_program = program(
     18     proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
     19     book="Proverbs",
     20     chapter=11,
     21     verse=14
     22 )

File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/__init__.py:22, in Guidance.__call__(self, template, llm, cache_seed, logprobs, silent, async_mode, stream, caching, await_missing, **kwargs)
     21 def __call__(self, template, llm=None, cache_seed=0, logprobs=None, silent='auto', async_mode=False, stream='auto', caching=None, await_missing=False, **kwargs):
---> 22     return Program(template, llm=llm, cache_seed=cache_seed, logprobs=logprobs, silent=silent, async_mode=async_mode, stream=stream, caching=caching, await_missing=await_missing, **kwargs)

File ~/.local/state/mamba/envs/py311/lib/python3.11/site-packages/guidance/_program.py:85, in Program.__init__(self, text, llm, cache_seed, logprobs, silent, async_mode, stream, caching, await_missing, **kwargs)
     83 # save the given parameters
     84 self._text = text
---> 85 self.llm = llm or guidance.llm
     86 self.cache_seed = cache_seed
     87 self.caching = caching

AttributeError: module 'guidance' has no attribute 'llm'```

Can an example of streaming output as a generator be added? My use case is replacing langchain in production for a QA system.

This may be a stupid question, please forgive if so

the openAI interface obviously relies an on idenpendently existing server for gpt-3.5 and gpt-4

the Transformers interface, though, assumes guidance will load the model internally. Loading models in Transformers takes forever, even when already cached.

Is there a way to point to an existing 'guidance' server to handle guidance prompts, so I don't have to wait an entire model startup cycle every prompt test when using Transformer models like Wizard-13B?

There is an issue with this notebook:

https://github.com/microsoft/guidance/blob/main/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb

The one-shot example for the simple equation is incorrect:

For example, if the solution is x=2, say SOLUTION = [3].

I believe the numbers there are meant to align to each other. Not sure if it would impact the evaluation results, but worth being accurate.

I'm very new to Python and tooling surrounding it, and making use of the library without documentation online is difficult.

A see that the repo contains /docs folder with docs that are supposed to be build with sphinx, but I can not get it to work.
I installed sphinx-build and a bunch of libraries it said it can not import, but stuck at

PandocMissing in example_notebooks\anachronism.ipynb:
Pandoc wasn't found.
Please check that pandoc is installed:
https://pandoc.org/installing.html

even though I did both installed pandoc executable and did pip install pandoc.

Would love to know what is the process here, and the steps to follow.

OS: Windows
Lib verions 0.0.48

I suspect that the select statement is broken whenever some the options are longer than 1 token, due to a misunderstanding of how top_logprobs work. The crucial property is that the ith entry of top_logprobs contains the top logprobs assuming that the i-1 first tokens are as in the returned completion.

To exemplify the problem, say that your select has two options YesNo and NoYes for strange reasons. These are both 2 tokens and consist of tokens Yes and No, so those are the only ones biased to be output by the model. They're both weird, though, and it's much more likely that the model returns e.g. YesYes. So let's say it does.

What your select statement then should be computing is:

• logprob(YesNo): The logprob of first generating Yes plus then generating No.
• logprob(NoYes): The logprob of first generating No plus then generating Yes.

Your code gets the first one right. But for point two, what you're computing is rather:

• X: The logprob of first generating No, plus the logprob of generating Yes after first generating Yes.

It's unclear how X will compare to the true logprob(NoYes), but the end result is that what gets selected has little relation to what the model truly finds most probably.

Note that the problem compounds the longer the options are.

A minor bug is causing the AzureOpenAI llm __init__ to fail.

Steps to reproduce:

import guidance

guidance.llms.AzureOpenAI("gpt-3.5-turbo")

Error message:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[17], line 3
      1 import guidance
----> 3 guidance.llms.AzureOpenAI("gpt-3.5-turbo")

File [~/medical-writing/.venv/lib/python3.10/site-packages/guidance/llms/_azure_openai.py:34](https://file+.vscode-resource.vscode-cdn.net/Users/kevinblissett/medical-writing/~/medical-writing/.venv/lib/python3.10/site-packages/guidance/llms/_azure_openai.py:34), in AzureOpenAI.__init__(self, model, client_id, authority, caching, max_retries, max_calls_per_min, token, endpoint, scopes, temperature, chat_mode)
     31 if os.path.exists(self._token_cache_path):
     32     self._token_cache.deserialize(open(self._token_cache_path, 'r').read())
---> 34 self._rest_headers["X-ModelType"] = self.model_name

AttributeError: 'AzureOpenAI' object has no attribute '_rest_headers'

Hello,

Like #11 , so I suppose the resolution will be similar. Creating this for future reference for other searches.

Pure intuition, Streamlit spawns worker threads to run the Python Script in, which may be decorrelated from any Asyncio loop you would start on initialization on the main thread through nest_asyncio. Though just creating/setting an asyncio loop in the script also doesn't register correctly :/

Using guidance==0.0.44 and streamlit==1.22.0

"""Run with `streamlit run app.py`"""
import guidance
import streamlit as st

#import asyncio
#loop = asyncio.new_event_loop()
#asyncio.set_event_loop(loop)

st.title("Test Guidance")

guidance.llm = guidance.llms.OpenAI("text-davinci-003")

prompt = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')

st.write(prompt())

  File "guidance\__init__.py", line 15, in <module>
    nest_asyncio.apply()
  File "nest_asyncio.py", line 16, in apply
    loop = loop or asyncio.get_event_loop()
  File "nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "lib\asyncio\events.py", line 656, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'ScriptRunner.scriptThread'.

Have a nice day

I'm getting an error I think on trying to read the response from the llm. I'm using OpenAI via self.guide = guidance.llms.OpenAI(model='gpt-3.5-turbo'), and certain questions trigger the traceback below - not all questions, it seems to be fairly arbitrary. Looking at the code, it possibly needs a guard against empty string responses?

agent      |   File "/usr/local/lib/python3.9/site-packages/guidance/library/_gen.py", line 133, in gen
agent      |     gen_obj = await parser.llm_session(
agent      |   File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 310, in __call__
agent      |     out = self.llm.caller(**call_args)
agent      |   File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 192, in _library_call    
agent      |     out = add_text_to_chat_mode(out)
agent      |   File "/usr/local/lib/python3.9/site-packages/guidance/llms/_openai.py", line 57, in add_text_to_chat_mode
agent      |     c['text'] = c['message']['content']
agent      |   File "/usr/local/lib/python3.9/site-packages/openai/openai_object.py", line 71, in __setitem__        
agent      |     raise ValueError(
agent      | ValueError: You cannot set text to an empty string. We interpret empty strings as None in requests.You may set {
agent      |   "finish_reason": "stop",
agent      |   "index": 0,
agent      |   "message": {
agent      |     "content": "",
agent      |     "role": "assistant"
agent      |   }
agent      | }.text = None to delete the property

The bug
This is based on #19 which has been resolved. When I run the same code as that issue I get the error "No module named 'guidance'". This is despite having done pip install guidance
To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

"""Run with `streamlit run app.py`"""
import guidance
import streamlit as st

#import asyncio
#loop = asyncio.new_event_loop()
#asyncio.set_event_loop(loop)

st.title("Test Guidance")

guidance.llm = guidance.llms.OpenAI("text-davinci-003")

prompt = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''')

st.write(prompt())

System info (please complete the following information):
MacOS

• Guidance Version (guidance.__version__):

Is your feature request related to a problem? Please describe.
JavaScript support is currently not available.

Describe the solution you'd like
I would like to request JavaScript support.

Describe alternatives you've considered
I have considered using alternative libraries or frameworks to achieve similar functionality in JavaScript like langchainjs and my own version. However, the Guidance library stands out due to its specific features and capabilities.

What's the best way to take advantage of the streaming capabilities of hugging face transformers in this library? I see that streaming is all done internally but it's unclear how its exposed to the library user (me)

I figured out a few hacky methods. The only promising one so far is using list_append with a List-like object. Code needs a lot of work. append is called once at the beginning, and then __setitem__ is repeatedly called with -1, so the whole thing works more like a callback itself (output down below):

class dynlist:
    def __init__(self, callback):
        self.data = list()
        self.callback = callback
        
    def append(self, item):
        self.data.append(item)
        self.callback(self.data)
        
    def __setitem__(self, key, val):
        self.data.__setitem__(key, val)
        self.callback(self.data)
        
def update_x(session_id, data: List[str]):
    ....

prompt = guidance("...{{~gen "response" list_append=True temperature=0.4 top_p=0.9}}")
my_session_id = ...
response = dynlist(functools.partial(update_x, my_session_id))
await prompt("...", llm=llm, stream=True, async_mode=True, response=response)

list-append: 
set-item: -1  I
set-item: -1  I like
set-item: -1  I like hanging
set-item: -1  I like hanging out
set-item: -1  I like hanging out with
set-item: -1  I like hanging out with you
set-item: -1  I like hanging out with you.
set-item: -1  I like hanging out with you.
set-item: -1  I like hanging out with you.

The bug
Bump into this error:

File "/home/lucas/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/library/_role.py", line 12, in role
    partial_output(parser.program.llm.role_start(name))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Transformers' object has no attribute 'role_start'

Error in program:  'Transformers' object has no attribute 'role_start'
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 21
      1 program = guidance('''
      2 {{#system}}You are an expert unix systems admin.{{/system}}
      3 
   (...)
     19 {{[~/assistant](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/assistant)}}
     20 ''', llm=chat_llm)
---> 21 out = program(os="Linux", unique=lambda x: list(set(x)), caching=False)

File [~/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/_program.py:225](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/site-packages/guidance/_program.py:225), in Program.__call__(self, **kwargs)
    223     loop = asyncio.new_event_loop()
    224     loop.create_task(new_program.update_display.run()) # start the display updater
--> 225     loop.run_until_complete(new_program.execute())
    227 return new_program

File [~/miniconda3/envs/seed/lib/python3.11/site-packages/nest_asyncio.py:90](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
     87 if not f.done():
     88     raise RuntimeError(
     89         'Event loop stopped before Future completed.')
---> 90 return f.result()

File [~/miniconda3/envs/seed/lib/python3.11/asyncio/futures.py:203](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/lucas/seed/wip/guidance/~/miniconda3/envs/seed/lib/python3.11/asyncio/futures.py:203), in Future.result(self)
    201 self.__log_traceback = False
...
     18     next_next_node=_parser_context["next_next_node"]
     19 )
     21 # send the role-end special tokens

AttributeError: 'Transformers' object has no attribute 'role_start'

Run
To Reproduce(direct copy from use_clear_syntax.ipynb)

chat_llm = guidance.llms.Transformers("stabilityai/stablelm-tuned-alpha-3b", device=1)

program = guidance('''
{{#system}}You are an expert unix systems admin.{{/system}}

{{#user~}}
What are the most common commands used in the {{os}} operating system?
{{~/user}}

{{#assistant~}}
{{#block hidden=True~}}
Here is a common command: "{{gen 'commands' stop='"' n=10 max_tokens=20 temperature=0.7}}"
{{~/block~}}

{{#each (unique commands)}}
{{@index}}. {{this}}
{{~/each}}

Perhaps the most useful command from that list is: "{{gen 'cool_command'}}", because{{gen 'cool_command_desc' max_tokens=100 stop="\\n"}}
On a scale of 1-10, it has a coolness factor of: {{gen 'coolness' pattern="[0-9]+"}}.
{{~/assistant}}
''', llm=chat_llm)
out = program(os="Linux", unique=lambda x: list(set(x)), caching=False)

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Win11 WSL2 ubuntu.
• Guidance Version (guidance.__version__): 0.0.47

Error was raised running the following:

prompt = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "Nein", or "Vielleicht".
Sentence: {{example}}
Answer:{{#select "answer" logprobs='logprobs'}} Ja{{or}} Nein{{or}} Vielleicht{{/select}}''')
prompt = prompt(example='I hate tacos.')
prompt

I tried multiple variations. When the answer format specified at the beginning of the prompt and the options inside the select tag vary then very often this error happens.

When importing guidance from a read-only filesystem, there are a couple of issues:

• Opening the log.txt file (in https://github.com/microsoft/guidance/blob/main/guidance/_utils.py#L10) will always raise an exception as it can't be opened for writing. Can this be switched to use the standard python logging module instead?

• Setting caching=False still tries to create the cache directory. Can be worked around by setting os.environ['XDG_CACHE_HOME'] = '/tmp' (or somewhere that is definitely writeable) before importing, but isn't ideal.

System info:

• OS: AWS Lambda Python 3.9
• Guidance Version: 0.0.47

Hi,

I'm trying to run guidance within a Gradio app. I'm not sure about the async framework behind Gradio but it seems to be AnyIO and is using some worker threads.

The problem I am experiencing is that when I attempt to run a program I get an error that there is no current event loop in the thread. I am not trying to use the program asynchronously (the flag passed in).

  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 194, in __call__
    new_program = Program(
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 110, in __init__
    self._execute_complete = asyncio.Event() # fires when the program is done executing to resolve __await__
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/locks.py", line 177, in __init__
    self._loop = events.get_event_loop()
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.

Any help is welcome.

Message continues: Please post a GitHub issue since this should not happen :)

So here I am.

Raised from:

  File "[..]\guidance\_program_executor.py", line 94, in run
    await self.visit(self.parse_tree)
  File "[..]\guidance\_program_executor.py", line 423, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "[..]\guidance\_program_executor.py", line 423, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "[..]\guidance\_program_executor.py", line 390, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "[..]\guidance\library\_select.py", line 138

Running the following from tutorial.ipynb:

import guidance
guidance.llm = guidance.llms.OpenAI("text-davinci-003")

prompt = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: {{example}}
Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''')
prompt = prompt(example="I hate pineapple on pizza, so that I am ready to kill because of it") //not my actual opinion, was trying to get it to reply "Yes".
print(prompt)

option_logprobs is:

{' Yes': -1000, ' No': -1000, ' Maybe': -1000}

Pattern guides/regex patterns don't seem to have quite the impact one would expect

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Neko-Institute-of-Science/LLaMA-7B-HF")
model = AutoModelForCausalLM.from_pretrained("Neko-Institute-of-Science/LLaMA-7B-HF")
llama = guidance.llms.Transformers(model=model, tokenizer=tokenizer, device=5)
statement_gen = guidance("""
Today we want to say that our new tech company with the name {{gen 'companyname' max_tokens=10 pattern='[a-zA-Z]{8}'}} went public under the ticker {{gen 'ticker' pattern='[A-Z]{4}' temperature=0.3}}
""")
statement_gen(llm=llama)

I would expect this to output

Today we want to say that our new tech company with the name NameOfCompany went public under the ticker TCKR

where NameOfCompany matches [a-zA-Z]{8}, i.e. consisting of letters, no whitespace and exactly eight characters.

Actual output:

Today we want to say that our new tech company with the name ofTechno 2000 is going went public under the ticker TKOO

, i.e. companyname = ofTechno 2000 is going (bad) and ticker = TKOO (good)

Do I misunderstand pattern guides or is this an issue?

Issue: Library fails to import on Windows 10 because termios has no Windows package.
Possible Solutions: Instead of termios, we must import msvcrt. For the pseudoterminals (pty), one could perhaps look at using andfoy/pywinpty.
Stacktrace:

File [{...}\lib\site-packages\guidance\library\__init__.py:1]({...}/lib/site-packages/guidance/library/__init__.py:1)
----> 1 from ._shell import shell
      2 from ._gen import gen
      3 from ._await import await_

File [{...}\lib\site-packages\guidance\library\_shell.py:7]({...}/lib/site-packages/guidance/library/_shell.py:7)
      5 import os
      6 import subprocess
----> 7 import pty
      8 import asyncio
     11 def shell(command, partial_output, safe=True):

File [{...}\lib\pty.py:12]({...}/lib/pty.py:12)
     10 import os
     11 import sys
---> 12 import tty
     14 __all__ = ["openpty","fork","spawn"]
     16 STDIN_FILENO = 0

File [{...}\lib\tty.py:5]({...}/lib/tty.py:5)
      1 """Terminal utilities."""
      3 # Author: Steen Lumholt.
----> 5 from termios import *
      7 __all__ = ["setraw", "setcbreak"]
      9 # Indexes for termios list.

ModuleNotFoundError: No module named 'termios'

Code reference: https://github.com/microsoft/guidance/blob/ff31b31177ee9ad036ef5130daada5008444c734/guidance/library/_shell.py#LL7C11-L7C11

The bug
ValueError: No valid option generated in #select

To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

import os
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

import guidance

d = '''
    User: Question: Which is the best MCU film?

    Assistant: Yes. I need to search the web for the best MCU film.

    User: Question: {{question}}

    Assistant {{#select "answer"}} Yes{{or}} No{{/select}}
'''

chatgpt = guidance.llms.OpenAI("text-davinci-003")

prompt = guidance(d)

prompt = prompt(llm = chatgpt, question = "What is Google's Headquarter address?")

Traceback (most recent call last):
  File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 94, in run
    await self.visit(self.parse_tree)
  File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 423, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 423, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
  File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program_executor.py)", line 390, in visit
    command_output = await command_function(*positional_args, **named_args)
  File "[/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/library/_select.py](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/library/_select.py)", line 138, in select
    raise ValueError("No valid option generated in #select! Please post a GitHub issue since this should not happen :)")
ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)

Error in program:  No valid option generated in #select! Please post a GitHub issue since this should not happen :)
Output exceeds the [size limit](command:workbench.action.openSettings?%5B%22notebook.output.textLineLimit%22%5D). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?3f937783-9df6-4db8-8cc8-185d951c685d)---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 20
     16 chatgpt = guidance.llms.OpenAI("text-davinci-003")
     18 prompt = guidance(d)
---> 20 prompt = prompt(llm = chatgpt, question = "What is Google's Headquarter address?")

File [~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program.py:225](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/guidance/_program.py:225), in Program.__call__(self, **kwargs)
    223     loop = asyncio.new_event_loop()
    224     loop.create_task(new_program.update_display.run()) # start the display updater
--> 225     loop.run_until_complete(new_program.execute())
    227 return new_program

File [~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/nest_asyncio.py:90](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
     87 if not f.done():
     88     raise RuntimeError(
     89         'Event loop stopped before Future completed.')
---> 90 return f.result()

File [~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/futures.py:201](https://file+.vscode-resource.vscode-cdn.net/Users/jackzhou/test/jack4/agents/~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/futures.py:201), in Future.result(self)
    199 self.__log_traceback = False
    200 if self._exception is not None:
--> 201     raise self._exception
    202 return self._result
...
    137 if max(option_logprobs.values()) <= -1000:
--> 138     raise ValueError("No valid option generated in #select! Please post a GitHub issue since this should not happen :)")
    140 partial_output(selected_option)

ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Mac OS
• Guidance Version (guidance.__version__): guidance-0.0.47

I'm on Python 3.11

pip3 install guidance==0.0.42

Then a simple script with

import guidance

Gets error

Traceback (most recent call last):
  File "scrubbed...", line 1, in <module>
    import guidance
  File "/opt/homebrew/lib/python3.11/site-packages/guidance/__init__.py", line 7, in <module>
    from ._program import Program
  File "/opt/homebrew/lib/python3.11/site-packages/guidance/_program.py", line 19, in <module>
    from . import library
  File "/opt/homebrew/lib/python3.11/site-packages/guidance/library/__init__.py", line 11, in <module>
    from ._geneach import geneach
  File "/opt/homebrew/lib/python3.11/site-packages/guidance/library/_geneach.py", line 112
    partial_output( block_content = parser_context
                                    ^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

I run pip3 install guidance==0.0.41 and import works.

Hi,

Firstly, thanks for sharing the idea of prompt templating.

I'm playing with use_clear_syntax.ipynb in notebooks/art_of_prompt_design/use_clear_syntax.ipynb,
and trying to understand how it works with openai API using the following code:

https://gist.github.com/wooparadog/5765952394c33c5c9091f2835467eeec

But I got an exception:

Traceback (most recent call last):
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 94, in run
    await self.visit(self.parse_tree)
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 434, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 434, in visit
    visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/_program_executor.py", line 394, in visit
    command_output = await command_function(*positional_args, **named_args)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/library/_system.py", line 13, in system
    return await role(name="system", hidden=hidden, _parser_context=_parser_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wooparadog/Codes/github.com/microsoft/guidance/guidance/library/_role.py", line 4, in role
    block_content = _parser_context['block_content']
                    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

Is there anything wrong with my test case? Thanks again

Thanks ! Love the token control.

Just wish your example was Keynes v Hayek.

I would like to understand how Guidance compares to LMQL when reading the docs. Can we gather a list of additional features?

This project is seriously embarrassing. The sooner someone in management comes to their senses and gets this deleted the better off you will all be.

The bug
Although producing individual fields as numbers seems to work fine (although invariably the actual parsed variable is still a string as seen below), and arrays of strings also seem to work fine, arrays of numbers do not seem work properly.

I'd love to be able to fully understand why this is happening and am also very happy to attempt to fix this if someone is able to point me in the right direction.

To Reproduce

import guidance
import os

os.environ["OPENAI_API_KEY"] = "sk-..."

llm = guidance.llms.OpenAI("text-davinci-003")


def example():
    # we can pre-define valid option sets
    valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]

    # define the prompt
    program = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' stop=','}},
    "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra'}}",
    "strength": {{gen 'strength' stop=','}},
    "item_ids": [{{#geneach 'numbers' num_iterations=3}}
        {{gen 'this'}},{{/geneach}}
    ]
}```""")

    # execute the prompt
    result = program(
        description="A quick and nimble fighter.", valid_weapons=valid_weapons, llm=llm
    )

    print(result)
    print(result.variables())

    return result


if __name__ == "__main__":
    example()

The outputs of the print statements are as follows:

print(result)

The following is a character profile for an RPG game in JSON format.
```json
{
    "description": "A quick and nimble fighter.",
    "name": "Fighter",
    "age":  25,
    "armor": "leather",
    "weapon": "sword",
    "class": "warrior",
    "mantra": "Strength in numbers.",
    "strength":  8,
    "item_ids": [
         1,
         2,
         3
    ]
},
    ]
}```

print(result.variables())

{
'llm': <guidance.llms._openai.OpenAI object at 0x101205df0>,
'description': 'A quick and nimble fighter.', 
'valid_weapons': ['sword', 'axe', 'mace', 'spear', 'bow', 'crossbow'],
'name': 'Fighter',
'age': ' 25',
'armor': 'leather', 
'weapon': 'sword',
'class': 'warrior',
'mantra': 'Strength in numbers.', 
'strength': ' 8', 
'numbers': [' 1', ' 2', ' 3\n    ]\n}']
}

Most of the problem here seems to be that the generation of the final number in the array goes a bit wonky - not sure if I could tweak my prompt, or if it's a deeper issue.

System info

• Mac OS:
• Guidance Version 0.0.48

My results with LLaMA-7B don't match the tutorial, please see some examples below.

import guidance
guidance.llm = guidance.llms.Transformers('Neko-Institute-of-Science/LLaMA-7B-HF', device=0)

1. Character Creator

# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]

# define the prompt
character_maker = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
  "id": "{{id}}",
  "description": "{{description}}",
  "name": "{{gen 'name'}}",
  "age": {{gen 'age' pattern='[0-9]+' stop=','}},
  "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
  "weapon": "{{select 'weapon' options=valid_weapons}}",
  "class": "{{gen 'class'}}",
  "mantra": "{{gen 'mantra' temperature=0.7}}",
  "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
  "items": [{{#geneach 'items' num_iterations=5 join=', '}}"{{gen 'this' temperature=0.7}}"{{/geneach}}]
}```""")

# generate a character
character_maker(
  id="e1f491f7-7ab8-4dac-8c20-c92b5e7d883d",
  description="A quick and nimble fighter.",
  valid_weapons=valid_weapons
)

Result:

The following is a character profile for an RPG game in JSON format.

{
    "id": "e1f491f7-7ab8-4dac-8c20-c92b5e7d883d",
    "description": "A quick and nimble fighter.",
    "name": "Fighter",
    "age": 18,
    "armor": "plateplate",
    "weapon": "swordsword",
    "class": "fighter",
    "mantra": "Weaponry is my strength.",
    "strength": 10,
    "items": ["sword", "armor", "helmet", "sword", "fishing pole"]
}

select for the armor and weapon duplicates the output.

2. Basic Templating

Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: I hate tacos
Answer: Yes Yes

prompt = guidance('''Generate a response to the following email:
{{email}}.
Response:{{gen "response"}}

Is the response above offensive in any way? Please answer with a single word, either "Yes" or "No".
Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{/select}}''')
prompt = prompt(email='I hate tacos')
prompt

Generate a response to the following email:
I hate tacos.
Response: I hate tacos too.
I hate tacos.
Response: I hate tacos too. I hate tacos too.
I hate tacos too.
Response: I hate tacos too. I hate tacos too. I hate tacos too.
I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate tacos too. I hate

Is the response above offensive in any way? Please answer with a single word, either "Yes" or "No".
Answer: Yes Yes

select duplicates the Yes

Just walking through the tutorial.

guidance.llm = guidance.llms.OpenAI("text-curie-001")

p = guidance('''Is the following sentence offensive? Yes, no, or maybe.
Sentence: {{example}}
Answer: {{#select 'answer' logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''')

p = p(example='You are such an asshole!')

What did I miss?

ps: text-davinci-003 failed as well.

This library from HF is pretty great and I get use out of it in production settings for LLMs. Would love to figure out how to integrate a system like this for LLM safety with it so I can use HF models, get dynamic batching, and be able to stream tokens with the guidance library!

I think that the output for hidden generation on README.me is wrong. Can someone double check it?

https://github.com/microsoft/guidance#hidden-generation

Is your feature request related to a problem? Please describe.
Seeing that guidance has some ability to generate syntax specific outputs, I'm curious to know if the team has thoughts on generating more complex programming language grammars.

Describe the solution you'd like

When the points to a reference syntax, the llm output would be syntactically correct specific to the given language grammar.

Describe alternatives you've considered
None

Additional context
I have a bunch of domain specific languages I've built and I've been thinking that if I can basically validate each token that comes out of the stream against a language grammar, it should basically ensure that it works correctly all the time. It's kind of an extension of the valid options you have in the examples.

I'm happy to help build this but I'd need some help is outlining this.

Can there be a way to load a transformer from huggingface in bits and bytes. That could make the model loading easier. I might add this after work but it would be nice to have.

Where?
https://github.com/microsoft/guidance/blob/b1e656f87abc543c68f3d07057a23f2f5922a878/notebooks/chat.ipynb

Steps to reproduce:

• Set openAI api key
• Open the chat notebook
• change the openAI model from gpt-4 to gpt-3.5-turbo
• Run the first 2 blocks in the notebook

Error

IndexError                                Traceback (most recent call last)
Cell In[14], line 45
      3     return options[best]
      5 create_plan = guidance('''{{#system~}}
      6 You are a helpful assistant.
      7 {{[~/system](https://file+.vscode-resource.vscode-cdn.net/Users/user/guidance/notebooks/~/system)}}
   (...)
     42 {{gen 'plan' max_tokens=500}}
     43 {{[~/assistant](https://file+.vscode-resource.vscode-cdn.net/Users/user/guidance/notebooks/~/assistant)}}''')
---> 45 out = create_plan(goal='read more books', parse_best=parse_best)
     46 out

File [/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/guidance/_program.py:216](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/guidance/_program.py:216), in Program.__call__(self, **kwargs)
    214     loop = asyncio.new_event_loop()
    215     loop.create_task(new_program.update_display.run()) # start the display updater
--> 216     loop.run_until_complete(new_program.execute())
    218 return new_program

File [/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nest_asyncio.py:90](https://file+.vscode-resource.vscode-cdn.net/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nest_asyncio.py:90), in _patch_loop..run_until_complete(self, future)
     87 if not f.done():
     88     raise RuntimeError(
     89         'Event loop stopped before Future completed.')
---> 90 return f.result()
...
      1 def parse_best(prosandcons, options):
----> 2     best = int(re.findall(r'Best=(\d+)', prosandcons)[0])
      3     return options[best]

IndexError: list index out of range

Hello,

I've been using your role_simulator function and encountered a KeyError for the 'first_question' variable. The error message suggests that 'first_question' should have been passed as an argument when calling the program, or it should have been set as a default value when creating the program.

Here's the code that's causing the error:

role_simulator = guidance('''
...
{{#if first_question}}You can also start the conversation.{{/if}}
...
{{#if first_question}}Let me start the conversation now:
{{role}}: {{first_question}}{{/if}}
...
''')

republican = role_simulator(role='Republican')
democrat = role_simulator(role='Democrat')

first_question = '''What do you think is the best way to stop inflation?'''
...

The error occurs because 'first_question' is not defined at the time when the role_simulator function is called.

Is your feature request related to a problem? Please describe.

For non-executed programs storing and loading from text files works fine. But for partially executed programs with custom variables an ugly workaround was necessary. Otherwise I would get the error

Error in program:  Can't set a property of a non-existing variable: conversation[-1].user_text

when trying to continue the chat after loading the text file and converting it to a program.

Describe the solution you'd like

Ideally we can just program.save(filename) and program.load(filename) and it just works without worrying about storing and loading the program.variables() separately.

Describe alternatives you've considered

I was trying to do this:

import guidance
guidance.llm = guidance.llms.OpenAI('gpt-3.5-turbo')
guidance.llm.cache.clear()

prompt = guidance(
'''{{#system~}}
You are a helpful assistant
{{~/system}}
{{~#geneach 'conversation' stop=False~}}
{{#user~}}
{{set 'this.user_text' (await 'user_text')}}
{{~/user}}
{{#assistant~}}
{{gen 'this.ai_text' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''', stream=False, silent=True)

prompt = prompt(user_text ='Hello there :)')

with open('prompt.txt', 'w') as f:
    f.write(str(prompt))

with open('prompt.txt', 'r') as f:
    prompt = f.read()

prompt = guidance(prompt)

prompt = prompt(user_text ='I want to travel to the moon')

But got the error mentioned above.

My solution was:

def save_prompt(prompt, filename):
    variables = prompt.variables()
    del variables['llm']
    to_store = {'text': str(prompt), 'variables': variables}
    with open(filename, 'w') as f:
        json.dump(to_store, f)
    
def load_prompt(filename):
    with open(filename, 'r') as f:
        loaded = json.load(f)
    prompt = guidance(loaded['text'], **loaded['variables'])
    return prompt

Note that the del variables['llm'] is necessary because the llm object is not serializable.

I'm seeing this popup around twitter. I've been thinking of workarounds, but perhaps you could highlight the issue as it seems to be pervasive. I'm also curious what you folks think on how to best address

installed on windows 10 with Anaconda Python 3.10.9 openai-0.27.6
ran pip install guidance
notebook tutorial.ipynb worked until "Using Tools" section

IncompleteParseError Traceback (most recent call last)
Cell In[8], line 87
35 program = guidance('''
36 {{#system~}}
37 You are a helpful assistant.
(...)
82 {{~/assistant}}
83 ''')
85 query = "What is Facebook's stock price right now?"
---> 87 program = program(
88 user_query=query,
89 search=search,
90 is_search=is_search,
91 practice=practice_round
92 )

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:204, in Program.call(self, **kwargs)
195 new_program = Program(
196 text=self.marked_text,
197
(...)
200 {{k: v if callable(v) else copy.deepcopy(v) for k,v in self._variables.items()}, **kwargs}
201 )
203 # create an executor for the new program (this also marks the program as executing)
--> 204 new_program._executor = ProgramExecutor(new_program)
206 # if we are in async mode schedule the program in the current event loop
207 if new_program.async_mode:

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:50, in ProgramExecutor.init(self, program)
48 except parsimonious.exceptions.ParseError as e:
49 self._check_for_simple_error(text)
---> 50 raise e

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:47, in ProgramExecutor.init(self, program)
45 # parse the program text
46 try:
---> 47 self.parse_tree = grammar.parse(text)
48 except parsimonious.exceptions.ParseError as e:
49 self._check_for_simple_error(text)

File ~\AppData\Roaming\Python\Python310\site-packages\parsimonious\grammar.py:112, in Grammar.parse(self, text, pos)
106 """Parse some text with the :term:default rule.
107
108 :arg pos: The index at which to start parsing
109
110 """
111 self._check_default_rule()
--> 112 return self.default_rule.parse(text, pos=pos)

File ~\AppData\Roaming\Python\Python310\site-packages\parsimonious\expressions.py:143, in Expression.parse(self, text, pos)
141 node = self.match(text, pos=pos)
142 if node.end < len(text):
--> 143 raise IncompleteParseError(text, node.end, self)
144 return node

IncompleteParseError: Rule 'template' matched in its entirety, but it didn't consume all the text. The non-matching portion of the text begins with '{{#each practice}}
{' (line 14, column 1).

how do i call guidance but laod the models as int8 so i can fit them on even an 80Gb GPU?

Issue: we can't import ggml (ex: llama-cpp) models.

I'm trying to generate some text using a modified version of the main example and am running into an issue where the model is picking a token containing a non-numerical character for the 'chapter' variable despite that not matching the pattern. Here's my code:

prompt = '''
Tweak this proverb to apply to machine learning model instructions instead.

{{proverb}}
- {{book}} {{chapter}}:{{verse}}

UPDATED
Where there is no guidance{{gen 'rewrite' stop='- '}}
- GPT {{gen 'chapter' pattern='[0-9]' max_tokens=1}}:{{gen 'verse' pattern='[0-9]+' stop='\\n'}}
'''[1:-1]

model = guidance.llms.Transformers('stabilityai/stablelm-base-alpha-3b', device=0)
program = guidance(prompt, llm = model)
executed_program = program(
    proverb="Where there is no guidance, a people falls,\nbut in an abundance of counselors there is safety.",
    book="Proverbs",
    chapter=11,
    verse=14,
)

print(executed_program)
print(executed_program["chapter"])

and the generated output:

Tweak this proverb to apply to machine learning model instructions instead.

Where there is no guidance, a people falls,
but in an abundance of counselors there is safety.
- Proverbs 11:14

UPDATED
Where there is no guidance, a people falls, but in an abundance of counselors there is safety.

- GPT 2::2
2:

I should also add that I saw a similar issue earlier today with pattern not working as intended so I made sure I was on the latest version just before running this just now.

I skimmed the documentation and notebooks, and I found some similarities between guidance and langchain. Can you please explain the reasons one should use guidance over langchain? More importantly, some of the features (e.g., token healing) seem not to be available for GPT-4 (or closed-source models in general). Is that true?

installed on windows 10 with Anaconda Python 3.10.9 openai-0.27.6
ran pip install guidance
notebook tutorial.ipynb worked until section 21

Traceback

Traceback (most recent call last):
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py", line 143, in geneach
gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 310, in call
out = self.llm.caller(**call_args)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 186, in _library_call
kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 24, in prompt_to_messages
assert prompt.endswith("<|im_start|>assistant\n"), "When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting."
AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 94, in run
await self.visit(self.parse_tree)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 423, in visit
visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py", line 390, in visit
command_output = await command_function(*positional_args, **named_args)
File "C:\Users\David\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py", line 145, in geneach
raise Exception(f"Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.")
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.

Error in program: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.

AssertionError Traceback (most recent call last)
File ~\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py:143, in geneach(list_name, stop, max_iterations, min_iterations, num_iterations, hidden, join, single_call, single_call_temperature, single_call_max_tokens, single_call_top_p, _parser_context)
142 try:
--> 143 gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
144 except Exception:

File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:310, in OpenAISession.call(self, prompt, stop, stop_regex, temperature, n, max_tokens, logprobs, top_p, echo, logit_bias, token_healing, pattern, stream, cache_seed, caching)
309 call_args["logit_bias"] = {str(k): v for k,v in logit_bias.items()} # convert keys to strings since that's the open ai api's format
--> 310 out = self.llm.caller(**call_args)
312 except openai.error.RateLimitError:

File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:186, in OpenAI._library_call(self, **kwargs)
185 if self.chat_mode:
--> 186 kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
187 del kwargs['prompt']

File ~\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py:24, in prompt_to_messages(prompt)
21 # if len(start_tags) != len(end_tags):
22 # raise MalformedPromptException("Malformed prompt: start and end tags are not properly paired")
---> 24 assert prompt.endswith("<|im_start|>assistant\n"), "When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting."
26 pattern = r'<|im_start|>(\w+)(.*?)(?=<|im_end|>|$)'

AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
Cell In[21], line 13
1 prompt = guidance(
2 '''{{#system~}}
3 You are a helpful assistant
(...)
11 {{/assistant}}
12 {{/geneach}}''')
---> 13 prompt= prompt(user_text ='hi there')
14 prompt

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:225, in Program.call(self, **kwargs)
223 loop = asyncio.new_event_loop()
224 loop.create_task(new_program.update_display.run()) # start the display updater
--> 225 loop.run_until_complete(new_program.execute())
227 return new_program

File C:\anaconda3\lib\site-packages\nest_asyncio.py:90, in _patch_loop..run_until_complete(self, future)
87 if not f.done():
88 raise RuntimeError(
89 'Event loop stopped before Future completed.')
---> 90 return f.result()

File C:\anaconda3\lib\asyncio\futures.py:201, in Future.result(self)
199 self.__log_traceback = False
200 if self._exception is not None:
--> 201 raise self._exception.with_traceback(self._exception_tb)
202 return self._result

File C:\anaconda3\lib\asyncio\tasks.py:232, in Task.__step(failed resolving arguments)
228 try:
229 if exc is None:
230 # We use the send method directly, because coroutines
231 # don't have __iter__ and __next__ methods.
--> 232 result = coro.send(None)
233 else:
234 result = coro.throw(exc)

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program.py:311, in Program.execute(self)
309 else:
310 with self.llm.session(asynchronous=True) as llm_session:
--> 311 await self._executor.run(llm_session)
312 self._text = self._executor.prefix
314 # delete the executor and so mark the program as not executing

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:98, in ProgramExecutor.run(self, llm_session)
96 print(traceback.format_exc())
97 print("Error in program: ", e)
---> 98 raise e

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:94, in ProgramExecutor.run(self, llm_session)
88 self.llm_session = llm_session
89 try:
90 # first parse all the whitespace control
91 # self.whitespace_control_visit(self.parse_tree)
92
93 # now execute the program
---> 94 await self.visit(self.parse_tree)
95 except Exception as e:
96 print(traceback.format_exc())

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:423, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
421 else:
422 inner_prev_node = prev_node
--> 423 visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
424 # visited_children = [self.visit(child) for child in node.children]
426 if len(visited_children) == 1:

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:423, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
421 else:
422 inner_prev_node = prev_node
--> 423 visited_children.append(await self.visit(child, inner_next_node, inner_next_next_node, inner_prev_node, node, parent_node))
424 # visited_children = [self.visit(child) for child in node.children]
426 if len(visited_children) == 1:

File ~\AppData\Roaming\Python\Python310\site-packages\guidance_program_executor.py:390, in ProgramExecutor.visit(self, node, next_node, next_next_node, prev_node, parent_node, grandparent_node)
388 # call the optionally asyncronous command
389 if inspect.iscoroutinefunction(command_function):
--> 390 command_output = await command_function(*positional_args, **named_args)
391 else:
392 command_output = command_function(*positional_args, **named_args)

File ~\AppData\Roaming\Python\Python310\site-packages\guidance\library_geneach.py:145, in geneach(list_name, stop, max_iterations, min_iterations, num_iterations, hidden, join, single_call, single_call_temperature, single_call_max_tokens, single_call_top_p, _parser_context)
143 gen_obj = await parser.llm_session(strip_markers(parser.prefix), stop=stop, max_tokens=len(stop_tokens)+2, temperature=0, cache_seed=0)
144 except Exception:
--> 145 raise Exception(f"Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.")
146 if gen_obj["choices"][0]["finish_reason"] == "stop":
147 break

Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.

Does not work

Assistant: {{#select "answer"}} Yes{{or}} No{{/select}}

Error: ValueError: No valid option generated in #select! Please post a GitHub issue since this should not happen :)

Works

Assistant: {{#select "answer"}}Yes{{or}}No{{/select}}

Originally posted by @PenutChen in #40 (comment)

Is your feature request related to a problem? Please describe.
I know this lib is new. It would be nice to see more LLMs to be added, especially Anthropic Claudes

Describe the solution you'd like
More LLMs support

Describe alternatives you've considered
N/A

Additional context
Claudes-100k is super useful.

The bug

AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.
...
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.

The error goes away after seting stop=False as suggested in the error message but I think this should be added to the readme. I can do the pull request if you agree.

To Reproduce
I copy pasted the example Agents with geneach given in the readme.

import guidance
guidance.llm = guidance.llms.OpenAI('gpt-3.5-turbo')
guidance.llm.cache.clear()

prompt = guidance(
'''{{#system~}}
You are a helpful assistant
{{~/system}}
{{~#geneach 'conversation'}}
{{#user~}}
{{set 'this.user_text' (await 'user_text')}}
{{~/user}}
{{#assistant~}}
{{gen 'this.ai_text' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''')
prompt= prompt(user_text ='hi there')
prompt

System info (please complete the following information):

• OS: Ubuntu
• Guidance Version: 0.0.47 and 0.0.48