Comments (5)
After poking around some more with the chat models, I don't think we can get surprisals after all :(
The logprobs
parameter lets us get the logprobs corresponding to the model's generated output, but without the echo
parameter (as was previously supported by the legacy Completion models), we can't get the logprobs assigned to an arbitrary input string. It's possible I might be missing something, though!
from surprisal.
Reopening, as reported by OpenAI on X, log probs seem to be available again, so it would be nice to re-build and test the (existing) support for this in our package. Thanks @jennhu!
from surprisal.
@jennhu do you have a MWE for this with the new API? I just haven't used openai models in a while so if you have something that already works I wouldn't have to start over:)
from surprisal.
if you have the bandwidth to PR even better but if not I'll take anything to start off of!
from surprisal.
I've already added a deprecation warning in README as of a926f9b; the API should continue working for older models supporting this endpoint (though I haven't tested it recently), but otherwise the broader issue about adding support is going to be a wontfix
. Alternatives to OpenAI models include: MPT, Mistral, Llama, Falcon (not an exhaustive list or endorsement).
from surprisal.
Related Issues (13)
- Error when using Python-based tokenizers HOT 1
- Support GPU HOT 2
- Support BERT-like bidirectional encoders for surprisal computation
- Slicing in SurprisalArray is not fully Pythonic
- compute surprisal for Chinese characters HOT 6
- cyclical python version reqs causing `pip install` to fail
- Are surprisal values across different batch sizes slightly different?
- Make the CI stuff work
- Indexing into SurprisalArray using singletons fails in NGramSurprisal.
- Conflating causal LM and "gpt" model class
- Adding support for OpenAI chat models HOT 1
- Dependency issues in Python 3.12 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from surprisal.