Comments (11)
Ah, very good! Have you tried anything like few shot inference here? People would probably like to see that too.
from k2.
Hi, Richard, againπ, thanks for your advice. We uploaded some sample geoscience questions to the generation folder path.
from k2.
Good issue, and we are not showing this kind of cases. From my perspective, we use few shot to inference mostly on specific tasks like multiple-choice, information extraction or stimulate the ICL for specific format generation (which 7B model may not good at it).
We are undergoing training and evaluating a larger language model for geo, and we indeed design some cases to make our model be equipped with more abilities.
from k2.
Yeah, have the 'here are a few examples of texts' - is this text one of those type sort of check.
from k2.
When you are using your benchmark, maybe example :- paper suggests something like this?
{'id': 'apstudy_question_hg', 'question': {'stem': "The umbrella theory explaining the Earth's movement, contact, and flattening of large land plates is known as", 'choices': [{'text': 'the Coriolis effect', 'label': 'A'}, {'text': 'plate tectonics', 'label': 'B'}, {'text': 'hotspots', 'label': 'C'}, {'text': 'the Richter Magnitude Scale', 'label': 'D'}, {'text': 'the subduction zone', 'label': 'E'}]}, 'The answer is?': ''}
from k2.
Formatting this wrong will probably mean results will be somewhat different.?
from k2.
As we mentioned in the paper (Sec. 5.1 Objective tasks in GeoBenchmark), we prompt ending with the phrase "The answer is" and calculate the ππππ‘πππ₯ of the probability of next token among the alphabet "A,B,C,D,E".
from k2.
I have run a test, with the generic generate function
[
{
"id": "apstudy_question_hg",
"question": {
"stem": "Because he figured out that sedimentary rock must have been compacted and compressed, over many ages, _______ is known as the father of modern geology.",
"choices": [
{
"text": "Richard Palmer",
"label": "A"
},
{
"text": "James Hutton",
"label": "B"
},
{
"text": "W",
"label": "C"
},
{
"text": "Nicholas Steno",
"label": "D"
},
{
"text": "Aubrey Hough",
"label": "E"
}
]
},
"The Answer is?": ""
}
]
[{"question": {"id": "apstudy_question_hg", "question": {"stem": "Because he figured out that sedimentary rock must have been compacted and compressed, over many ages, _______ is known as the father of modern geology.", "choices": [{"text": "Richard Palmer", "label": "A"}, {"text": "James Hutton", "label": "B"}, {"text": "W", "label": "C"}, {"text": "Nicholas Steno", "label": "D"}, {"text": "Aubrey Hough", "label": "E"}]}, "The Answer is?": ""}, "answer": "The correct answer is: B"}]
A different test?
from k2.
e.g. if want to adapt it to something not-Llama like Falcon or something else to try there
from k2.
As mentioned above, here I give you a toy example:
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("/path/to/your/model", use_fast=True)
model = AutoModelForCausalLM.from_pretrained("/path/to/your/model", device_map="auto")
input = "Please select the correct option: The following substances are mainly formed by groundwater metasomatism: () \n A. nodule\n B. Quanhua\n C. silicified wood\n D. halite pseudocrystal\n\nThe answer is: ### Output\n"
input_ids = tokenizer(input, return_tensors='pt')
outputs = model(input_ids["input_ids"])
the outputs["logits"]
is the probs
of each token in vocab, and we can use tokenizer("A")['input_ids'][-1]
to get the ids
of the candidates alphabets and further do softmax
among the probs
of the candidates alphabets.
p.s. Since these questions are no longer related to the input_ls itself, I suggest to close this issue.
from k2.
Right, can add that to another one.
from k2.
Related Issues (13)
- Tokenizer not available error HOT 3
- Example website generation parameters? HOT 7
- Data and Scripts for Further Pretraining HOT 1
- How to evaluate geobench HOT 3
- Different answer with the same question
- Full version of instruction-tuning geo-signal data HOT 1
- Issues in evaluation code.
- π I need the geosignal.json dataset for finetunning. HOT 2
- OSError: Can't load tokenizer for 'daven3/k2-v1' HOT 1
- Due to the dependencies conflict in the k2.yaml, running the k2 model is impossible. Please update the right file.
- how to deal with the formula in paper?
- Request for GeoTools Dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k2.