clarkkev / attention-analysis Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Hi,
Thanks for sharing these excellent resources.
I wanted to ask if there is an easy fix in the model loading part and if we can extend the same attention analysis on pytorch pertained models also ?
Looking forward for your response.
Is it possible to share the text file (not-tokenized, and in the correct format) which was used to obtain the attention values in unlabeled_attn.pkl
?
Will be a big help to compare different model on the exact same corpus. (since different models use different tokenizers)
I have used preprocess_unlabelled to create my json file. Can you provide a preprocess file which would help word level attention.
When I use this file and when I pass the argument --word_level to extract_attention I get the following error:
Converting to word-level attention...
Traceback (most recent call last):
File "extract_attention.py", line 144, in
main()
File "extract_attention.py", line 134, in main
feature_dicts_with_attn, tokenizer, args.cased)
File "/home/manasi/Documents/BERT/attention-analysis-master/bpe_utils.py", line 74, in make_attn_word_level
words_to_tokens = tokenize_and_align(tokenizer, features["words"], cased)
KeyError: 'words'
Please help me fix it.
Can you please clarify what does each number (8,10) stand for in "Head 8-10" in Figure 5 of your article, i.e. layer or head number?
Thank you very much for your clarification.
The instructions say:
We include two pre-processing scripts for going from a raw data file to JSON that can be supplied to attention_extractor.py
I think what is meant is the extract_attention.py script in the repo?
I am trying this with tf 2.0 and am having a lot of issues. can you tell us what version you were using or maybe add a requirements.txt? .. thanks!
Dear Kevin, hello! I'm trying to reproduce your work described at https://github.com/clarkkev/attention-analysis but it's not clear how to get the train.txt and dev.txt files used in preprocess_depparse.py. Can you clarify how I can have such files? I would like to take this opportunity to congratulate you on the experiment. Thank you very much!
My sample contains the word 'Silsby', but it does not exist in the vocab.
How to deal with the OOV situation?
python extract_attention.py --preprocessed-data-file samples_10.json --bert-dir data/cased_L-12_H-768_A-12 --max_sequence_length 256 --word_level --cased
Creating examples...
Traceback (most recent call last):
File "extract_attention.py", line 144, in <module>
main()
File "extract_attention.py", line 108, in main
example = Example(features, tokenizer, args.max_sequence_length)
File "extract_attention.py", line 29, in __init__
self.input_ids = tokenizer.convert_tokens_to_ids(self.tokens)
File "/Users/smap10/Project/attention-analysis-master/bert/tokenization.py", line 182, in convert_tokens_to_ids
return convert_by_vocab(self.vocab, tokens)
File "/Users/smap10/Project/attention-analysis-master/bert/tokenization.py", line 143, in convert_by_vocab
output.append(vocab[item])
KeyError: 'Silsby'
In readme ,I can see ''Each line in the files should contain a word followed by a space followed by <index_of_head>-<dependency_label> (e.g., 0-root).''
So how can I get the index_of_head, and what's the meaning of it?
Should I know which heads(12*12) in bert is for which syntactic function firstly ,and then judge it?
Sorry for my pool English.
It should have been inserted at 0th index in -1 dimension, currently its added at last index. Since attention approximated for ROOT by adding start/end tokens would be at 0th index, it would expect word rep for root should also be at 0.
By fixing this bug I got around 3% higher UAS.
word_reprs = tf.concat([word_reprs, tf.zeros((n_words, 1, 200))], 1) # dummy for ROOT
Should be replaced with,
word_reprs = tf.concat([tf.zeros((n_words, 1, 200)), word_reprs], 1) # dummy for ROOT
Any plans of releasing the code for coreference analysis in the paper?
Alternatively, it is possible to explain the methodology? Mainly around how the "head" word is chosen, and what is the exact set of antecedents used.
First of I want to say this is a great insight into the BERT model. But there seems to be a file missing from General Analysis. The file as said can be used to generate one's own wiki data.
(create_wiki_data.py). It would be quite helpful if you provide it .
Thank you for releasing the codes.
An explanation for head_distances.py
is missing in README.
Could you add the explanation?
A small question: in head_distances.py
, the line utils.write_pickle(js_distances, args.outfile)
should be outside the "for loop" for i, doc in enumerate(data):
?
The code in this line means set segment ids to zeros for both segments, I don't know whether this is a bug?
attention-analysis/extract_attention.py
Line 30 in 7b4ed20
Hi, I found that dev_attn.pkl and train_attn.pkl for syntax analysis have all heads of 0. And the relns are all 'root'.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.