clarkkev / attention-analysis Goto Github PK

Hi,

Thanks for sharing these excellent resources.

I wanted to ask if there is an easy fix in the model loading part and if we can extend the same attention analysis on pytorch pertained models also ?

Looking forward for your response.

Is it possible to share the text file (not-tokenized, and in the correct format) which was used to obtain the attention values in unlabeled_attn.pkl?

Will be a big help to compare different model on the exact same corpus. (since different models use different tokenizers)

I have used preprocess_unlabelled to create my json file. Can you provide a preprocess file which would help word level attention.

When I use this file and when I pass the argument --word_level to extract_attention I get the following error:
Converting to word-level attention...
Traceback (most recent call last):
File "extract_attention.py", line 144, in
main()
File "extract_attention.py", line 134, in main
feature_dicts_with_attn, tokenizer, args.cased)
File "/home/manasi/Documents/BERT/attention-analysis-master/bpe_utils.py", line 74, in make_attn_word_level
words_to_tokens = tokenize_and_align(tokenizer, features["words"], cased)
KeyError: 'words'

Please help me fix it.

Can you please clarify what does each number (8,10) stand for in "Head 8-10" in Figure 5 of your article, i.e. layer or head number?
Thank you very much for your clarification.

The instructions say:

We include two pre-processing scripts for going from a raw data file to JSON that can be supplied to attention_extractor.py

I think what is meant is the extract_attention.py script in the repo?

I am trying this with tf 2.0 and am having a lot of issues. can you tell us what version you were using or maybe add a requirements.txt? .. thanks!

Dear Kevin, hello! I'm trying to reproduce your work described at https://github.com/clarkkev/attention-analysis but it's not clear how to get the train.txt and dev.txt files used in preprocess_depparse.py. Can you clarify how I can have such files? I would like to take this opportunity to congratulate you on the experiment. Thank you very much!

My sample contains the word 'Silsby', but it does not exist in the vocab.
How to deal with the OOV situation?

python extract_attention.py --preprocessed-data-file samples_10.json --bert-dir data/cased_L-12_H-768_A-12  --max_sequence_length 256 --word_level --cased

Creating examples...
Traceback (most recent call last):
  File "extract_attention.py", line 144, in <module>
    main()
  File "extract_attention.py", line 108, in main
    example = Example(features, tokenizer, args.max_sequence_length)
  File "extract_attention.py", line 29, in __init__
    self.input_ids = tokenizer.convert_tokens_to_ids(self.tokens)
  File "/Users/smap10/Project/attention-analysis-master/bert/tokenization.py", line 182, in convert_tokens_to_ids
    return convert_by_vocab(self.vocab, tokens)
  File "/Users/smap10/Project/attention-analysis-master/bert/tokenization.py", line 143, in convert_by_vocab
    output.append(vocab[item])
KeyError: 'Silsby'

In readme ,I can see ''Each line in the files should contain a word followed by a space followed by <index_of_head>-<dependency_label> (e.g., 0-root).''

So how can I get the index_of_head, and what's the meaning of it?

Should I know which heads(12*12) in bert is for which syntactic function firstly ,and then judge it?
Sorry for my pool English.

It should have been inserted at 0th index in -1 dimension, currently its added at last index. Since attention approximated for ROOT by adding start/end tokens would be at 0th index, it would expect word rep for root should also be at 0.

By fixing this bug I got around 3% higher UAS.

word_reprs = tf.concat([word_reprs, tf.zeros((n_words, 1, 200))], 1) # dummy for ROOT

Should be replaced with,

word_reprs = tf.concat([tf.zeros((n_words, 1, 200)), word_reprs], 1) # dummy for ROOT

Any plans of releasing the code for coreference analysis in the paper?

Alternatively, it is possible to explain the methodology? Mainly around how the "head" word is chosen, and what is the exact set of antecedents used.

First of I want to say this is a great insight into the BERT model. But there seems to be a file missing from General Analysis. The file as said can be used to generate one's own wiki data.
(create_wiki_data.py). It would be quite helpful if you provide it .

Thank you for releasing the codes.

An explanation for head_distances.py is missing in README.
Could you add the explanation?

A small question: in head_distances.py, the line utils.write_pickle(js_distances, args.outfile) should be outside the "for loop" for i, doc in enumerate(data):?

The code in this line means set segment ids to zeros for both segments, I don't know whether this is a bug?

Hi, I found that dev_attn.pkl and train_attn.pkl for syntax analysis have all heads of 0. And the relns are all 'root'.