[Question]: Subtoken Labeling? about flair

[alanakbik]

Hello @quantarb we've had such issues before. In this case, I first use a regular tokenizer, and then additionally split all tokens on the offset positions to get the final tokenization. There is no helper function in Flair for this, so you would need to write your own tokenization code.

[quantarb]

Hi @alanakbik , thank you for your quick response. I tried to split up the tokens based on the offset positions, but I'm having problems restructoring my original flair sentence from tokens. What is the best way to reconstruct a flair sentence from tokens.

I tried several different approaches but my new_sentence never matches the original sentence.

text = """ BLAH BLAH BLAH BLAH"""
old_sentence = Sentence(text)
tokens = [Token(token.text) for token in sentence]
new_sentence = Sentence(tokens)

text = """ BLAH BLAH BLAH BLAH"""
old_sentence = Sentence(text)
tokens = [Token(text[token.start_position:token.end_position]) for token in sentence]
new_sentence = Sentence(tokens)

[MostHumble]

Hi @alanakbik , thank you for your quick response. I tried to split up the tokens based on the offset positions, but I'm having problems restructoring my original flair sentence from tokens. What is the best way to reconstruct a flair sentence from tokens.

I tried several different approaches but my new_sentence never matches the original sentence.

text = """ BLAH BLAH BLAH BLAH"""
old_sentence = Sentence(text)
tokens = [Token(token.text) for token in sentence]
new_sentence = Sentence(tokens)

text = """ BLAH BLAH BLAH BLAH"""
old_sentence = Sentence(text)
tokens = [Token(text[token.start_position:token.end_position]) for token in sentence]
new_sentence = Sentence(tokens)

it might have something to do with the fact that some (all?) tokenizer are lossy, you can try with a different tokenizer:

tokenized = your_tokenizer.tokenize(raw)
#print(tokenized)
sentence = Sentence(tokenized)
tagger.predict(sentence)