Use T5 as a highlighter about covidex

[daemon]

Writing down some notes:

• Eyeballing the T5-based highlighter as implemented seems to yield worse results (e.g., more super random highlightings of completely unrelated material, such as extraneous CC-BY attribution text).
• We tried dynamic query representation as well as fixed query representation, i.e., f'Query: {query} Document: {document} Relevant:' vs '{query}' and '{document}'.
• If we want to take advantage of caching the reranker, the T5-based highlighter is limited to a maximum sequence length of 256, whereas BioBERT to 512.
• The T5-based highlighter is 0-25% faster with reranker caching and 10-20% slower without.

[rodrigonogueira4]

I think the main problem is that we are using 256 tokens for the reranker. Could you please try increasing to 512 tokens? There might be only a small increase in latency because we were underutilizing the GPU when feeding it with 256 tokens..

Also, since we will then have a spare GPU, we can use it to cut the inference time by half (but that we can leave for another PR)

[daemon]

Sure, but the results won't be the same as the TensorFlow implementation. Is that okay?

[daemon]

I guess I can evaluate it on R04.

[rodrigonogueira4]

Yeah, evaluating on R04 is an even better idea

[santhoshkolloju]

can you throw some light on what do u mean highlighting...
can i get link to this BioBERT's highlighter?