Visual communication of model status (loading)

Some reader requests can take longer than others (when reading long documents), implement a better "loading" visual cute. Right now it just says asking bert for answers .. perhaps a more prominent visual bar.

Revise Scoring and Answer Span Selection Method

Currently score is a sum of the start and end token probabilities. This might not be optimal.
Current there is no text token preprocessing (e.g. strip spaces, remove \n etc) which might introduce unexpected behaviour.
Currently, answer span is selected as highest proabability (start/end). There is opportunity to do better.

Resources

https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L1395

Add documentation on how to compile and test locally

It would be great to add instructions on how to compile and test the program on a local machine (without having to pip install neuralqa). Instructions must include:

how to build react front-end app
how to build python-based server

Contextual Query Expansion (api and explanation viz)

Enable CQE as a standalone API

Export CQE as standalone api ... eg. import MLMExpander, .expand()
Support visualization in notebooks

Retrieval username & password ignored

I specified an elasticsearch retrieval that requires authentication. I specified the username and password in the options section of the retrieval. When I try to run a query, the app fails with:

Error Fetching Passages. ConnectionError(<urllib3.connection.HTTPConnection object at 0x15cd3a490>: Failed to establish a new connection: [Errno 61] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x15cd3a490>: Failed to establish a new connection: [Errno 61] Connection refused)

Looking at the code seems to indicate that the username and password are not used anywhere in the code.

http_auth=('user', 'secret') required in ElasticSearchRetriever - will submit a pull request.

Backend

Deployment/Testing CI/CD

.. add build passing,
test passing,
doc passing badges to keep track of build errors, tests, docs

Support Addditional Retriever Type

Perhaps a similar Class to ElasticSearchRetriever for Solr.

Method to run query with support for relsnip and all other kwargs

Endpoint Access Control

Currently, there are no in built security check to manage access to the neuralqa rest end point. On the minimum, we want to enable/disable open access to the rest endpoint when we launch the ui.

Implement accesscontrol (e.g. apikey or uname/pass etc) for calls to the endpoints
find a clean way to enable just the frontned ui component authenticate

Resources

https://jwt.io/introduction/
Explore some of the Fastapi abstractions for this https://fastapi.tiangolo.com/tutorial/security/first-steps/

Improve RelSnip with Snippet Ranking

Currently, RelSnip provides lucene highlights to the reader. Can we further re-rank these highlights with BM25 query + highlight scores?. Explore implementation here and evaluate runtime costs. https://github.com/renatoviolin/Question-Answering-Albert-Electra/blob/master/rank_models/bm25_model.py
enable this as a backend config.yaml feature only. show all parameters in front end on demand.

Pytorch in requirements?

Both torch and torchvision are in requirements.txt and setup.py but they don't seem to be used anywhere. Any particular reason you have both torch and tensorflow in requirements?

Github actions - build and deploy docs

Github actions - build and deploy docs
Host docs on gh-pages branch

Config Interface Should Specify Retriever query

Specify the fields to search from
specify the main result field
specify any maximum retrieved length on the result field

Failed import when attempting to run `python cli.py test'

I am attempting to follow the steps here: https://github.com/victordibia/neuralqa/blob/master/notes.md#notes-on-runningtesting-locally to run neuralqa locally. Everything seems to install fine, but when I run:

$ python3 neuralqa/cli.py test

I get the error:

Traceback (most recent call last):
  File "neuralqa/cli.py", line 2, in <module>
    from neuralqa.server import launch_server
ImportError: cannot import name 'launch_server'

'TFBertEmbeddings' object has no attribute 'word_embeddings'

Hi , when I run the code : model.bert.embeddings.word_embeddings
there is an error : 'TFBertEmbeddings' object has no attribute 'word_embeddings' .
Could anyone has the same problem ? Could you give me sone insights on how to fix this problem ?

Models in Memory

Figure out the right way to handle which models to hold in memory

Best way to hold models in memory and configure this from config.yaml
Use the readerPool class to manage which models are loaded in memory

Improve Explanations: Better explanation visualization, more Explanation methods

Improved Explanation Visualization

Currently, we use a simple color density approach to visualize importance. Early feedback suggests this is helpful for the user immediately see the most important words/tokens, but does not offer quantities or further interaction (e.g. top n). Can we make this better or provide better alternatives?

convert explanations to a single modal view: switcher between visualization ntypes
bar + density visualization for easier comparisons similar to what was done here?
Top n important words: show only highlights for the top x most important words?

More Explanation Methods

Currently, explanations are based on vanilla gradients. We might want to explore:

Integrated gradients - maybe explore
SmoothGrad -
GradCam maybe?

Useful Resources

https://www.tensorflow.org/tutorials/interpretability/integrated_gradients
See https://github.com/experiencor/deep-viz-keras
https://keras.io/examples/vision/grad_cam/
https://blog.fastforwardlabs.com/2020/06/22/how-to-explain-huggingface-bert-for-question-answering-nlp-models-with-tf-2.0.html
add this as a config.yaml option, and front end toggle too?

Write Tests

UI
- rest end points
Reader
Expander

Results in NeuralQA inconsistent with same model running on HF

I've tested a model that I've deployed on NeuralQa vs one deployed on HF and noticed that the same inputs are yielding different outputs even though it's using the exact same model. This can of course be attributed to a few things but I can't seem to identify the culprit.

Here's the context:

Question:
Are your handsets locked or unlocked?

Corpus:
['No, all our handsets are unlocked.','Since your SIM isn’t working in your handset while other SIM cards are, it might be an issue with your handset provider; or the mobile phone could be locked, meaning it only accepts SIM cards from a particular service provider. Please contact the handset dealer for more assistance.']

The following returns 'unlocked' which is the correct response:
See Demo on HuggingFace

I've configured the exact same model in NeuralQA (with relsnip disabled) and the result is 'locked' even though I'm feeding exactly the same inputs.
Here my log:

0:No, all our handsets are unlocked.
[{'answer': 'unlocked', 'took': 0.35032129287719727, 'start_probability': '0.92030567', 'end_probability': '0.00026586326', 'probability': '0.460418697912246', 'question': 'Are your handsets locked or unlocked?', 'context': 'no, all our handsets are unlocked '}]
1:Since your SIM isn’t working in your handset while other SIM cards are, it might be an issue with your handset provider; or the mobile phone could be locked, meaning it only accepts SIM cards from a particular service provider. Please contact the handset dealer for more assistance.
[{'answer': 'locked', 'took': 0.5319299697875977, 'start_probability': '0.9462091', 'end_probability': '0.007203659', 'probability': '0.48030819557607174', 'question': 'Are your handsets locked or unlocked?', 'context': 'since your sim isn ’ t working in your handset while other sim cards are, it might be an issue with your handset provider ; or the mobile phone could be locked , meaning it only accepts sim cards from a particular service provider. please contact the handset dealer for more assistance'}]

As you can see the 2nd answer gets a higher probability but that doesn't really make sense as it's exactly the same model.
The main difference is that the NeuralQA model is feeding the corpus content independently while in the HF example, we're feeding the entire corpus.

Any ideas on why this is happening?

Add documentation for customizing retrieval process

Firstly, thank you for creating one of the most useful project I have seen for Q&A and NLP. It would be great to add documentation for configuring the retrieval with:
1- Our own data imported into ElasticSearch
2- Custom retrievals.

Add documentation

Initial docs page describing the project which build tracking
Add interaction tracking for docs https://www.ericholscher.com/blog/2009/apr/5/adding-google-analytics-sphinx-docs/ . Add styling to docs too
Document configuration file
Document api endpoints

Update Webserver - Migrate to ASGI Server (fastapi)

Convert Flask to fastapi with its benefits (error handling, docs, testing)
Integrate with full asgi server .. uvicorn with support for threads and workers
add some documentation to readthedocs

Enable stemmer filter in elasticsearch index

I think it would be a good idea to update data_utils.py to include a Stemming filter by default when creating Elasticsearch indices. This would tremendously improve the results returned by ES.

Add description of Advanced Options

Add a link/popup that describes what each advanced settings option does?

Query Expansion - User in the loop design

Formalize definitions of QE (if QE .. then display/visualize). Add a dropdown to select which BERT model is used for expansion.
Create separate endpoint for expansion. This workflow decouples expansion from QA, making it an optional "user in the loop" step. Currently, we add the expand button, which the user can click to generate potential expansion terms.
If QE selected, show/highlight which terms were added to the query.
include explanation visualization. Show which token was expanded and why, i.e. a chart/map that shows how each token was expanded.
Add logic to backend to also detect named entities and not expand them.

In the example screenshot implementation above some terms are not expanded, while others are (blue). By default named entities (and pos terms such as proper nouse) are not expanded.

RuntimeError on a fresh install

(Seemingly) I successfully installed NeuralGIA with the following command: pip3 install neuralqa (based on the information on this page: https://victordibia.com/neuralqa/index.html). But when I would like to start the ui with the following command: neuralqa ui --port 5000, I just receive a lot of errors, with "RuntimeError: random_device could not be read" as the last one. What shall I do, what I can check? I did not found anything in the docs or in the FAQ...
Thanks
log.txt

Integrate and Test relevant BERT Models (Roberta, Albert, TinyBert) etc

Test other models beyond bert and distilbert ... roberta, albert, bart, t5 etc

v0.0.1alpha*

v0.01

Transformers 4.x Compatibility Issue

Issue

NeuralQA's requirements do not have pinned package versions, instead they use >= to pull the latest version of each library. With the latest release of transformers, there is a compatibility issue with NeuralQA <> Tensorflow <> Transformers.

UI Error

Error Message

INFO:     127.0.0.1:53966 - "POST /api/answers HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 394, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 376, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 188, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.8/site-packages/neuralqa/server/routehandlers.py", line 45, in get_answers
    answers = self.reader_pool.model.answer_question(
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 111, in answer_question
    answer = self.get_chunk_answer_span(model_input)
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 26, in get_chunk_answer_span
    answer_start, answer_end = self.get_best_start_end_position(
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 18, in get_best_start_end_position
    answer_start = tf.argmax(start_scores, axis=1).numpy()[0]
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 173, in argmax_v2
    return gen_math_ops.arg_max(input, axis, name=name, output_type=output_type)
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/gen_math_ops.py", line 837, in arg_max
    return arg_max_eager_fallback(
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/gen_math_ops.py", line 872, in arg_max_eager_fallback
    _result = _execute.execute(b"ArgMax", 1, inputs=_inputs_flat, attrs=_attrs,
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Could not find valid device for node.
Node:{{node ArgMax}}
All kernels registered for op ArgMax :
  device='XLA_CPU'; output_type in [DT_INT32, DT_INT64]; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]
  device='XLA_CPU_JIT'; output_type in [DT_INT32, DT_INT64]; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]
  device='CPU'; T in [DT_INT64]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT64]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT32]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT32]; output_type in [DT_INT32]
  device='CPU'; T in [DT_UINT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_UINT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_UINT8]; output_type in [DT_INT64]
  device='CPU'; T in [DT_UINT8]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT8]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT8]; output_type in [DT_INT32]
  device='CPU'; T in [DT_HALF]; output_type in [DT_INT64]
  device='CPU'; T in [DT_HALF]; output_type in [DT_INT32]
  device='CPU'; T in [DT_BFLOAT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]; output_type in [DT_INT64]
  device='CPU'; T in [DT_FLOAT]; output_type in [DT_INT32]
  device='CPU'; T in [DT_DOUBLE]; output_type in [DT_INT64]
  device='CPU'; T in [DT_DOUBLE]; output_type in [DT_INT32]
 [Op:ArgMax]
ERROR:uvicorn.error:Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 394, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 376, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 188, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.8/site-packages/neuralqa/server/routehandlers.py", line 45, in get_answers
    answers = self.reader_pool.model.answer_question(
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 111, in answer_question
    answer = self.get_chunk_answer_span(model_input)
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 26, in get_chunk_answer_span
    answer_start, answer_end = self.get_best_start_end_position(
  File "/usr/local/lib/python3.8/site-packages/neuralqa/reader/bertreader.py", line 18, in get_best_start_end_position
    answer_start = tf.argmax(start_scores, axis=1).numpy()[0]
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 173, in argmax_v2
    return gen_math_ops.arg_max(input, axis, name=name, output_type=output_type)
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/gen_math_ops.py", line 837, in arg_max
    return arg_max_eager_fallback(
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/ops/gen_math_ops.py", line 872, in arg_max_eager_fallback
    _result = _execute.execute(b"ArgMax", 1, inputs=_inputs_flat, attrs=_attrs,
  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Could not find valid device for node.
Node:{{node ArgMax}}
All kernels registered for op ArgMax :
  device='XLA_CPU'; output_type in [DT_INT32, DT_INT64]; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]
  device='XLA_CPU_JIT'; output_type in [DT_INT32, DT_INT64]; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]
  device='CPU'; T in [DT_INT64]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT64]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT32]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT32]; output_type in [DT_INT32]
  device='CPU'; T in [DT_UINT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_UINT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_UINT8]; output_type in [DT_INT64]
  device='CPU'; T in [DT_UINT8]; output_type in [DT_INT32]
  device='CPU'; T in [DT_INT8]; output_type in [DT_INT64]
  device='CPU'; T in [DT_INT8]; output_type in [DT_INT32]
  device='CPU'; T in [DT_HALF]; output_type in [DT_INT64]
  device='CPU'; T in [DT_HALF]; output_type in [DT_INT32]
  device='CPU'; T in [DT_BFLOAT16]; output_type in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]; output_type in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]; output_type in [DT_INT64]
  device='CPU'; T in [DT_FLOAT]; output_type in [DT_INT32]
  device='CPU'; T in [DT_DOUBLE]; output_type in [DT_INT64]
  device='CPU'; T in [DT_DOUBLE]; output_type in [DT_INT32]
 [Op:ArgMax]
INFO:     127.0.0.1:53982 - "GET /images/icon.png HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:53984 - "GET /manifest.json HTTP/1.1" 304 Not Modified
INFO:     127.0.0.1:53982 - "GET /android-chrome-192x192.png HTTP/1.1" 304 Not Modified

victordibia / neuralqa Goto Github PK

neuralqa's People

Contributors

Stargazers

Watchers

Forkers

neuralqa's Issues

Resources

Backend

Deployment/Testing CI/CD

Resources

Improved Explanation Visualization

More Explanation Methods

Useful Resources

Issue

UI Error

Error Message

Recommend Projects

Recommend Topics

Recommend Org