ML-model for solving the problem of searching semantically similar text documents.
The model processes a new incoming text question and returns a list of N similar questions from an existing dataset of 4,567 medical-related questions.
Unsupervised learner for implementing neighbor searches: Nearest Neighbors
Metric to use for distance computation: cosine
- Tokenization: TfidfVectorizer
tokenizer
: custom function with regular patternr"(?u)\b\w\w+\b"
token_pattern
:None
- Stop words: nltk.corpus.stopwords
- Minimum word length:
2
- Remove punctuation
- Decode special symbols:
html.unescape
- Lemmatization: spaCy
en_core_web_sm
model
question
- string
input question
questions
- List[str]
output similar questions (from Dataset)
Top-N accuracy (accuracy@n)
Accuracy@5
- 0.887
git clone https://github.com/DimionX/semantic-text-matching.git
cd semantic-text-matching
python -m venv .venv
source .venv/bin/activate
Dev:
pip install -r requirements.dev.txt
pre-commit install
Production:
pip install -r requirements.txt
Dataset info - Medical Questions Pairs
python ./src/load_dataset.py
python -m spacy download en
python ./src/save_model.py
streamlit run ./src/stream.py
Streamlit app in your browser: http://127.0.0.1:8501
python ./src/matcher.py
docker compose up
Streamlit app in your browser: http://127.0.0.1:8501