Coder Social home page Coder Social logo

explosion / healthsea Goto Github PK

View Code? Open in Web Editor NEW
84.0 10.0 17.0 58.32 MB

Healthsea is a spaCy pipeline for analyzing user reviews of supplementary products for their effects on health.

License: MIT License

Python 100.00%
machine-learning natural-language-processing pipeline text-classification spacy healthcare named-entity-recognition

healthsea's Introduction

Welcome to Healthsea โœจ

Create better access to health with spaCy.

Healthsea is a pipeline for analyzing user reviews to supplement products by extracting their effects on health.

Learn more about Healthsea in our blog post!

๐Ÿ’‰ Creating better access to health

Healthsea aims to analyze user-written reviews of supplements in relation to their effects on health. Based on this analysis, we try to provide product recommendations. For many people, supplements are an addition to maintaining health and achieving personal goals. Due to their rising popularity, consumers have increasing access to a variety of products.

However, it's likely that most of the products on the market are redundant or produced in a "quantity over quality" fashion to maximize profit. The resulting white noise of products makes it hard to find the right supplements.

Healthsea automizes the analysis and provides information in a more digestible way. โœจ


๐ŸŸข Requirements

To run this project you need:

spacy>=3.2.0
benepar>=0.2.0
torch>=1.6.0
spacy-transformers>=1.1.2

You can install them in the project folder via spacy project run install

๐Ÿ“– Documentation

Documentation
๐Ÿงญ Usage How to use the pipeline
โš™๏ธ Pipeline Learn more about the architecture of the pipeline
๐Ÿช spaCy project Introduction to the spaCy project
โœจ Demos Introduction to the Healthsea demos

๐Ÿงญ Usage

The pipeline processes reviews to supplements and returns health effects for every found health aspect.

You can either train the pipeline yourself with the provided datasets in the spaCy project or directly download the trained Healthsea pipeline from Huggingface via pip install https://huggingface.co/explosion/en_healthsea/resolve/main/en_healthsea-any-py3-none-any.whl

import spacy

nlp = spacy.load("en_healthsea")
doc = nlp("This is great for joint pain.")

# Clause Segmentation & Blinding
print(doc._.clauses)

>    {"split_indices": [0, 7],
>    "has_ent": true,
>    "ent_indices": [4, 6],
>    "blinder": "_CONDITION_",
>    "ent_name": "joint pain",
>    "cats": {
>        "POSITIVE": 0.9824668169021606,
>        "NEUTRAL": 0.017364952713251114,
>        "NEGATIVE": 0.00002889777533710003,
>        "ANAMNESIS": 0.0001394189748680219
>    },
>    "prediction_text": ["This", "is", "great", "for", "_CONDITION_", "!"]}

# Aggregated results
print(doc._.health_effects)

>    {"joint_pain": {
>        "effects": ["POSITIVE"],
>        "effect": "POSITIVE",
>        "label": "CONDITION",
>        "text": "joint pain"
>    }}


โš™๏ธ Pipeline

The pipeline consists of the following components:

pipeline = [sentencizer, tok2vec, ner, benepar, segmentation, clausecat, aggregation]

It uses Named Entity Recognition to detect two types of entities Condition and Benefit.

Condition entities are defined as health aspects that are improved by decreasing them. They include diseases, symptoms and general health problems (e.g. pain in back). Benefit entities on the other hand, are desired states of health (muscle recovery, glowing skin) that improve by increasing them.

The pipeline uses a modified model that performs Clause Segmentation based on the benepar parser, Entity Blinding and Text Classification. It predicts four exclusive effects: Positive, Negative, Neutral, and Anamnesis.


๐Ÿช spaCy project

The project folder contains a spaCy project with all the training data and workflows.

Use spacy project run inside the project folder to get an overview of all commands and assets. For more detailed documentation, visit the project folders readme.

Use spacy project run install to install dependencies needed for the pipeline.

โœจ Demo

Healthsea Demo

A demo for exploring the results of Healthsea on real data can be found at Hugging Face Spaces.

Healthsea Pipeline

A demo for exploring the Healthsea pipeline with its individual processing steps can be found at Hugging Face Spaces.

healthsea's People

Contributors

svlandeg avatar thomashacker avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

healthsea's Issues

ValueError: [E853] Unsupported component factory name 'healthsea.aggregation.v1'. The character '.' is not permitted in factory names.

These are the errors I get when I run this code:

import spacy

nlp = spacy.load("en_healthsea")
doc = nlp("This is great for joint pain")
print(doc._.health_effects)

These errors happen when I install spaCy 3.4.4

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
en-healthsea 0.0.1 requires spacy<3.3.0,>=3.2.0, but you have spacy 3.4.4 which is incompatible.
Traceback (most recent call last):
  File "/home/my_pc/scispaCy/spaCy lessons/Entity_recognition.py", line 24, in <module>
    nlp = spacy.load("en_healthsea")
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/__init__.py", line 54, in load
    return util.load_model(
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 432, in load_model
    return load_model_from_package(name, **kwargs)  # type: ignore[arg-type]
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 467, in load_model_from_package
    cls = importlib.import_module(name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/vasya/.local/lib/python3.10/site-packages/en_healthsea/__init__.py", line 4, in <module>
    from . import clause_aggregation
  File "/home/vasya/.local/lib/python3.10/site-packages/en_healthsea/clause_aggregation.py", line 6, in <module>
    @Language.factory("healthsea.aggregation.v1")
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/language.py", line 470, in factory
    raise ValueError(Errors.E853.format(name=name))
ValueError: [E853] Unsupported component factory name 'healthsea.aggregation.v1'. The character '.' is not permitted in factory names.

These errors happen when I install spaCy 3.2.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
spacy-transformers 1.1.9 requires spacy<4.0.0,>=3.4.0, but you have spacy 3.2.0 which is incompatible.
en-core-web-sm 3.4.1 requires spacy<3.5.0,>=3.4.0, but you have spacy 3.2.0 which is incompatible.
en-core-web-md 3.4.1 requires spacy<3.5.0,>=3.4.0, but you have spacy 3.2.0 which is incompatible.
en-core-web-lg 3.4.1 requires spacy<3.5.0,>=3.4.0, but you have spacy 3.2.0 which is incompatible.
Traceback (most recent call last):
  File "/home/my_pc/scispaCy/spaCy lessons/Entity_recognition.py", line 24, in <module>
    nlp = spacy.load("en_healthsea")
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/__init__.py", line 51, in load
    return util.load_model(
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 420, in load_model
    return load_model_from_package(name, **kwargs)  # type: ignore[arg-type]
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 453, in load_model_from_package
    return cls.load(vocab=vocab, disable=disable, exclude=exclude, config=config)  # type: ignore[attr-defined]
  File "/home/vasya/.local/lib/python3.10/site-packages/en_healthsea/__init__.py", line 14, in load
    return load_model_from_init_py(__file__, **overrides)
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 615, in load_model_from_init_py
    return load_model_from_path(
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 488, in load_model_from_path
    nlp = load_model_from_config(config, vocab=vocab, disable=disable, exclude=exclude)
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/util.py", line 525, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/language.py", line 1755, in from_config
    nlp = lang_cls(vocab=vocab, create_tokenizer=create_tokenizer, meta=meta)
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy/language.py", line 162, in __init__
    util.registry._entry_point_factories.get_all()
  File "/home/vasya/.local/lib/python3.10/site-packages/catalogue/__init__.py", line 119, in get_all
    result.update(self.get_entry_points())
  File "/home/vasya/.local/lib/python3.10/site-packages/catalogue/__init__.py", line 134, in get_entry_points
    result[entry_point.name] = entry_point.load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/__init__.py", line 1, in <module>
    from . import architectures
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/architectures.py", line 6, in <module>
    from .layers import TransformerModel, TransformerListener
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/layers/__init__.py", line 1, in <module>
    from .listener import TransformerListener
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/layers/listener.py", line 4, in <module>
    from ..data_classes import TransformerData
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/data_classes.py", line 12, in <module>
    from .util import transpose_list
  File "/home/vasya/.local/lib/python3.10/site-packages/spacy_transformers/util.py", line 9, in <module>
    from thinc.api import get_torch_default_device
ImportError: cannot import name 'get_torch_default_device' from 'thinc.api' (/home/vasya/.local/lib/python3.10/site-packages/thinc/api.py)

Minor change in the code

Hi!I noticed that in the following line of code in the preprocess_clausecat.py file, at line 61 in the for loop while splitting the dataset into train and test set

for label in label_dict:
        split = int(len(label_dict[label]) * eval_split)
        train += label_dict[label][split:]
        dev += label_dict[label][:split]
        checksum += len(label_dict[label])
        table_data.append(
            (
                label,
                len(label_dict[label]),
                len(label_dict[label][split:]),
                len(label_dict[label][:split]),
            )
        )

the train and dev assignment statements need to be interchanged. As per the existing assignment, The train set has fewer samples than the dev set. Shouldn't it be the other way round?
Something like this?

train += label_dict[label][:split]
dev += label_dict[label][split:]

Case with 2 conditions mix reviews

Sample input: Yam is great for joint pain but bad for liver disease
Code below:

`import spacy

nlp = spacy.load("en_healthsea")
doc = nlp("Yam is great for joint pain but bad for liver disease")

print(doc._.clauses)
print(doc._.health_effects)`

Output below:

[{'split_indices': (0, 11), 'has_ent': True, 'ent_indices': (4, 6), 'blinder': '_CONDITION_', 'ent_name': 'joint pain', 'cats': {'POSITIVE': 0.9715917110443115, 'NEUTRAL': 0.004003751091659069, 'NEGATIVE': 0.01367472019046545, 'ANAMNESIS': 0.010729866102337837}, 'prediction_text': ['Yam', 'is', 'great', 'for', '_CONDITION_', 'but', 'bad', 'for', 'liver', 'disease']}, {'split_indices': (0, 11), 'has_ent': True, 'ent_indices': (9, 11), 'blinder': '_CONDITION_', 'ent_name': 'liver disease', 'cats': {'POSITIVE': 0.9650901556015015, 'NEUTRAL': 0.004530671052634716, 'NEGATIVE': 0.023644359782338142, 'ANAMNESIS': 0.0067347269505262375}, 'prediction_text': ['Yam', 'is', 'great', 'for', 'joint', 'pain', 'but', 'bad', 'for', '_CONDITION_']}]

Expected 1 positive and another negative. Actually found 2 positive sentiments.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.