codeforafrica / healthtools.api

HealthTools API to retrieve, structure and return data being used by the health tools. Accessible at https://api.healthtools.codeforafrica.org/

HealthTools.API

Providing actionable health information.

The HealthTools API is a free to access, open, and structured service for newsrooms, civic organizations, and governments to access health information.

This API provides a simple wrapper around Elasticsearch data being made available through the HealthTools Scraper and as a cached proxy for websites unable to be reliably scraped such as Kenya's nursing council website.

Through the API you can access data that we've scraped from:

1. Doctors: http://medicalboard.co.ke/online-services/retention/
2. Foreign Doctors: http://medicalboard.co.ke/online-services/foreign-doctors-license-register
3. Clinical Officers: http://clinicalofficerscouncil.org/online-services/retention/
4. Health Facilities: http://kmhfl.health.go.ke
5. NHIF accredited outpatient facilities for civil servants: http://www.nhif.or.ke/healthinsurance/medicalFacilities
6. NHIF accredited inpatient facilities: http://www.nhif.or.ke/healthinsurance/inpatientServices
7. NHIF accredited outpatient facilities: http://www.nhif.or.ke/healthinsurance/outpatientServices

Usage

Specification for the API are:

URL Endpoint	HTTP Methods	Functionality	Parameters
/search/doctors	GET	Search a doctor by the name	q=[name]
/search/nurses	GET	Search a nurse by the name	q=[name]
/search/clinical-officers	GET	Search a clinical officer by the name	q=[name]
/search/health-facilities	GET	Search a health facility by the name	q=[name]
/search/nhif-outpatient	GET	Search a NHIF accredited outpatient facility by the name	q=[name]
/search/nhif-outpatient-cs	GET	Search a NHIF accredited outpatient facility for civil servants	q=[name]
/search/nhif-inpatient	GET	Search a NHIF accredited inpatient facility by the name	q=[name]
/search	GET	Search a query that contains a keyword Eg. HF Kitale	q=[query]
/sms	GET	Search a query received from sms Eg. Dr Jane	q=[phoneNumber, message]
/sms/[adapter]	GET	Search a query received from sms using the specified adapter Eg. mtech	q=[query]

Requirments

---

Development

To get started with developing on HealthTools.API, do the following:

git clone https://github.com/CodeForAfricaLabs/HealthTools.API.git`
cd HealthTools.API
touch .env
docker-compose build web
docker-compose up web

# Load latest data
docker-compose exec web flask htools loaddata

Reset Index

To reset the elasticsearch index, run:

docker-compose exec web flask htools resetindex

Tests

Use nosetests to run tests (with stdout) like so:

$ nosetests --nocapture
$ nosetests healthtools/tests
$ nosetests healthtools/tests/doctor.py
$ nosetests healthtools/tests/nurse.py
$ nosetests healthtools/tests/clinical_officer.py
$ nosetests healthtools/tests/health_facilities.py
$ nosetests healthtools/tests/nhif_inpatient.py
$ nosetests healthtools/tests/nhif_outpatient.py
$ nosetests healthtools/tests/nhif_outpatient_cs.py

Wit.ai

Wit.ai is a simple natural language processing API that helps developers turn speech and text into actionable data. In healthtools.API, we use it to distinguish the doc_type and the query param from the parameters being sent to the API. For example: dr mary

{
    "msg_id": "0csXmLxhRzgowQFJI",
    "_text": "dr mary",
    "entities": {
        "doctors": [
            {
                "confidence": 0.96676127393653,
                "value": "dr",
                "type": "value"
            }
        ],
        "query": [
            {
                "suggested": true,
                "confidence": 0.99538476989175,
                "value": "mary",
                "type": "value"
            }
        ]
    }
}

The doc_type has been identified as doctors and mary has been identified as query param. To configure wit.ai:

• Sign up on wit.ai website with either your github or facebook account
• Create an app with the name you prefer to use.
• Detect your first entities:

Improve the detection

This will be done by continuously training your app to identify keyword or sentences. Input a message and check its prediction, if wrong, correct it if right validate it.

Configure Server Access Token in the ENV.

The token can be found from the settings tab, API details segment.

To set it in the ENV, run export WIT_ACCESS_TOKEN='Token'

The following endpoint is used to run wit.ai on healthtools.API

URL Endpoint	HTTP Method	Functionality	Parameter
/search/wit	GET	Search Query	q=[query]

Contributing

If you'd like to contribute to HealthTools.API, check out the CONTRIBUTING.md file on how to get started.

---

License

MIT License

Copyright (c) 2017 Code for Africa

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

healthtools.api's Issues

[BROKEN] Nurses + Clinical Officers search is broken

Description

The nurses and clinical officers search is broken as they both have new sites:

• https://osp.nckenya.com/LicenseStatus#
• https://portal.clinicalofficerscouncil.org/practice/LicenseStatus

Screenshots

N/A

Files

N/A

To Reproduce

Self-explanatory from description.

Tasks

• Update scrapers on healthtools_ke
• Update API
• Update documentation

[SMS] Print the SMS in log and console in development

When in development, SMS should print the expected output in log and console for debug purposes.

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Create base SMS processor

This is the base SMS processor that processes the incoming SMSs.

Publish documentation with Github pages

The README.md is getting to small a space for the documentation of the HealthTools API. We should therefore move it to docs folder structured for Jekyll deployment with Github Pages (win!).

Read more on this: https://github.com/blog/2233-publish-your-project-documentation-with-github-pages

Use improved Github issues templates

Description

Github recently improved issue templates, we should therefore take advantage of this for improved contributing process - https://blog.github.com/2018-05-02-issue-template-improvements/

Screenshots

When a user opens a new issue:

Settings tab in repo to achieve this:

Files

• https://github.com/CodeForAfrica/HealthTools.API/tree/master/.github

Tasks

• Create the issue templates
• Delete issue templates in code

Provide user interaction feature

If we receive a story lead instead of a query, the API should allow journalists/newsrooms to follow up with the user via a dashboard.

The platform should provide for this functionality possibly allowing API calls using a securely generated token. Not necessarily providing the interface/dashboard. The dashboard can be separate to keep the API lean.

Use RabbitMQ for query processing

When a service we are depending on such as Elasticsearch fails, we will lose the queries (particularly via SMS that are costly) and will be unable to respond to the user.

Using RabbitMQ will allow storing the message in a queue and track the job until it is finished successfully on a number of retries or manually if intervention is needed.

Add Slack log handler

We should send notifications to Slack using a Slack Log handler https://github.com/mathiasose/slacker_log_handler

Related #52

Cache queries

On running run_query, we should cache the queries in Memcached.

Ref: #42

Upgrade runtime to Python3

This might bring about speed and other benefits.

Improve Documentation

The documentation should be updated to improve ability for contribution and generally tool build.

This includes in-line documentation and the README.md.

Dependabot can't evaluate your Python dependency files

Dependabot can't evaluate your Python dependency files.

As a result, Dependabot couldn't check whether any of your dependencies are out-of-date.

The error Dependabot encountered was:

Illformed requirement ["=="]

View the update logs.

Create base Bot processor

The base bot processor determine which "adapter" a request should be routed to and appropriate response.

Research using Django Rest Framework

The Django Rest Framework implements a very nice template look and feel for APIs. We should consider using it for the HealthTools.API - http://www.django-rest-framework.org

Specific to ElasticSearch, there is also http://django-rest-elasticsearch.readthedocs.io/en/latest/index.html

Documentation + Tests for v0.2.0

We should write documentation + test for v0.2.0 before closing the milestone.

Related #39

Create Facebook Bot adapter

Add feature to allow working with Facebook Bot. Related to #47

Development - v0.2

Through development of the HealthTools.API, we've learnt a lot and want to make it more extensible. v0.2 seeks to do a couple of things:

1. Views
1.1. Search

• Have a single search API view to receive requests and output results in JSON
• Send requests to a single query function used by sms, bots, etc.

1.2. SMS

• Have a single SMS view that receives requests and outputs results in JSON.
• Send requests to a single sms process function (detailed below)

2. Search

2.1 Query

• This would be the primary query module that determines the doc-type (e.g nurses, doctors, cos etc.)
• This would determine the search "vector" i.e elastic search or proxy for nurses etc.
• Return dictionary/list of results.

2.2. Nurses Search "Vector"

• Proxy the nurses website because we are unable to scrape it.

2.3. Elastic search "vector"

• Search from elastic search.

3. SMS

• The base module would process SMS requests using the query function and structure for an SMS response.
• This would then use "adapters" for the different providers e.g M-Tech, Twilio.

4. Other considerations:

• Tests to be rewritten
• Analytics to work as a Flask "Middleware" or have as part of an "events" module.
• Slack integration with Python's logging module.
• Cache queries using Memcached (all not just nurses as before)
• RabbitMQ integration for SMS processing.
• Wit.ai to replace determination from keywords what doc-type is being queried.
• Bots architecture similar to SMS. Probably re-using a lot of SMS so might be combined into just "Chat" that both inherit.

Wrap external calls with try catch statements

Currently external calls aren't "sandboxed" to avoid complete failure. We should therefore have every method doing an external call in try statements and raise exceptions appropriately.

Ref: #52

[SMS] Number each result

Currently we aren't numbering each result to be sent back via SMS. We should do that.

Add Twilio SMS "adapter"

Twilio is a great service that allows easy deployment in more countries. We should therefore create a Twilio adapter that returns the same SMS we are currently returning via M-Tech (sms provider).

In this case, we would re-work SMS functionality to have providers as "adapters".

Create Telegram Bot adapter

As part of the bot methods, we should add module to work with Telegram.

Include M-Tech API docs in our documentation

M-Tech has shared their API docs and we should probably have it living within the code instead of in Slack group. We should also check if instead there are docs online that we can point to.

• Mtech API.pdf
• MTECH SMS API V2.pdf

This is related to the "adapter" approach in #31 .

Publish documentation using Sphinx

While looking at #33 that seemed like the simplest approach / lowest hanging fruit and exciting to publish docs with, the better documentation publishing for Python is Sphinx.

Let's therefore use Sphinx instead following this - http://docs.python-guide.org/en/latest/writing/documentation/

Update test scripts

Description

Currently, the test runner cannot run any test case, it reports that there is no test (which is false).
After including the test files in the init file of the test module, some tests are failing due to changes in the implementation of some features, such as the returned value when a document was not found.

Screenshots

Nosetests report that no test was found

Some test failing

Files

The concerned files reside in the healthtools/tests folder

To Reproduce

Run the tests, some would fail

Use wit.ai to determine doc_type being searched for

Instead of the current keyword use to determine the type of document being searched for, we should use wit.ai for this to be more comprehensive.

Create M-Tech SMS Adapter

The M-Tech SMS adapter is the default SMS adapter and would be added to allow using the M-Tech service.

[SMS] Add Africa's Talking adapter

Add functionality for SMS to work with Africa's Talking.

Log significant info using Python logging module

We would like to log significant events using Python's logging module and eventually send specific notifications to Slack. For our purposes we want to log:

1. When the doc_type isn't understood so that:
   • Dev team can add keyword if it is appropriate.
   • Could be a question instead or a tip-off for newsroom to follow up with.
2. When an error occurs e.g ElasticSearch is down or the nurses website is down.

Add analytics as middleware or event

We should track the searches happening as either some sort of Flask middleware approach or as an event module. This would be sent to Google Analytics. We should track:

• Query
• Source (IP, SMS/Web, Bot etc)
• Results total found

Searching does not bring exact matches

When a search is carried out on the api, the search results brought aren't that accurate. For example searching for a doctor "john" will bring up multiple doctors with different names who went to John Hopkins university before the first "john" shows up(https://api.healthtools.codeforafrica.org/doctors/search.json?q=john). It is due to the searching being done over the qualifications fields