pradeepvemulakonda / snomed

Create a Neo4j database to import AMT/Snomed drug files and provide a REST interface to access the data

Snomed - Python

For a Groovy version => https://github.com/pradeepvemulakonda/snomed-groovy

Wiki => https://github.com/pradeepvemulakonda/Snomed/wiki

Environment: mac(any linux or windows env) with python 3.6.1 and py2neo 3.1.2. Use a virtual environment to avoid any conflicts or errors.

CreateCreate a Neo4j database to import Snomed drug files and provide a REST interface to access the data.

For information reg Neo4j please visit http://neo4j.com/

Application is written in python and accepts a Snomed-CT zip file. The data from Snomed-CT around 3 miliion rows of TDV files will be uploaded to the graph database Neo4j.

Neo4j is a NoSql graph database which has a rich query capabilities which makes it ideal for decesion support and BI.

Prepare files to be uploaded to Neo4j

• Use Neo4j version 3.2.1
• update the config.ini file in the config folder as required. Only update the zip folder path.
• Run the run_snomed_upload.py
• the updated concepts and relationship files with the term and description type id is present in the results folder.
• the script uploads concepts, synonyms and relations into a neo4j instance.
• The graph database url should be specified in config.ini file.
• The entire script takes about ~ 2 hours to upload a snomed snapshot into a new neo4j database.
• Update of the Snomed Concepts is done using threads.
• Update of Relationships is done using a single thread as Neo4j is throwing deadlock exceptions if we process the relations in parallel.

snomed's Issues

ImportError: No module named configparser

Hi Pradeep,
I am encountering the below error: any solutions pls

:\Python27\python.exe E:/SNOMED-CT/NEO4J1/run_snomed_upload.py
Traceback (most recent call last):
File "E:/SNOMED-CT/NEO4J1/run_snomed_upload.py", line 3, in
from config.read_config import SnomedConfig
File "E:\SNOMED-CT\NEO4J1\config\read_config.py", line 1, in
import configparser
ImportError: No module named configparser

Process finished with exit code 1

attribute error queue

HI pradeep - I'm getting the following error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/home/arron/Downloads/Snomed/worker/neo4j_upload_worker.py", line 36, in run
    self.queue.task_done()
AttributeError: 'Queue' object has no attribute 'task_done'

and using python 3.4.3 and py2neo 2.0.9.

have you come across this?

could not convert string to float: 'id'

Hi Pradeep -do you get this error?:

Traceback (most recent call last):
  File "run_snomed_upload.py", line 84, in <module>
    rte.execute(sg, sp)
  File "/home/arron/Downloads/Snomed/worker/thread_executor.py", line 27, in execute
    for item in item_generator.generate:
  File "/usr/lib/python3.4/csv.py", line 109, in __next__
    self.fieldnames
  File "/usr/lib/python3.4/csv.py", line 96, in fieldnames
    self._fieldnames = next(self.reader)
ValueError: could not convert string to float: 'id'

data is like -

|id|,|effectiveTime|,|active|,|moduleId|,|sourceId|,|destinationId|,|relationshipGroup|,|typeId|,|characteristicTypeId|,|modifierId|,|term|,|descType|,|relLabel|
|100022|,|20090731|,|0|,|900000000000207008|,|100000000|,|102272007|,|0|,|116680003|,|900000000000011006|,|900000000000451002|,|Is a (attribute)|,|900000000000003001|,|IS_A|
|101021|,|20020131|,|1|,|900000000000207008|,|10000006|,|29857009|,|0|,|116680003|,|900000000000011006|,|900000000000451002|,|Is a (attribute)|,|900000000000003001|,|IS_A|
|102025|,|20020131|,|1|,|900000000000207008|,|10000006|,|9972008|,|0|,|116680003|,|900000000000011006|,|900000000000451002|,|Is a (attribute)|,|900000000000003001|,|IS_A|
|103024|,|20030131|,|0|,|900000000000207008|,|1000004|,|19130008|,|0|,|116680003|,|900000000000011006|,|900000000000451002|,|Is a (attribute)|,|900000000000003001|,|IS_A|

and same quote/delimiter format put into snomed_concept_processor.py (which works fine).

The run_snomed_upload.py script throws an exception

We tried your run_snomed_upload.py script and it fails with

python run_snomed_upload.py
starting processing snomed zip file ...
start time Wed Feb 10 17:55:57 2016
Found the zip file /home/ronan/SNOMED/archive/SnomedCT_RF1Release_US1000124_20150901.zip and stating processing ...
/home/ronan/SNOMED/archive/SnomedCT_RF1Release_US1000124_20150901.zip
<zipfile.ZipFile object at 0xf85dd0>
<zipfile.ZipFile object at 0xf90a10>
Zip file is extracted into required files
loading descriptions ...
Traceback (most recent call last):
  File "run_snomed_upload.py", line 42, in <module>
    descMap = ld.process()
  File "/data/home/ronan/SNOMED/load/load_desc.py", line 12, in process
    input_file = super().get_files('descfile')
TypeError: super() takes at least 1 argument (0 given)

We modified super() to

super(DescProcessor,self).get_files('descfile')

and now it fails with

File "run_snomed_upload.py", line 42, in <module>
  descMap = ld.process()
File "/data/home/ronan/SNOMED/load/load_desc.py", line 14, in process
  with open(input_file[0], "rt", encoding='utf-8') as tsvin:
TypeError: 'NoneType' object has no attribute '__getitem__'

Earlier in the script, we get "Zip file is extracted into required files". We were expecting they would be placed in snomed_files/snomed per the configuration file. However, this folder is empty.