juliahealth / biomedquery.jl

Julia utilities for interacting with biomedical databases and biomedical application programming interfaces (APIs)

BioMedQuery

BioMedQuery is tested against Julia 1.X on Linux, and OS X.

Latest CI Build

Documentation

Stable	Latest	Examples

Description

Julia utilities to process and save results from BioMedical databases/APIs.

BioServices.jl (part of BioJulia) provides the basic interface to some of the APIs, while BioMedQuery helps parse and save results into MySQL, SQLite, DataFrames, CSV etc.

Supported APIs include:

National Library of Medicine (NLM)

• Entrez Programming Utilities (E-Utilities)
• Unified Medical Language System (UMLS)
• Clinical Trials (clinicaltrials.gov)
• MEDLINE (PubMed MEDLINE)

Installation

BioMedQuery is a registered package. To install the latest stable version, use the package manager.

using Pkg
Pkg.add("BioMedQuery")

To use the latest development version:

using Pkg
Pkg.add("BioMedQuery#master")

To checkout the latest development version:

using Pkg
Pkg.dev("BioMedQuery")

biomedquery.jl's Issues

Pkg.build(BioMedQuery) error on Julia 0.6.4

Hello,

I am having an issue when I attempting to build BioMedQuery on Julia 0.6.4 and get the following build error with one of BioMedQuery's dependencies SQLite. I am running the Julia on windows 10 machine.

`julia> Pkg.build("BioMedQuery")
INFO: Building CodecZlib
INFO: Building LibCURL
INFO: Building LightXML
INFO: Building MbedTLS
INFO: Building SpecialFunctions
INFO: Building DecFP
INFO: Building MySQL
INFO: Building Gumbo
INFO: Building WinRPM
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Building SQLite
INFO: Updating WinRPM package list
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Nothing to do
===================================================[ ERROR: SQLite ]====================================================

LoadError: Provider BinDeps.PackageManager failed to satisfy dependency sqlite3_lib
while loading C:\Users\Thor.julia\v0.6\SQLite\deps\build.jl, in expression starting on line 19

========================================================================================================================

====================================================[ BUILD ERRORS ]====================================================

WARNING: SQLite had build errors.

packages with build errors remain installed in C:\Users\Thor.julia\v0.6
build the package(s) and all dependencies with Pkg.build("SQLite")
build a single package by running its deps/build.jl script

julia>`

Save additional fields from articles

• The PMCID and DOI (when they are listed)
• Journal information (including the ISSN, title, and abbreviation)
• The publication month

Support JuliaDB Backend

How do we feel about this?

CT module still uses Requests

Needs to be updated to HTTP

Use Literate.jl for up keeping examples

Literate.jl - neat way to marry notebooks + documenter

Update to DataFrames v0.11

Exclude tests that require ENV during pull requests

Need to modify travis.yml

Update to MySQL v0.5.x

Travis isn’t passing on master

EXCEPTION_ACCESS_VIOLATION

Hello, I am trying to use MySQL to access my local database and submit a query, however when I run the following code I get this error. This seems like a privileges issue however, I am unsure on how to resolve it.
This was done on Julia v0.7.0, on a Windows 10 PC.

`
analysis = "ptb"
con = MySQL.connect("127.0.0.1", "root", password, db = "$(analysis)_pubmed")

command = """SELECT DISTINCT g.pmid, g.hgnc_id, count(*) as count
FROM pubtator_genes g
WHERE g.passage_type="abstract"
GROUP BY g.pmid, g.hgnc_id
ORDER BY count;"""

abstract_df = BioMedQuery.DBUtils.db_query(con, command)
`

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6fc4ce0b -- jl_get_nth_field at /home/Administrator/buildbot/worker/package_win32/build/src\datatype.c:837
in expression starting at untitled-f9149ad6a13ab84805b2ee55adc59766:19
jl_get_nth_field at /home/Administrator/buildbot/worker/package_win32/build/src\datatype.c:837
jl_f_getfield at /home/Administrator/buildbot/worker/package_win32/build/src\builtins.c:757
getproperty at C:\Users\Thor.julia\packages\Tables\0M7Kt\src\utils.jl:27 [inlined]
macro expansion at C:\Users\Thor.julia\packages\Tables\0M7Kt\src\utils.jl:55 [inlined]
eachcolumn at C:\Users\Thor.julia\packages\Tables\0M7Kt\src\utils.jl:47
unknown function (ip: 10DDF932)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
buildcolumns at C:\Users\Thor.julia\packages\Tables\0M7Kt\src\fallbacks.jl:95 [inlined]
columns at C:\Users\Thor.julia\packages\Tables\0M7Kt\src\fallbacks.jl:149 [inlined]
Type at C:\Users\Thor.julia\packages\DataFrames\IKMvt\src\other\tables.jl:21
jl_fptr_args at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1839
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
|> at .\operators.jl:813
jl_fptr_args at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1839
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
db_query at C:\Users\Thor.julia\packages\BioMedQuery\K4kwC\src\DBUtils\mysql_db_utils.jl:87
jl_fptr_args at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1839
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:363 [inlined]
eval_body at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:799
unknown function (ip: FFFFFFFE)
unknown function (ip: 1726EA9F)
unknown function (ip: 00000001)
jl_interpret_toplevel_thunk at /home/Administrator/buildbot/worker/package_win32/build/src\interpreter.c:808
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win32/build/src\toplevel.c:831
jl_parse_eval_all at /home/Administrator/buildbot/worker/package_win32/build/src\ast.c:841
jl_load_file_string at /home/Administrator/buildbot/worker/package_win32/build/src\ast.c:871
include_string at .\loading.jl:1002
include_string at C:\Users\Thor.julia\packages\CodeTools\xGemk\src\eval.jl:30
unknown function (ip: 10DA2EF1)
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
#116 at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:94
withpath at C:\Users\Thor.julia\packages\CodeTools\xGemk\src\utils.jl:30
withpath at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:46 [inlined]
#115 at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:93 [inlined]
with_logstate at .\logging.jl:397
with_logger at .\logging.jl:493 [inlined]
#114 at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:92 [inlined]
hideprompt at C:\Users\Thor.julia\packages\Atom\OZKl0\src\repl.jl:87
jl_fptr_args at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1839
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
macro expansion at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:91 [inlined]
#113 at C:\Users\Thor.julia\packages\Atom\OZKl0\src\eval.jl:86
jl_fptr_args at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1839
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
handlemsg at C:\Users\Thor.julia\packages\Atom\OZKl0\src\comm.jl:164
unknown function (ip: 10D85D2A)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win32/build/src\julia.h:1538 [inlined]
jl_f__apply at /home/Administrator/buildbot/worker/package_win32/build/src\builtins.c:563
#19 at .\task.jl:262
unknown function (ip: 10D82BB6)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win32/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win32/build/src\julia.h:1538 [inlined]
start_task at /home/Administrator/buildbot/worker/package_win32/build/src\task.c:268
wmain at /home/Administrator/buildbot/worker/package_win32/build/ui\repl.c:233
Allocations: 37273076 (Pool: 37270984; Big: 2092); GC: 195

Bug: Travis is currently not passing on the master branch

Execute notebooks from literate

Execution is turned off. This should change

Implement MTI in pure Julia

At the moment, the interface to the Medical Text Indexer batch utility, MTI calls the Java API through and example-executable. Not only it depends on Java utilities installed to compile the code, it also does not allow a clean manipulation of all inputs/and outputs.

Improve coverage

Our coverage has dropped, although the report is strange... Seems that initialization lines are not hit. Is this a Julia compiler trick?

Add support for recent versions of MySQL.jl

Greetings!

I tried using BioMedQuery.jl for the first time today. Installation went smoothly, however, when I attempt to load the package, I run into an error during the pre-compilation step:

julia> using BioMedQuery
[ Info: Precompiling BioMedQuery [e96904bf-1073-5077-9b57-b0ce0ff5555a]
ERROR: LoadError: LoadError: LoadError: UndefVarError: MySQLHandle not defined
Stacktrace:
 [1] top-level scope at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/DBUtils/mysql_db_utils.jl:55
 [2] include(::String) at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/BioMedQuery.jl:4
 [3] top-level scope at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/DBUtils/DBUtils.jl:4
 [4] include(::String) at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/BioMedQuery.jl:4
 [5] top-level scope at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/BioMedQuery.jl:21
 [6] top-level scope at none:2
 [7] eval at ./boot.jl:331 [inlined]
in expression starting at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/DBUtils/mysql_db_utils.jl:55
in expression starting at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/DBUtils/DBUtils.jl:4
in expression starting at /home/keith/.julia/packages/BioMedQuery/Ps32f/src/BioMedQuery.jl:21
ERROR: Failed to precompile BioMedQuery [e96904bf-1073-5077-9b57-b0ce0ff5555a] to /home/keith/.julia/compiled/v1.4/BioMedQuery/I7mWp_L3y04.ji.

Looking at the code for src/DBUtils/mysql_db_utils.jl, I noticed that the function db_query() is attempting to call MySQL.query(), which doesn't appear to be in the current API for MySQL.jl.

I'm guessing that the MySQL.jl API has changed in some recent version, and that BioMedQuery.jl is still referring to the old interface? It's also possible that I am just misunderstanding something..

As a side note, I also noticed that init_mysql_database() function that gets automatically called during the initial attempt to load BioMedQuery.jl attempts to look for the "test" database. Since it is common to remove the test db after installation, it might be good to avoid this, if possible.

Thanks!

System Info

• Arch Linux 5.6.14 (64-bit)
• Julia 1.4.2
• MariaDB 10.4.13

Package versions:

• [39abe10b] MySQL v1.0.4
• [e96904bf] BioMedQuery v0.6.6 #master (https://github.com/JuliaHealth/BioMedQuery.jl.git)

ERROR: LoadError: (2): File not found

Hey I am trying to run the following the save.efetch function in Julia v1.4.0. On Windows 10.

When I attempt to run this code:
BioMedQuery.PubMed.save_efetch!(conn, efetch_doc, true, true)

I get the following error:

Saving 15 articles to database
┌ Warning: getindex(df::DataFrame, col_ind::ColumnIndex) is deprecated, use df[!, col_ind] instead.
│ caller = select_columns(::MySQL.Connection, ::String) at mysql_db_utils.jl:62
└ @ BioMedQuery.DBUtils C:\Users\antho.julia\packages\BioMedQuery\K4kwC\src\DBUtils\mysql_db_utils.jl:62
ERROR: LoadError: (2): File 'C:UsersanthoDocumentsMachine_LearningWork2020-04-17_PubMed_basic.csv' not found (Errcode: 2)
Stacktrace:
[1] execute!(::MySQL.Connection, ::String) at C:\Users\antho.julia\packages\MySQL\R2gKS\src\MySQL.jl:84
[2] db_insert!(::MySQL.Connection, ::Dict{String,DataFrame}, ::String, ::String; verbose::Bool, drop_csv::Bool) at C:\Users\antho.julia\packages\BioMedQuery\K4kwC\src\PubMed\pubmed_sql_utils.jl:482
[3] save_efetch!(::MySQL.Connection, ::XMLElement, ::Bool, ::Bool) at C:\Users\antho.julia\packages\BioMedQuery\K4kwC\src\PubMed\eutils_sql_save.jl:25
[4] create_mysql_pubmed_db(::String, ::String, ::XMLElement) at C:\Users\antho\Documents\Machine_Learning\Work\collect_data.jl:132
[5] main() at C:\Users\antho\Documents\Machine_Learning\Work\collect_data.jl:604
[6] top-level scope at C:\Users\antho\Documents\Machine_Learning\Work\collect_data.jl:627
[7] include(::Module, ::String) at .\Base.jl:377
[8] exec_options(::Base.JLOptions) at .\client.jl:288
[9] _start() at .\client.jl:484
in expression starting at C:\Users\antho\Documents\Machine_Learning\Work\collect_data.jl:627

This code seems to work on MacOS however I have not been able to get this running on Windows.

Make MySQL/SQLite optional

With the DF backend completed, we should move towards an optional backend approach

Update BioMedQuery to be v0.7 & v1.0 compatible

Parsed DataFrames are type unstable

Data-frames created here are type unstable. We should predetermined the types of values and missing ones.

Allow for use of local MEDLINE database

Check if the host has available a local redline database for faster access to basic and mesh info.

Is a flag the cleanest way to implement this?
Or are functors in a similar fashion as what's currently done for MySQL/SQLite backend

Pkg.build(BioMedQuery) error on Julia 0.6.4

Hello,

I am having an issue when I attempting to build BioMedQuery on Julia 0.6.4 and get the following build error with one of BioMedQuery's dependencies SQLite. I am running the Julia on windows 10 machine.

`julia> Pkg.build("BioMedQuery")
INFO: Building CodecZlib
INFO: Building LibCURL
INFO: Building LightXML
INFO: Building MbedTLS
INFO: Building SpecialFunctions
INFO: Building DecFP
INFO: Building MySQL
INFO: Building Gumbo
INFO: Building WinRPM
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Building SQLite
INFO: Updating WinRPM package list
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_Leap_42.2/repodata/repomd.xml
INFO: Nothing to do
===================================================[ ERROR: SQLite ]====================================================

LoadError: Provider BinDeps.PackageManager failed to satisfy dependency sqlite3_lib
while loading C:\Users\Thor.julia\v0.6\SQLite\deps\build.jl, in expression starting on line 19

========================================================================================================================

====================================================[ BUILD ERRORS ]====================================================

WARNING: SQLite had build errors.

• packages with build errors remain installed in C:\Users\Thor.julia\v0.6
• build the package(s) and all dependencies with Pkg.build("SQLite")
• build a single package by running its deps/build.jl script

========================================================================================================================

julia>`

Bring back mkdocs for documentation

Documenter default output is niw html. Using mkdocs with material theme requires few extra steps. Do we want to do so?

Option of using DataFrame as a "backend"

In addition to the current MySQL and SQLite backends, it would be nice to give users the option of using a DataFrame as a "backend". This removes the dependency on a relational database and makes sense for smaller queries where we would prefer to do everything in memory.

Migrate from Nullables to Missings

Error tagging new release

The tag name "v.0.3" is not of the appropriate SemVer form (vX.Y.Z).
cc: @mirestrepo

UML processes with DataFrame backend

Right now most of the processes that involve UMLS queries support only a MySQL backend. It would be neat to have all work only with DataFrames

TODO: Set up bors in this repository

@ibacher Can you do this? You need to make a pull request that does the following two things:

1. Edit the .travis.yml to only build these three branches: master, staging, trying.

2. Create a bors.toml file.

This link gives instructions for both of those steps: https://bors.tech/documentation/getting-started/

Once you’ve done that, comment “bors r+” on that pull request.

You also will need to disable Travis pull request builds, which are redundant when we use bors, and are thus a waste or resources. I’ve already done this part.

Return a dictionary of idx2pmid for umls filtered occurrences

• Also create a generic type holding matrix - and indices dictionaries together