bio2bel / kegg

A Bio2BEL package for integrating pathway-related information from KEGG in BEL

Bio2BEL KEGG

This package allows the enrichment of BEL networks with KEGG information by wrapping its RESTful API. Furthermore, it is integrated in the ComPath environment for pathway database comparison.

If you find this package useful, please consider citing [domingofernandez2018]:

[domingofernandez2018]

Domingo-Fernandez, D., et al (2018). ComPath: an ecosystem for exploring, analyzing, and curating mappings across pathway databases. Npj Systems Biology and Applications, __5__(1), 3.

Warning This package creates partOf relationships in BEL, but does not convert KEGG mechanistic relationships to BEL. That functionality is implemented in the PathMe project.

Installation

bio2bel_kegg can be installed easily from PyPI with the following code in your favorite terminal:

$ pip install bio2bel_kegg

or from the latest code on GitHub in development mode with:

$ git clone https://github.com/bio2bel/kegg.git
$ cd kegg
$ pip install -e .

Setup

KEGG can be downloaded and populated from either the Python REPL or the automatically installed command line utility.

Python REPL

>>> import bio2bel_kegg
>>> kegg_manager = bio2bel_kegg.Manager()
>>> kegg_manager.populate()

Command Line Utility

bio2bel_kegg populate

Other Command Line Utilities

• Run an admin site for simple querying and exploration python3 -m bio2bel_kegg web (http://localhost:5000/admin/)
• Export gene sets for programmatic use python3 -m bio2bel_kegg export-gene-sets -f tsv

Citation

• Kanehisa, Furumichi, M., Tanabe, M., Sato, Y., and Morishima, K.; KEGG: new perspectives on genomes, pathways, diseases and drugs. Nucleic Acids Res. 45, D353-D361 (2017).
• Kanehisa, M., Sato, Y., Kawashima, M., Furumichi, M., and Tanabe, M.; KEGG as a reference resource for gene and protein annotation. Nucleic Acids Res. 44, D457-D462 (2016).
• Kanehisa, M. and Goto, S.; KEGG: Kyoto Encyclopedia of Genes and Genomes. Nucleic Acids Res. 28, 27-30 (2000).

kegg's Issues

Get flake8 to passing

Add UniProt accession and identifier fields to Protein class

Website use cases

Pybel web connection

Exporting geneset + Pybel web importing

Select your pathway and export it as a genelist -> infer relationships in all databases in pybel web

Overlap

Select one pathway first and calculate overlap against other -> rendering figure

setup travis

Add hierarchy

http://www.kegg.jp/kegg-bin/download_htext?htext=br08901.keg&format=json

Manual mapping of KEGG/Reactome table

Aim: Diagram representing the overlap of corresponding pathways (add NeuroMMSig)

Add codecov reporting to .travis.yml

Upgrade to PyBEL 15 / Bio2BEL 4

Blocked by:

• Upgrade Bio2BEL with PyBEL 14 bio2bel/bio2bel#24
• Upgrade ComPath Utils with new PyBEL/Bio2BEL ComPath/ComPath-Utils#5
• Upgrade Bio2BEL-HGNC with new PyBEL/Bio2BEL bio2bel/hgnc#10

urllib.error.HTTPError: HTTP Error 403: Forbidden

kegg_manager.populate() failed as below.

(base) C:\Users\hoge>python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:22:46) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bio2bel_kegg
INFO: [2022-02-04 17:13:08] protmapper.resources - Could not access 'hgnc' resource file, will download.
INFO: [2022-02-04 17:13:08] protmapper.resources - Downloading 'hgnc' resource file into C:\Users\hoge\.data\protmapper\0.0.24\hgnc_entries.tsv.gz from cache.
INFO: [2022-02-04 17:13:10] protmapper.resources - Could not access 'up' resource file, will download.
INFO: [2022-02-04 17:13:10] protmapper.resources - Downloading 'up' resource file into C:\Users\hoge\.data\protmapper\0.0.24\uniprot_entries.tsv.gz from cache.
INFO: [2022-02-04 17:14:20] protmapper.resources - Could not access 'upsec' resource file, will download.
INFO: [2022-02-04 17:14:20] protmapper.resources - Downloading 'upsec' resource file into C:\Users\hoge\.data\protmapper\0.0.24\uniprot_sec_ac.txt.gz from cache.
>>> kegg_manager = bio2bel_kegg.Manager()
>>> kegg_manager.populate()
INFO: [2022-02-04 17:16:39] bio2bel.utils - downloading from source http://rest.kegg.jp/list/organism to C:\Users\hoge\.data\bio2bel\kegg\organisms.tsv
INFO: [2022-02-04 17:16:40] bio2bel.utils - downloading from source http://rest.kegg.jp/list/pathway/hsa to C:\Users\hoge\.data\bio2bel\kegg\pathways.tsv
Fetching protein information (3 threads):  29%|███████████▏                           | 99/345 [00:07<00:19, 12.81it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel\manager\abstract_manager.py", line 40, in populate_wrapped
    cls._populate_original(self, *populate_args, **populate_kwargs)
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\manager.py", line 172, in populate
    self._populate_pathways(url=pathways_url)
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\manager.py", line 95, in _populate_pathways
    pathways_lines = get_entities_lines(pathways_df['kegg_pathway_id'])
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\client.py", line 55, in get_entities_lines
    return list(results)
  File "C:\Users\hoge\miniforge3\lib\site-packages\tqdm\std.py", line 1180, in __iter__
    for obj in iterable:
  File "C:\Users\hoge\miniforge3\lib\multiprocessing\pool.py", line 868, in next
    raise value
  File "C:\Users\hoge\miniforge3\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\client.py", line 69, in ensure_kegg_entity
    urlretrieve(f'http://rest.kegg.jp/get/{entity_id}', entity_text_path)  # noqa:S310
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 640, in http_response
    response = self.parent.error(
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 502, in _call_chain
    result = func(*args)
  File "C:\Users\hoge\miniforge3\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
>>>

I'm using

>>> sys.version
'3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:22:46) [MSC v.1916 64 bit (AMD64)]'

Release new version fixing the connection problem with many requests

Add methods for calculating overlaps of genesets looking up kegg/reactome

Error: No such command 'export'.

python3 -m bio2bel_kegg export in
does not work.

c93f91e
is the commit I used for this.

(base) C:\Users\hoge>python -m bio2bel_kegg export
Usage: python -m bio2bel_kegg [OPTIONS] COMMAND [ARGS]...
Try 'python -m bio2bel_kegg --help' for help.

Error: No such command 'export'.

(base) C:\Users\hoge>python -m bio2bel_kegg --help
Usage: python -m bio2bel_kegg [OPTIONS] COMMAND [ARGS]...

  Default connection at
  sqlite:///C:/Users/hoge/.data/pybel/pybel_0.14.0_cache.db

  using Bio2BEL v0.4.2

Options:
  -c, --connection TEXT  [default: sqlite:///C:/Users/hoge/.data/pybel/pybel_0
                         .14.0_cache.db]
  --help                 Show this message and exit.

Commands:
  bel               Manage BEL.
  belns             Manage BEL namespace.
  cache             Manage cached data.
  drop              Drop the database.
  export-gene-sets  Export all pathway - gene info to a excel file.
  populate          Populate the database.
  summarize         Summarize the contents of the database.
  web               Run the web app.

Pass Pyroma

ModuleNotFoundError: No module named 'sqlalchemy.ext.declarative.api'

I got an error like

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\__init__.py", line 19, in <module>
    from .manager import Manager  # noqa: F401
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel_kegg\manager.py", line 10, in <module>
    from bio2bel.compath import CompathManager
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel\__init__.py", line 45, in <module>
    from .manager import AbstractManager, get_bio2bel_manager_classes  # noqa: F401
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel\manager\__init__.py", line 5, in <module>
    from .abstract_manager import AbstractManager, get_bio2bel_manager_classes
  File "C:\Users\hoge\miniforge3\lib\site-packages\bio2bel\manager\abstract_manager.py", line 14, in <module>
    from sqlalchemy.ext.declarative.api import DeclarativeMeta
ModuleNotFoundError: No module named 'sqlalchemy.ext.declarative.api'

when I did

(base) C:\Users\hoge>python
Python 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:22:46) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import bio2bel_kegg

Please let me know if you have any ideas.

I'm using the following package versions

bio2bel                   0.4.2                    pypi_0    pypi
bio2bel-kegg              0.3.0                    pypi_0    pypi
sqlalchemy                1.4.31           py38h294d835_0    conda-forge

Error on populate

I just did a totally clean installation with bio2bel_kegg populate and got the following error (some number of hours later...). It looks like it's not handling situations when multiple UniProt identifiers are listed.

Loading pathways: 100%|███████████████████████████████████████████████████████████████████████████████| 330/330 [00:00<00:00, 1082.08it/s]
Fetching meta information: 7528it [1:35:51,  1.31it/s]
Loading proteins: 100%|███████████████████████████████████████████████████████████████████████████| 30036/30036 [00:29<00:00, 1027.97it/s]
Traceback (most recent call last):
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
psycopg2.DataError: value too long for type character varying(255)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/manager/abstract_manager.py", line 38, in populate_wrapped
    cls._populate_original(self, *populate_args, **populate_kwargs)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/bio2bel_kegg/manager.py", line 215, in populate
    self._pathway_entity(url=protein_pathway_url, metadata_existing=metadata_existing)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/bio2bel_kegg/manager.py", line 172, in _pathway_entity
    self.session.commit()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 153, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
    self._flush(objects)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 249, in reraise
    raise value
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
    flush_context.execute()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
    execute(statement, params)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(255)
 [SQL: 'INSERT INTO kegg_protein (kegg_id, uniprot_id, hgnc_id, hgnc_symbol) VALUES (%(kegg_id)s, %(uniprot_id)s, %(hgnc_id)s, %(hgnc_symbol)s) RETURNING kegg_protein.id'] [parameters: {'kegg_id': 'hsa:3111', 'uniprot_id': 'P06340 X5CF87 A0A1V0E3P9 A0A1V0E3R6 A0A1V0E3Q3 A0A1V0E3Q1 A0A1V0E3N6 A0A1V0E3Q5 A0A1V0E3S4 A0A1V0E3P3 A0A1V0E3N7 A0A1V0E3R0 A0A1V0E3Q4 A0A1V0E3R4 A0A ... (34 characters truncated) ... V0E3R3 A0A1V0E3S6 A0A1V0E3P1 A0A1V0E3M7 A0A1V0E3P0 A0A1V0E3S0 A0A1V0E3T1 A0A1V0E3P6 A0A1V0E3Q2 A0A1V0E3Q8 A0A1V0E3R8 A0A1V0E3S7 A0A1V0E3N3 A0A1V0E3Q6', 'hgnc_id': '4936', 'hgnc_symbol': 'HLA-DOA'}] (Background on this error at: http://sqlalche.me/e/9h9h)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cthoyt/.virtualenvs/pybel/bin/bio2bel", line 11, in <module>
    load_entry_point('bio2bel', 'console_scripts', 'bio2bel')()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/cli.py", line 162, in write
    r = manager.write_directory(directory)
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/manager/namespace_manager.py", line 384, in write_directory
    self.write_bel_namespace(file, use_names=False)
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/manager/namespace_manager.py", line 346, in write_bel_namespace
    self.populate()
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/manager/abstract_manager.py", line 40, in populate_wrapped
    self._store_populate_failed()
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/manager/connection_manager.py", line 93, in _store_populate_failed
    Action.store_populate_failed(self.module_name, session=self.session)
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/models.py", line 95, in store_populate_failed
    _store_helper(action, session=session)
  File "/Users/cthoyt/dev/bio2bel/src/bio2bel/models.py", line 140, in _store_helper
    session.commit()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 153, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 465, in commit
    self._assert_active(prepared_ok=True)
  File "/Users/cthoyt/.virtualenvs/pybel/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 276, in _assert_active
    % self._rollback_exception
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (psycopg2.DataError) value too long for type character varying(255)
 [SQL: 'INSERT INTO kegg_protein (kegg_id, uniprot_id, hgnc_id, hgnc_symbol) VALUES (%(kegg_id)s, %(uniprot_id)s, %(hgnc_id)s, %(hgnc_symbol)s) RETURNING kegg_protein.id'] [parameters: {'kegg_id': 'hsa:3111', 'uniprot_id': 'P06340 X5CF87 A0A1V0E3P9 A0A1V0E3R6 A0A1V0E3Q3 A0A1V0E3Q1 A0A1V0E3N6 A0A1V0E3Q5 A0A1V0E3S4 A0A1V0E3P3 A0A1V0E3N7 A0A1V0E3R0 A0A1V0E3Q4 A0A1V0E3R4 A0A ... (34 characters truncated) ... V0E3R3 A0A1V0E3S6 A0A1V0E3P1 A0A1V0E3M7 A0A1V0E3P0 A0A1V0E3S0 A0A1V0E3T1 A0A1V0E3P6 A0A1V0E3Q2 A0A1V0E3Q8 A0A1V0E3R8 A0A1V0E3S7 A0A1V0E3N3 A0A1V0E3Q6', 'hgnc_id': '4936', 'hgnc_symbol': 'HLA-DOA'}] (Background on this error at: http://sqlalche.me/e/9h9h)

Error while populating kegg

While running kegg_manager.populate(), it throws error after 3hrs of loading