hetio / hetionet Goto Github PK
View Code? Open in Web Editor NEWHetionet: an integrative network of disease
Home Page: https://neo4j.het.io
Hetionet: an integrative network of disease
Home Page: https://neo4j.het.io
@dongbohu emailed me regarding switching to Google Cloud Platform to host the public neo4j cloud instance at https://neo4j.het.io.
This sounds great to me, as I find the GCP interface is a bit better. And the pricing too right?
Let's coordinate via this issue.
Hello,
While I see that a TSV version is available it appears to be much older than the most up-to-date release. Can it be possibly updated?
Thank you,
Luca
Hi,
I'm interested in obtaining the entity alignment links between Hetionet and the original databases, such as DrugBank. Can you please provide some guidance on how to obtain these links?
Thank you very much for your help.
Could you please provide some queries like the aggregated query to the path from target to the gene? If you can share the python notebook with the queries, that would be very fantastic. Thanks!
Hi! Would it be possible to get a .dump version of Hetionet? I use multiple databases in neo4j and have found that this is easiest for loading new databases. By dump version, I mean one created via the neo4j-admin dump command which has file extension .dump (I can give more details if necessary)
Thanks!
Hi, I've been trying to reach the hetionet at neo4j platform (https://neo4j.het.io/browser/) but it doesn't load - I've tried several internet explorers, both on Linux and Windows. It just spins and shows nothing.
I noticed today that https://neo4j.het.io/browser/ ran out of memory while running the following query:
MATCH ()-[rel]->()
WITH type(rel) AS rel_type, collect(rel) AS rels
WITH rels[toInteger(rand() * size(rels))] AS rel
RETURN startNode(rel), rel, endNode(rel)
The error is as follows:
There is not enough memory to perform the current task. Please try increasing 'dbms.memory.heap.max_size' in the neo4j configuration (normally in 'conf/neo4j.conf' or, if you you are using Neo4j Desktop, found through the user interface) or if you are running an embedded installation increase the heap by using '-Xmx' command line flag, and then restart the database.
What's odd is that this query used to consistently not error. But today it's consistently caused a memory overflow. https://neo4j.het.io is currently running Neo4j 3.1.4 Community. See the docker image here.
Great work! I've used the Neo4j interface for Hetionet, but wanted to get a bit nicer figure, for which Cytoscape is my tool. However, there were some problems...
I've installed cyNeo4j app in Cytoscape and it works for simple queries. I want to extract paths from Hetionet database, and so I think I need to extract nodes and relationships separately, but Cytoscape just doesn't do anything, e.g.:
MATCH (node1:Gene { name: 'ESRRG' }), (node2:PharmacologicClass),
p = allShortestPaths((node1)-[*..3]-(node2))
WHERE node2.name STARTS WITH 'Vitamin'
RETURN nodes(p)
Have you tried anything similar? Do you have a tip how to move the paths from Hetionet to Cytoscape?
Any chance that you have a docker image of dhimmel/hetionet for the arm64?
We talked about this a while ago. I feel that edges that can only be in one direction should be explicitly shown to be directional, like compound -> side effect
:
Maybe there are database reasons why the direction
property should be left "neutral"/undirected for these types of edges, but at least for the front end, I don't see why they shouldn't be shown as arrows.
Currently, the Neo4j Hetionet Browser is hosted on a DigitalOcean Droplet as discussed here. Unfortunately however DigitalOcean's sponsorship credits have expired and they have not returned my communications about continuing to sponsor the project. Therefore, now makes sense to transfer hosting to the Greene Lab infrastructure where hopefully we can find a more sustainable setup.
CCing @dongbohu who manages the Greene Lab servers. The Docker and instance information is available in hetnet/neo4j/docker
. The instance needs not more than 20 GB disk space. Currently, we have 2 GB of RAM, but 4 GB would potentially help alleviate some of the crashes we periodically experience.
Some goals:
Any easy way to convert permutation graph of hetnet to networkx
Hello,
I have a list of genes and I want to print their immediate connection to the following node labels: Compound, BiologicalProcess, and Molecular Function. I want precisely three nodes for each relation from every gene in the array.
The following command is working for two OPTIONAL MATCH conditions. I have ran the following query to get the Compound and BiologicalProcess relations in NEO4J browser:
UNWIND ["ACVR1B","ACVRL1","ADCY6","AQP2","AQP6","ARF3","C12orf10","C1QL4","CCDC184","CNTN1","COX14","DNAJC22","EIF4B","ENDOU","ESPL1","FAIM2","GALNT6","KIF21A","LMBR1L","MFSD5","NPFF","PCED1B","PDZRN4","PFDN5","POU6F1","PRPH","PRR13","RACGAP1","RAPGEF3","RND1","RPAP3","SLC4A8","SPRYD3","TENC1","TMBIM6","TROAP","TUBA1A","TUBA1B","TUBA1C"] as text
MATCH (g:Gene) WHERE g.name = text
OPTIONAL MATCH (g)--(c:Compound)
WITH g,collect(c.name)[..3] as comp
OPTIONAL MATCH (g)--(b:BiologicalProcess)
RETURN g.name,comp,collect(b.name)[..3] as bio
But now I want to get all three relations with the following script:
UNWIND ["ACVR1B","ACVRL1","ADCY6","AQP2","AQP6","ARF3","C12orf10","C1QL4","CCDC184","CNTN1","COX14","DNAJC22","EIF4B","ENDOU","ESPL1","FAIM2","GALNT6","KIF21A","LMBR1L","MFSD5","NPFF","PCED1B","PDZRN4","PFDN5","POU6F1","PRPH","PRR13","RACGAP1","RAPGEF3","RND1","RPAP3","SLC4A8","SPRYD3","TENC1","TMBIM6","TROAP","TUBA1A","TUBA1B","TUBA1C"] as text
MATCH (g:Gene) WHERE g.name = text
OPTIONAL MATCH (g)--(c:Compound)
WITH g,collect(c.name)[..3] as comp
OPTIONAL MATCH (g)--(b:BiologicalProcess)
WITH g, collect(b.name)[..3] as bio
OPTIONAL MATCH (g)--(m:MolecularFunction)
RETURN g.name,comp,bio,collect(m.name)[..3] as mol
When I run this, the following error occurs:
Neo.ClientError.Statement.SyntaxError: Variable `comp` not defined (line 8, column 15 (offset: 586))
"RETURN g.name,comp,bio,collect(m.name)[..3] as mol"
Anyone know a more efficient way of dealing with this problem? Or understand why the comp variable is suddenly not defined after a third optional match statement?
Thank you in advance!
Hi, I am a postgraduate studying in the domain adaptation of pre-trained language models. I've been following your work in the realm of biomedical data integration.
I was particularly intrigued by your 2017 paper titled "Systematic Integration of Biomedical Knowledge Prioritizes Drugs for Repurposing." In it, you mentioned that 709 of the 1206 metapaths exhibited a statistically significant AUROC at a false discovery rate cutoff of 5%. However, while trying to replicate some of the results and delve deeper into the open-source data, I was unable to locate these 709 metapaths. Would it be possible for you to provide the specific metapaths and their instance paths? I am keen on further exploring these paths and your assistance would be of great help as I continue my journey in the biomedical domain.
I encountered challenges while trying to load Hetionet data into my updated MacBook's Neo4j version 5.13. The existing Neo4j dumps were no longer compatible, and directly importing the data in JSON format was too time-consuming, taking an estimated 10+ hours.
To address this, I've written a script that efficiently converts JSON data to CSV format without any loss in node, edge, or property value information. The JSON-to-CSV conversion takes approximately 30 seconds, while uploading the CSV to Neo4j takes around 40 seconds.
I've organized each node and edge type into its own respective CSV file and accompanying Cypher script. I believe this will make it easier for people to understand and work with the data.
If this sounds useful, I'd be open to integrating these changes into the main branch. Let me know your thoughts.
You can find the revised code at:
https://github.com/nickzren/hetionet/tree/csv
When trying to connect to: https://neo4j.het.io/browser/
Which is new, I've been querying it for several days now without problems. Any idea why it suddenly needs a username/password?
We are graduate student researchers from Yonsei University who are fascinated by your team’s work, Hetionet.
We are currently working on knowledge graph in biomedical domain and we found your work of constructing an integrated network containing data from different sources very inspiring. Thanks for the great work.
While we were studying Hetionet’s github(https://github.com/hetio/hetionet), we came up with several questions regarding the provided data set. We would be very grateful if you can help us go through this.
These are the questions we came up with:
What is the difference between Hetmat-v1.0-permutations and hetionet-v1.0-perm-1~5?
In the github, it was explained that 001 to 005 was constructed by project Rephito and network constructed afterwards were based on greenlab and hetmech. Would this be the only difference between the two?
The following image shows loaded form of npz file for Hetnet006 CcSE.sparse.npz file.
In the data, what is the difference between row and column? Is it safe to understand that row contains data of first entity type(this case, Compound) and column contains the second entity type(this case, Side effect)?
Also, does the true & false indicates that pair(for example, [Compound 0 & Side Effect 2] information is contained in that particular hetnet set(this case, 006)?
Compound 0 & Side Effect 2 is True in Hetnet 006 but false in hetnet 41. Is this because the pair’s information is only contained in Hetnet 006?
Thank you for reading this bunch of questions. All these question came up in the process of studying Hetionet(Hemat) with fascination and deep interest.
Thanks.
Starting up a container for the Docker image created in 3196731, here are the logs:
Downloading and extracting database
Downloading and extracting guides
WARNING: dbms.memory.heap.initial_size will require a unit suffix in a
future version of Neo4j. Please add a unit suffix to your
configuration. Example:
dbms.memory.heap.initial_size=512m
^
WARNING: dbms.memory.heap.max_size will require a unit suffix in a
future version of Neo4j. Please add a unit suffix to your
configuration. Example:
dbms.memory.heap.max_size=512m
^
Starting Neo4j.
2016-12-24 04:29:54.721+0000 WARN WARNING! Deprecated configuration options used. See manual for details
2016-12-24 04:29:54.722+0000 WARN unsupported.dbms.executiontime_limit.enabled is not supported anymore. Set dbms.transaction.timeout settings to some positive value to enable execution guard and set transaction timeout.
2016-12-24 04:29:54.722+0000 WARN unsupported.dbms.executiontime_limit.time has been replaced with dbms.transaction.timeout.
2016-12-24 04:29:55.402+0000 INFO Starting...
2016-12-24 04:29:56.734+0000 INFO Bolt enabled on 0.0.0.0:7687.
2016-12-24 04:30:05.479+0000 INFO Started.
2016-12-24 04:30:05.846+0000 INFO Mounted unmanaged extension [extension.web] at [/guides]
2016-12-24 04:30:07.069+0000 WARN The following warnings have been detected with resource and/or provider classes:
WARNING: A sub-resource method, public javax.ws.rs.core.Response extension.web.StaticWebResource.index() throws java.io.IOException, with URI template, "/", is treated as a resource method
2016-12-24 04:30:08.103+0000 INFO Remote interface available at http://localhost:7474/
So there looks to be several updates needed:
dbms.transaction.timeout
instead of unsupported.dbms.executiontime_limit.enabled
and unsupported.dbms.executiontime_limit.time
.dbms.memory.heap.initial_size
and dbms.memory.heap.max_size
I was able to get some paths and they are accompanied by two metrics PDP and percent_of_DWPC. Could you help me understand what those are?
Hello HetioNet team,
First of all, thank you for the hard work behind the hetionet
graph.
I've seen multiple open issues around accessing Neo4j, but I wasn't certain if I should comment under one of them or open a new one, so I chose to open a new one (apologies if it's a duplicate issue)
I am trying to use the Neo4j explorer to take a look at the HetioNet graph, however I keep getting a connection refused
error.
Are you aware of it being down? Is #33 related to it?
Thank you in advance for any insights on this.
With kind regards,
Dimitrios
Hi,
I am new to this area and would greatly appreciate any insight on the following:
Many thanks.
Best regards,
V
Hey!
Was toying around with a few metapaths are realized that some result in a drug "upregulating" a disease.
Does this indicate that the drug could make the symptoms of a disease worse in this specific pathway?
Hi,
I downloaded Hetionet onto my local machine and could browse it. However, I can't seem to locate my existing GraphDB which I created beforehand. How do I retrieve it?
Thanks
Thanks for building the great resource. The Neo4J browser, however, has not been accessible for the last couple of days.
Thanks in advance for looking into it.
Hi,
I sent an email to Daniel asking three broad questions about Hetionet that I was interested in for my research. He asked me to post this as a Github issue, so I'm writing them here.
What kind of queries and graph computations do you run on Hetionet? Did you (or other people) run any clustering algorithms on Hetionet? As another example, I see from Daniel's phd defense presentation that he ran some ML algorithms too. What were these?
What kind of software do you use to run these queries? From Daniel's slides, I see Neo4j for example as a graph software.
Were there any features that you think was missing from the software that you were using, or things that were difficult to do? One specific feature we had in mind is since your graph is quite heterogenous, I was curious if you extract simpler, more homogenous graphs, out of hetionet, say of only gene gene interactions, then store it as a separate graph, and do computations on it? If so, how many different "homogeneous graphs" inside hetionet do you think you have extracted so far?
Thanks!
Semih
Is the hetionet database down?
To produce the error: Run the Hetionet Docker locally:
docker run \
--publish=7474:7474 \
--publish=7687:7687 \
--volume=$HOME/neo4j/hetionet-data:/data \
--volume=$HOME/neo4j/hetionet-logs:/var/lib/neo4j/logs \
--env=NEO4J_dbms_memory_pagecache_size=1g \
--env=NEO4J_dbms_memory_heap_maxSize=512m \
dhimmel/hetionet:hetionet-v1.0_neo4j-3.1.0
Navigate to http://localhost:7474/browser/ and run MATCH p=()-[*..3]->() RETURN p
(an expansive query that will likely never complete). Here is the Docker log:
Not retrieving database as it already exists
Not retrieving guides as they already exist
Starting Neo4j.
2017-01-04 19:06:16.469+0000 INFO No SSL certificate found, generating a self-signed certificate..
2017-01-04 19:06:17.077+0000 INFO Starting...
2017-01-04 19:06:17.446+0000 INFO Bolt enabled on 0.0.0.0:7687.
2017-01-04 19:06:21.475+0000 INFO Started.
2017-01-04 19:06:21.625+0000 INFO Mounted unmanaged extension [extension.web] at [/guides]
2017-01-04 19:06:21.987+0000 WARN The following warnings have been detected with resource and/or provider classes:
WARNING: A sub-resource method, public javax.ws.rs.core.Response extension.web.StaticWebResource.index() throws java.io.IOException, with URI template, "/", is treated as a resource method
2017-01-04 19:06:22.268+0000 INFO Remote interface available at http://localhost:7474/
2017-01-04 19:07:01.093+0000 ERROR Failed to generate JSON output.
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:636)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:607)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
... 10 more
2017-01-04 19:07:01.093+0000 ERROR Failed to generate JSON output.
is the start of the error. The Neo4j browser dies:
If you refresh the page, the browser comes back. I never experienced this issue before 3.1, but have not confirmed it does not exist with earlier Neo4j versions. In any case, a better error message and sandboxed failure in the browser (only the frame dying) would be nice.
Do any of the relations in Hetionet imply the existence of another relation (that isn't the reverse relation)? For instance, there is a Disease - upregulates - Gene edge and a Disease - associates - Gene edge. In this case, does the Disease - upregulates - Gene edge imply that there should also be a Disease - associates - Gene edge?
From the documentation, I see that the data sources for the associates and upregulates edges are different, but from a biological perspective, if a disease upregulates a gene, does that mean it's also classified as being associated with that gene?
I noticed https://neo4j.het.io/browser/ is currently not working with ERR_CONNECTION_REFUSED
. I ran docker logs hetionet-container
to see what happened. Below is the tail of the docker logs:
2018-06-12 13:51:13.428+0000 INFO Transaction with id 14886 has been automatically rolled back due to transaction timeout.
2018-06-12 14:12:37.082+0000 WARN Illegal character 0x1B in state=START for buffer HeapByteBuffer@9be262e8[p=1,l=114,c=8192,r=113]={\x1b<<<$J\xB3\x00U\xBa\x05\xD4\xD4\x1f[U\xE8\xCf\xF0\xAb_...\xE7\x8b{\xE1\xE7\x8b{\xE5\xE7\x8b{\xE5\xE7\x04\x00>>>0 (Windows NT 6.1...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2018-06-12 14:12:37.082+0000 WARN badMessage: 400 Illegal character 0x1B for HttpChannelOverHttp@4fd9e509{r=0,c=false,a=IDLE,uri=-}
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14896 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14887 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14888 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14889 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14890 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14891 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.427+0000 INFO Transaction with id 14892 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.428+0000 INFO Transaction with id 14893 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.428+0000 INFO Transaction with id 14894 has been automatically rolled back due to transaction timeout.
2018-06-12 16:46:43.428+0000 INFO Transaction with id 14895 has been automatically rolled back due to transaction timeout.
2018-06-12 18:55:05.557+0000 WARN Illegal character 0x1C in state=START for buffer HeapByteBuffer@ab89f633[p=1,l=114,c=8192,r=113]={\x1c<<<o\x18af\xD1\xF48\t\x17 [8:\x15x\xBc\x8a...\xC1\x9c\xAe\x9f\xC1\x9c\xAe\x9b\xC1\x9c\xAe\x9b\xC1\x04\x00>>>leWebKit/537.36 (...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2018-06-12 18:55:05.557+0000 WARN badMessage: 400 Illegal character 0x1C for HttpChannelOverHttp@4304652d{r=0,c=false,a=IDLE,uri=-}
2018-06-13 01:32:29.278+0000 WARN Illegal character 0x5 in state=START for buffer HeapByteBuffer@d297ec0b[p=1,l=3,c=8192,r=2]={\x05<<<\x01\x00>>> / HTTP/1.0\r\n\r\ns/...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2018-06-13 01:32:29.279+0000 WARN badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@2bfbccf0{r=0,c=false,a=IDLE,uri=-}
2018-06-13 01:41:10.661+0000 WARN Illegal character 0x4 in state=START for buffer HeapByteBuffer@feae0e47[p=1,l=9,c=8192,r=8]={\x04<<<\x01\x00P.\xA1\t\x1f\x00>>>TP/1.0\r\nHost: 127...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2018-06-13 01:41:10.661+0000 WARN badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@a555b23{r=0,c=false,a=IDLE,uri=-}
2018-06-13 07:40:17.648+0000 WARN /browser/app-a5e172cd9d95a990973f.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:572)
at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:955)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:533)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:216)
... 33 more
2018-06-13 07:40:17.649+0000 WARN /browser/app-a5e172cd9d95a990973f.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:572)
at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:955)
at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:533)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:216)
... 33 more
2018-06-13 11:06:13.516+0000 WARN Exception thrown when submitting maintenance task unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
at java.util.concurrent.ForkJoinPool.signalWork(ForkJoinPool.java:1634)
at java.util.concurrent.ForkJoinPool.externalPush(ForkJoinPool.java:2414)
at java.util.concurrent.ForkJoinPool.execute(ForkJoinPool.java:2648)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:967)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:807)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1946)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:54)
at org.neo4j.cypher.internal.compiler.v3_2.LFUCache.getOrElseUpdate(LFUCache.scala:30)
at org.neo4j.cypher.internal.ExecutionEngine.preParseQuery(ExecutionEngine.scala:126)
at org.neo4j.cypher.internal.ExecutionEngine.parseQuery(ExecutionEngine.scala:112)
at org.neo4j.cypher.internal.ExecutionEngine.isPeriodicCommit(ExecutionEngine.scala:204)
at org.neo4j.cypher.internal.javacompat.ExecutionEngine.isPeriodicCommit(ExecutionEngine.java:86)
at org.neo4j.bolt.v1.runtime.TransactionStateMachineSPI.isPeriodicCommit(TransactionStateMachineSPI.java:120)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$1.run(TransactionStateMachine.java:188)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine.run(TransactionStateMachine.java:74)
at org.neo4j.bolt.v1.runtime.BoltStateMachine$State$2.run(BoltStateMachine.java:395)
at org.neo4j.bolt.v1.runtime.BoltStateMachine.run(BoltStateMachine.java:194)
at org.neo4j.bolt.v1.messaging.BoltMessageRouter.lambda$onRun$3(BoltMessageRouter.java:80)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.execute(RunnableBoltWorker.java:130)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:91)
at java.lang.Thread.run(Thread.java:748)
2018-06-13 11:06:14.219+0000 WARN Exception thrown when submitting maintenance task unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
at java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
at java.util.concurrent.ForkJoinPool.signalWork(ForkJoinPool.java:1634)
at java.util.concurrent.ForkJoinPool.externalPush(ForkJoinPool.java:2414)
at java.util.concurrent.ForkJoinPool.execute(ForkJoinPool.java:2648)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:967)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.afterRead(BoundedLocalCache.java:807)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1946)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:54)
at org.neo4j.cypher.internal.compiler.v3_2.LFUCache.getOrElseUpdate(LFUCache.scala:30)
at org.neo4j.cypher.internal.compiler.v3_2.LFUCache.apply(LFUCache.scala:48)
at org.neo4j.cypher.internal.compiler.v3_2.MonitoringCacheAccessor.getOrElseUpdate(CacheAccessor.scala:55)
at org.neo4j.cypher.internal.compiler.v3_2.QueryCache$$anonfun$getOrElseUpdate$1.apply(CacheAccessor.scala:34)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:162)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at org.neo4j.cypher.internal.compiler.v3_2.QueryCache.getOrElseUpdate(CacheAccessor.scala:46)
at org.neo4j.cypher.internal.ExecutionEngine.liftedTree1$1(ExecutionEngine.scala:170)
at org.neo4j.cypher.internal.ExecutionEngine.planQuery(ExecutionEngine.scala:156)
at org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:106)
at org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:61)
at org.neo4j.bolt.v1.runtime.TransactionStateMachineSPI$1.start(TransactionStateMachineSPI.java:144)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$1.run(TransactionStateMachine.java:202)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine.run(TransactionStateMachine.java:74)
at org.neo4j.bolt.v1.runtime.BoltStateMachine$State$2.run(BoltStateMachine.java:395)
at org.neo4j.bolt.v1.runtime.BoltStateMachine.run(BoltStateMachine.java:194)
at org.neo4j.bolt.v1.messaging.BoltMessageRouter.lambda$onRun$3(BoltMessageRouter.java:80)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.execute(RunnableBoltWorker.java:130)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:91)
at java.lang.Thread.run(Thread.java:748)
2018-06-13 11:07:06.068+0000 WARN unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:447)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:598)
at java.lang.Thread.run(Thread.java:748)
2018-06-13 11:07:06.170+0000 WARN unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
Exception in thread "qtp1340311417-29834" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:447)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:620)
at java.lang.Thread.run(Thread.java:748)
at java.lang.Thread.start(Thread.java:717)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:447)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:598)
at java.lang.Thread.run(Thread.java:748)
2018-06-13 11:07:06.173+0000 WARN Unexpected thread death: {} in {}
2018-06-13 11:07:36.537+0000 WARN unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:447)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:598)
at java.lang.Thread.run(Thread.java:748)
2018-06-13 11:07:36.547+0000 WARN Unexpected thread death: {} in {}
Exception in thread "qtp1340311417-30890" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:447)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:620)
at java.lang.Thread.run(Thread.java:748)
Killed
So it looks like Neo4j ran out of memory. We've had similar issues in the past with Neo4j 3.1: see neo4j/neo4j#9209. We have also had an issue where Neo4j runs out of memory but does not die as a process: see #5. Currently neo4j.het.io is running Neo4j 3.2.3:
So the two main questions for me are:
At the time of writing (UTC timestamp: 1637594788), i was not able to connect to http://neo4j.het.io/browser/ and got a time out error.
UTC timestamp: 1637594788
GMT: Monday, 22 November 2021 15:26:28
To convert the timestamp to your timezone time with https://www.epochconverter.com/
Thanks for your help in advance.
Hi! I am interested in the completed hetnet and I saw this from the import.ipynb
:
Several files were not uploaded to dhimmel/integrate due to filesize. These files are copied over locally.
Is it possible to download those local files?
Thanks!
Hi Daniel,
My name is Nilesh Dharajiya, MD, and I am a molecular pathologist by training. I came across het.io recently and am very impressed by it. I have been playing with Neo4j since last 3 years and like the prospect of graph db in medical science. I saw that Hetionet combines data from 29 databases, which does not include GWAS database.
However, your abstract "Heterogeneous network link prediction prioritizes disease-associated genes” shows 698 associations extracted from GWAS catalog.
Have you every tried margin gwas data into the het.io graph db? I am specially looking for detecting relation ships between clinical symptoms to disease to genes and using calculating polygenic risk scores from gwas data so that with a symptoms, I can go all the way to genes involved in the pathogenesis. Is this possible? Do you know anyone who has done this?
Looking forward to hearing from you.
Best regards,
Nilesh
Hi! Not sure if this is the right place to ask this, but here goes:
I've read through https://think-lab.github.io/p/rephetio/. Hetionet seems extremely impressive and useful. I need something very similar, if not identical, but with the emphasis on diagnosing rare diseases, and I would also strongly prefer to have the ability of automatically updating or adding new data to my database, e.g. to include some new GWAS findings, tailoring specificity of the disease terms to my needs, or maybe adding other node types like genetic variants. Hence, I'm wondering, how hard would it be to reproduce something like hetionet from scratch, possibly in litttle steps (for a group of a couple of people)? I see that https://think-lab.github.io/p/rephetio/#methods has some detailed information about what steps where taken, and also quite a number of links to files hosted on Zenodo. Would You say that all information is there or should I also look elsewhere? Was the main "mode of operation" to download text files from the internet, parse/preprocess/unify/join the data using python scripts, and then inject into Neo4j?
Apologies for a vague question. Many thanks for any suggestions! :)
I have gone to the link http://neo4j.het.io/browser/ a number of times over the past couple of days, and it appears that a password is required to log in. I reached out to Daniel Himmelstein and he mentioned that he did not believe a password was necessary, so I am creating this ticket.
Hi Daniel,
I am interested in using het.io as a tool to decide which small molecule drugs (and combinations of those drugs) could be promising treatments for acute illnesses.
I've tried using the Node Search function to see which compounds are connected with these diseases. Specifically, I searched for ARDS (Acute Respiratory Distress Syndrome), Ischaemic stroke, and Sepsis, but they only showed up as a side effects, not diseases.
Am I doing something wrong with the search, or is this just the way the database is curated?
Forgive me if this is an obvious question, I'm new to network pharmacology and these types of databases!
Thanks,
Mike
After a closer look, I'm having issues with the data source used for pathways.
Pathway::PC7_2008 BMAL1:CLOCK,NPAS2 activates circadian gene expression Pathway
Should correspond to the reactome pathway, https://reactome.org/content/detail/R-HSA-1368108, but it's not clear what this identifier is. I might guess "Pathway Commons 7"
Later, wikipathways identifiers (plus revisions) are used for other pathways
Pathway::WP516_r71358 Hypertrophy Model Pathway
It's nice to have the exact revision _r71358
, but this isn't what's necessary to resolve this pathway and merge with other resources.
I'm also not sure what the actionable item is for this. I don't think you would update the source data, would you? Are there plans for a Hetionet v2.0 that will include some of the other new updates?
In 23f6117, we began using LFS to store large files. While this commit uses LFS, the history still contains non-LFS files. Therefore, we can use the BFG Repo Cleaner to create a history where all files use LFS. We will keep the pre-LFS history available in branches, but not master
.
Hi,
I run the docker container as described in readme, but got the following error:
WARN [o.n.k.NeoStoreDataSource] Exception occurred while starting the datasource. Attempting to close things down. Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@595f4da5' failed to initialize. Please see the attached cause exception "/data/databases/graph.db/neostore.labelscanstore.db: Cannot map non-existing file".
org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@595f4da5' failed to initialize. Please see the attached cause exception "/data/databases/graph.db/neostore.labelscanstore.db: Cannot map non-existing file".
Is the database match with the version of neo4j?
Hi everyone,
I tried connecting to the NEO4J browser with hetionet from the het.io website (https://neo4j.het.io/browser/) but it seems to be down or not working. Does anyone know any workaround for this? Has the web address changed? The error reads 'The site cannot be reached'. I've tried connecting on different computers, and on different browsers too.
Thank you in advance!
Hey!
Is there a way for me to query the hetionet to get a sum of the path scores of all metapaths from a source node to a target node? I attached an image of how I can do this with the connectivity search GUI. However, I was wondering if there is a way for me to make many queries where I can get the path count sum, without having to manually enter it into the website each time.
In addition to the four formats with which you're distributing, would you consider supporting Biological Expression Language (BEL)?
If you're not familiar, BEL is another schema for storing causal, correlative, ontological, and associative relationships between biological entities. It's a good middle ground between the wild-west of RDF/Ontologies/OBO, where you can almost define anything you want, and having a strict/controlled vocabulary for expression. It has its own domain specific language, so it's a bit less accessible than JSON, XML, or TSV, but there is a growing number of applications that directly consume BEL.
I'm already writing a converter at pybel/pybel#406, so to make the conversion you'd only have to run the following few lines of code.
from pybel import to_bel_script_gz
from pybel.io.hetionet import get_hetionet
output_path = ...
graph = get_hetionet()
to_bel_script_gz(graph, output_path)
If you might be interested in this, let me know. I'd be happy to take feedback. I will also post an issue or two I've been having as I'm going through the JSON export
For example, if you think keeping the conversion script somewhere within your organization rather than in PyBEL is a better idea, then I'd also be happy to make it as a PR there.
Hello Daniel,
Thanks for your speedy reply on How do I run the Hetionet v1.0 docker container?. Per your suggestion I'm moving the conversation here and will update back to Stackoverflow (all credit to you).
Hetionet docker has a special run command,
sh ~/run-docker.sh
, which is executed after runningdocker pull dhimmel/hetionet
.
On digitalocean I pull the image and enter sh ~/run-docker.sh
then get this output:
The root directory looks like this:
If I run the image locally on kitematic I get this:
Running docker run --interactive --tty dhimmel/hetionet bash
to explore the image, per your suggestion, gives this output:
It turns out I have run hetionet eight times. docker ps -a
outputs this:
So I remove all the containers by running docker rm $(docker ps -a -q)
. However re-running docker run --interactive --tty dhimmel/hetionet bash
(or other memory consuming commands) result in the same "No space left on device" message. How do I reclaim this space if docker rm $(docker ps -a -q)
didn't do the job?
I'm going through metadata in the compound-binds-gene relationships, and taking a specific look at the actions
lists. In many examples, there are several actions, such as with drugbank:DB00502 binds ncbigene:1813. In the JSON GZ export, there are two actions listed: ['antagonist', 'inverse agonist']
. I made a query to the Neo4j instance to confirm this is also true there:
MATCH p=(s:Compound)-[r:BINDS_CbG]->(t:Gene)
WHERE s.identifier = 'DB00502' and t.identifier = 1813
RETURN p
LIMIT 25
However, on DrugBank I could only find the antagonist
label. Is it the case that the DrugBank source data that gets parsed and converted in Hetionet contains extra information that doesn't make it to the web page I linked? If so, do you have any idea on how they pick which of many gets displayed?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.