repmax / netmaker

A converter of NetML to json and csv formats that can easily be imported in network visualization and analysis software.

NetMaker

NetMaker is a project to develop tools and service that convert NetML files to a variety of file formats that can easily be imported by network visualization and analysis software.

NB: NetML is a network markup language which makes it easy to write down information as nodes with properties and links in a simple text file. Read more at https://repmax.github.io/netml/.

Status

The NetMaker python package has been developed to convert NetML files to json and cvs formats. It performs and detailed check of the NetML file to catch erroneous markup and can make use of a configuation file to expand abbreviated annotation.

Future development

First priority is to develop javascript based NetMaker web app, so there will be no need for installing and running the python package.

Instructions for NetMaker python package

1. Download files in root folder
2. Run:

from netmaker import *

dict_net = nml2json('input/sample_netml.txt','config_netml.yaml','sample.json')    
df_nodes, df_edges = json2csv(dict_net, 'sample_nodes.csv','sample_edges.csv','config_enrich.yaml')

Release notes for NetMaker v0.1

"""
===============================================
netmaker python package v0.1
python 3.5
_______________________________________________
NML2JSON()
dict_net = nml2json(netmlFile, configFile=None, writeJson=None)

Features:
- Returns a python dictionary from a NetML file
- Optionally save python dictionary as a json file.
- If NetML file contains errors, the function will print error information to console and continue checking rest of file for errors. No object is return if source nodes are erronerous. Missing target nodes for edges are printed to console and discarded.
- Referenced nodes can be declared both before and after being referenced.

Parameters:
- netmlFile: A NetML file with network information. Can be any text format: txt/md/nml.
- configFile: An optional YAML file to convert abbreviations to full words. See provided template config file. E.g. 'tw' property can be converted to 'twitter_handle'.
- writeJson: False/True
_______________________________________________
JSON2CSV()
df_nodes, df_edges = json2csv(dictObj, csvNodes=None, csvEdges=None, configFile=None):
    
Features:
- Returns two dataframes (nodes, edges) from a python dictionary.
- Optionally save nodes and edges as csv files
- The csv files are configured for Graph Commons import configuration. It entails that edge source and target are stated as the label of node and not and id.

Parameters:
- dictObj: A python dictionary with nodes and edges data.
- configFile: An optional YAML file to enhance data columnwise. E.g. '@maxmunnecke' in twitter handle column may be converted to 'https://twitter.com/maxmunnecke'.   
- csvNodes: Optionally write node data to csv file.    
- csvEdges: Optionally write edge data to csv file.
_______________________________________
EXAMPLES

from netmaker import *

dict_net = nml2json('sample_input_netml.txt','config_netml.yaml','sample.json')    
df_nodes, df_edges = json2csv(dict_net,'sample_nodes.csv','sample_edges.csv','config_enrich.yaml')

# ... or without any enrichment:

df_nodes, df_edges = json2csv(dict_net,'sample_nodes.csv','sample_edges.csv')

# ... if dict_net is not in memory but saved as a json file, it can be loaded with:

with open('sample.json', 'r') as f:
    dict_net = json.load(f)
_______________________________________
DEV NOTES

Tested with 'mixer/py35' (not tested on any other versions)

Problem: Json is a graph model where elements do not necessarily have the same properties. In a spreadsheet all elements needs a value for all properties.
Solution: 
    1) Make a dictionary for each element and make a dataframe from all those dictionaries. The Pandas will automatically fill in missing values.
    2) Rename columns as needed.
    3) Rearrange (reindex) dataframe columns. Any unknown column names will be add as empty (Useful for 'Reference' column).
===============================================""" 

Import json into Neo4j graph database

WITH  "file:///C:/dev/sample.json" AS url
CALL apoc.load.json(url) YIELD value as net
UNWIND net.nodes AS node
MERGE (n {uid:node.uid})
SET n = node
SET n.seed = true
WITH net, n, node.label AS newLabel
// Labels of nodes and relationships can not be set programmatically with Cypher
CALL apoc.create.addLabels(n, [newLabel]) YIELD node
REMOVE node.label
// Ending UNWIND loop by using DISTINCT
WITH distinct net
UNWIND net.edges AS edge
MATCH (s {uid:edge.source})
MATCH (t {uid:edge.target})
WITH s,t,edge.relType AS edgeLabel
CALL apoc.create.relationship(s,edgeLabel,{}, t) YIELD rel
RETURN count(*)

// ALTERNATIVE - via two operations

WITH  "file:///C:/dev/sample.json" AS url
CALL apoc.load.json(url) YIELD value as net
UNWIND net.nodes AS node
MERGE (n {uid:node.uid})
SET n = node
SET n.seed = true
WITH n, node.label AS newLabel
CALL apoc.create.addLabels(n, [newLabel]) YIELD node
REMOVE node.label
RETURN count(*)

WITH  "file:///C:/dev/sample.json" AS url
CALL apoc.load.json(url) YIELD value as net
UNWIND net.edges AS edge
MATCH (s {uid:edge.source})
MATCH (t {uid:edge.target})
WITH s,t,edge.relType AS edgeLabel
CALL apoc.create.relationship(s,edgeLabel,{}, t) YIELD rel
RETURN count(*)

netmaker's Issues

Is this a Triple notation?

in 2015 i got frustrated with the different RDF Triple notations and created

https://wiki.bitplan.com/index.php/SiDIF

instead with a java implementation at:
https://github.com/BITPlan/org.sidif.triplestore

in https://github.com/BITPlan/org.sidif.triplestore/tree/master/src/test/resources/sidif.canonical
you can see different triple notations in comparison with SiDIF e.g.

Json-LD example 5
http://www.w3.org/TR/json-ld/

{
  '@context':
  {
    'name': 'http://schema.org/name',
    'image': {
      '@id': 'http://schema.org/image',
      '@type': '@id'
    },
    'homepage': {
      '@id': 'http://schema.org/url',
      '@type': '@id'
    }
  },
  'name': 'Manu Sporny',
  'homepage': 'http://manu.sporny.org/',
  'image': 'http://manu.sporny.org/images/manu.png'
}

compared to SiDIF

" is comment of SiDIF
ModelLevel isA Context
"
 model level
 
" is comment of ModelLevel
HomePage isA Concept
name addsTo HomePage
image addsTo HomePage
image isA Property
homepage addsTo HomePage
id addsTo image
type addsTo image
InstanceLevel isA Context
" 
 Instance Level
" is comment of InstanceLevel
Manu_Sporny_Homepage isA HomePage
"Manu Sporny" is name of Manu_Sporny_Homepage
http://manu.sporny.org is homepage of Manu_Sporny_Homepage
http://manu.sporny.org/images/manu.png is image of Manu_Sporny_Homepage

There is a python implementation of SiDIF now
at
https://github.com/WolfgangFahl/py-sidif
and i might integrate NetML if it is compatible to these triple notations

Also https://linkml.io/ seems to have the bigger impact these days ...