carnival-data / carnival Goto Github PK
View Code? Open in Web Editor NEWJVM property graph data unification framework
Home Page: https://carnival-data.github.io/carnival/
License: GNU General Public License v3.0
JVM property graph data unification framework
Home Page: https://carnival-data.github.io/carnival/
License: GNU General Public License v3.0
See carnival-core/src/test/groovy/carnival/core/util/MethodsHolderSpec.groovy. Without this change, we cannot extend GraphMethods classes add new methods or functionality, which could be useful.
For example, there could be an abstract GraphMethods class that contains a set of standardized methods to interact with a resource and also has an abstract method getServerUrl() that points to that resource. Concrete sub-classes could get the server URL from a configuration utility without having to re-implement the GraphMethod classes.
It would be nice to be able to inherit from GraphMethods classes.
MappedDataTable in the base dataAdd(Map) method formats dates as per the default SqlUtils date format, which is not the same as the default SQLServer date format. Add an option to specify the data format to use so that the date strings in MappedDataTable match the default format of SQLServer enabling easy data upload via standard SQLServer clients.
The core functionality has been replaced by vine and graph. Not all of the functionality has been migrated.... Tag the repository before removing the classes, so they can be easily round in the future.
Clean up and finalize what is currently in the main branch as v1. Existing applications should be compatible with v1.
We will be doing a major refactor/reorg for v2.
When a reaper method is called, current behavior will create a reaper process vertex in the graph that has a label as per the reaper method, but does not contain any unique reference to the arguments that were used to call the reaper method. What this means is that the ensure() or optionallyRunSingletonProcess() methods are limited. Two calls to a reaper method with different arguments are viewed as the same call. So, the reaper method truly is a "singleton process" which is expected to be run only once.
optionallyRunSingletonProcess() methods can stay this way as they have "singleton" in the name. ensure() methods should be more flexible, taking the reaper method arguments into account so a reaper method can be called multiple times on different inputs.
There is probably deep design discussion to be had here.... Taking into account the reaper method and the arguments is nice, but it does not take into account the state of the graph. There is no guarantee that the same reaper method called with the same arguments at different times will have the same effect, which I suppose is the assumption here. Is that useful? Should we enable programmers to classify reaper methods and have different behavior of ensure() depending on the class? I do not know.
For now, making this change will at least bring us more in-line with vine behavior. There is no guarantee that a vine method, which queries a database, called with the same parameters at different times will have the same result. The database can change in that interval. Carnival vine caching makes the assumption that the programer is choosing to ignore any changes refreshing when they choose to do so by invalidating (deleting) cache files. Implementing a similar scheme at the reaper method level seems to make sense.
The Vine, CachingVine, and VineMethod classes/interfaces/traits are muddled.... Clean things up.
KeyType was important for Carnival v0 when there were complex data operations that consumed and produced data tables. With the introduction of the graph in Carnival v1, these operations are done in the graph. KeyType is no longer central and should be removed for simplicity's sake.
There is overlap here with secondaryIdFieldMap of GenericDataTable and MappedDataTable. Like KeyType, the secondaryIdFieldMap was important for data table operations. Because secondaryIdFieldMap depends on KeyType, it is not possible to remove KeyType without removing secondaryIdFieldMap.
Add validation to the setter methods to check that the property is valid for the provided Vertex.
The JsonVine model has proven to be a good one. Standardize Reaper methods so they follow the JsonVine pattern.
Probably will store argument hash in the vertex that is created when a reaper runs.
For applications that have their own configuration machinery, it would be nice not to require a separate configuration file for Carnival configuration. Figure out how to provide the config data to Carnival directly so no separate file is required.
Scope carnival configuration within a top level element, which would enable integration of a carnival configuration in an existing application.yaml file that includes other application configuration.
Update travis to build dynamic docs and move them to the gh-pages branch.
Remove references to PMBB wherever possible/practical.
Implement GenericDataTableVine* classes. Attempt to leverage inheritance/composition to share common code with MappedDataTableVine*.
Related to #20.
We can remove secondaryIdFieldMap prior to removing KeyType. Probably best to factor out into a distinct issue.
Harmonize VertexDefTrait and EdgeDefTrait.
// having to call instance() is potentially non-intuitive
Pmbb.VX.PATIENT.instance().withProperties(Core.PX.NAME, 'Hayden').vertex(graph, g)
// doesn't matter
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').create(graph, g)
// existing 0 or 1
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').getOrCreate(graph, g)
// existing 1
Pmbb.VX.PATIENT.withProperties(Core.PX.NAME, 'Hayden').get(graph, g)
// maybe add find() that will return all pre-existing
// exists() returns true/false
// just have get() return the GraphTraversal tryNext() thing???
// get returns vertex or null?
// get returns an optional like thing?
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.relate(pV, encV, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.instance().withProperties(Core.PX.THING, 'ada').from(pV).to(encV).create(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).create(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).getOrCreate(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').from(pV).to(encV).get(graph, g)
Pmbb.EX.PARTICIPATED_IN_ENCOUNTER.withProperties(Core.PX.THING, 'ada').create(pV, encV, graph, g)
Core.PX.NAME.of(pV).isPresent()
Core.PX.NAME.valueOf(pV)
Currently the Carnival Gradle plugin is about building an Carnival Micronaut application. It would be helpful to have a plugin that just supplies the Carnival dependencies for library building.
It is currently the default in CachingVine.com. I'm not sure much thought went into it and it is seeming more annoying than useful... Change it?
Add extensions to Tinkerpop classes GraphTraversal and Graph to work with VertexDef, EdgeDef, and PropertyDef.
The mechanics of ConstrainedPropertyDefTrait are causing problems. It turns out that adding ConstrainedPropertyDefTrait to a PropertyDefTrait using the 'as' keyword is super-neat, but also causes problems. See implementing traits at runtime. Keeping the original PropertyDefTrait object, which will be an Enum normally, is desirable.
Having PropertyDefTrait and ConstrainedPropertyDefTrait seemed cleaner, but since it is causing issues and is not really required, will move the functionality of ConstrainedPropertyDefTrait to PropertyDefTrait and call it a day.
Travis tests are failing.
The method:
Map execute(...)
is a pain. The result generally does not mean much and is just boilerplate. Add a setResult() method that can be used inside execute() to set a result. Change signature to void execute(...).
APOC is not working. Since the upgrade of Groovy, Tinkerpop, and Neo4j libraries, I cannot find an APOC library that will work. Tested the following versions:
APOC has been carved out of the Carnival Dockerfile.
Find an APOC library that works or verify that one does not exist.
Add a convenience method to instantiate a GraphMethod object by class. Currently, the method(String name) does so by name only.
Reasoners are very much like Reapers, except they can have inputs from the graph, whereas Reapers are expected not to read from the graph, only write to it.
inputs() - returns the input vertices of the process
The following cleanup can happen regardless of a potential refactor.
IterativeCsvWriter does not seem to be used anywhere.
localBuildAndPublish was implemented because the default publishToMavelLocal was not re-publishing if the version was already present in the local repo, which was an issue during a dev cycle. @hjwilli suggested that -SNAPSHOT versions will overwrite.
When interacting with an API that return JSON, it's not desirable to require shoe-horning data into a DataTable. It might possible to have a trait that leverages Java Generics and Jackson to cache JSON serialized objects.
Publishing to Github Packages is useful, but users need to set up proper github authentication even if they are just attempting to pull from a public repo. Also, due to the inability of deleting or updating public packages on github, we cannot easily maintain SNAPSHOT versions.
Investigate alternative OSS package repos that would more easily support these features.
Central Repository:
https://central.sonatype.org/
https://central.sonatype.org/pages/ossrh-guide.html
https://central.sonatype.org/pages/requirements.html
Currently, the methods that case bare strings as field name and identifier field values coerce the case of the bare string. Field names become upper case, identifier field values become lower case. This is probably the correct default behavior as it is safer in terms of detecting data collisions.
However, FeatureReport, which extends MappedDataTable, is restricted by this behavior. FeatureReport is useful as a class to create data exports that are intended to be imported into other systems. Case coercion is not desirable in this case.
Add a "case-sensitive" flag to DataTable with a default of false. When true, field names and identifier field values are not case-coerced. This would make the data set more dangerous of data integrity, but more flexible.
It's not used and it has libraries that are constantly going stale and causing Github dependabot alerts.
This method displays progress information ('current' of 'total' chunks complete). Total = 0 is valid and should not throw an exception.
For negative total value, display a warning instead of throwing an exception.
Add code coverage to current groovy reporting.
Generate aggregated code coverage report across all subproject
Integrate code coverage with travis and with coveralls.
For reference, see:
https://lkrnac.net/blog/2016/10/aggregate-test-coverage-report/
https://gist.github.com/aalmiray/e6f54aa4b3803be0bcac
csMACnz/coveralls.net#88
I think it's mostly there. The Groovy code should look in only one place for the home setting. Why not just pick carnival.home. If it's an app, the plugin will set this value automatically. If it's framework development, build.gradle can set carnival.home from the environment.
Migrate Reaper, so it follows the Vine pattern.
Basic Call
myReaper.method('doSomething').arguments(Map).call()
Vertices representing all executed reaper method processes
myReaper.method('doSomething').processes()
Vertices representing all reaper method processes given arguments
myReaper.method('doSomething').arguments(Map).processes()
Vertices representing all executed reaper method processes
myReaper.method('doSomething').processes().vertices()
Outputs to any reaper process regardless of arguments
myReaper.method('doSomething').outputs()
Outputs to reaper processes with given arguments
myReaper.method('doSomething').arguments(Map).outputs()
find a free Jekyll theme that works.
Currently test suites can take a long time to run, some of that time is due to the recommended way to clear a neo4j database is to stop the server and delete the files.
Research the feasibility and potential speed benefits of using a neo4j ImpermanentDatabase for testing.
See https://neo4j.com/docs/stable/tutorials-java-unit-testing.html
Standardize the behavior so Reasoner and Reaper take the arguments into account when matching on a previous run.
We've moved past a single institution.
See VertexDef, where one is created dynamically with no propertyDefs or anything. There would have to be a repository of created DynamicVertexDefs so they can be looked up to keep track of any properties they have. Maybe just get rid of the concept altogether.
GPars is no longer being maintained and has some very old dependencies. Java concurrency has come a long way. Our uses of GPars could be migrated to core Java (fairly) easily.
There are currently two styles of vines: Json and MappedDataTable. Json vines have a single unified cache file which contains both the data and meta-data. MappedDataTable has two files, a Yaml meta file and a CSV data file. The cache related vine methods for these two styles of vines differ. Json cache is a bare File. MappedDataTable cache is a DataTableFiles object. Unify these two into a single CacheFiles (or similar) object that has standardized functionality.
With GPars removed, it should be possible to upgrade a number of Carnival dependencies. We have fallen behind on many fronts, including Groovy and Tinkerpop.
Migrate v1 Vine/Method to the v2 JsonVine style. The two will be merged. The class JsonVine will be renamed to Vine as the code at that level is generic. At the VineMethod level, there will be JsonVineMethod and TabularVineMethod. This will be a multi-step migration. Should probably be a milestone rather than a single issue, but here we are.
Travis is migrating support for open source projects from travis-ci.org to travis-ci.com, and will be closing travis-ci.org at the end of the year.
https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom
Every time I compile any module in Windows, I see an error like this printed to STDOUT
Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
The performance of the application does not seem to be in any way effected by this message.
`
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.