gchq / gaffer-tools Goto Github PK
View Code? Open in Web Editor NEWgaffer-tools is deprecated. Use https://github.com/gchq/gafferpy instead
License: Apache License 2.0
gaffer-tools is deprecated. Use https://github.com/gchq/gafferpy instead
License: Apache License 2.0
Travis should run some code analysis checks to determine whether the Python shell is ready for release, e.g. pylint/PEP8.
On the 'schema' tab in the UI builder it would be good to have a 'save to file' button for each of the 'data schema', 'data types' and 'store types'
The user should be able to specify the path to the file
And similar for uploading a local file.
These modules are currently in the main Gaffer repo and should be moved to this repository.
We should use something like selenium to verify a basic query in the UI works. This can then be added to the travis ci scripts.
The UI needs updating due to changes in Gaffer 0.7.4.
Since Gaffer 0.7.4 the DirectedType enum and IncomingOutgoingType enum need updating to be EITHER instead of BOTH.
See gchq/Gaffer#752.
Update the release process for the gaffer-tools project by publishing artefacts using Travis.
In your README, you have:
start = '1'; end = '3'; g.V(start).hasLabel('id').store('v').repeat(bothE().where(without('e')).store('e').inV().hasLabel('id').where(without('v'))).until{it.get().id() == end || it.loops() == 6}.path().filter{it.get().last().id() == end}
This can be written without lambdas and bit cleaner as:
g.V('1').hasLabel('id').
repeat(bothE().otherV().hasLabel('id').simplePath()).
until(hasId('3').or().loops().is(6)).
hasId('3').path()
Things to note:
simplePath()
will ensure you don't go back to an already seen edge/vertex.until()
.end
, then simply hasId(end)
.bothE().otherV()
goes in both directions. If you don't care about the edges in your final path data structure, just use both()
....path().by('name')
and that would return the path where instead of the vertices, you would have name
properties from those vertices. Easier for the human to read the result. http://tinkerpop.apache.org/docs/current/reference/#path-stepDevelop a method of generating large amounts of random elements for testing purposes.
Also worth trying to remove the materialize css now that angular 2 material is more complete.
Currently the tinkerpop implementation connects directly to Gaffer/Accumulo. It would be useful to be able to connect via the Gaffer REST API.
Gaffer 0.4.6 has just been released.
Currently the last part of the walkthrough doesn't work.
This is due to the bug in Gaffer gh-642
error message is:
com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: org.apache.catalina.connector.CoyoteInputStream@15ca496; line: 1, column: 427] (through reference chain: uk.gov.gchq.gaffer.operation.OperationChain["operations"]->Object[][0]->uk.gov.gchq.gaffer.operation.impl.get.GetElements["view"]->uk.gov.gchq.gaffer.data.elementdefinition.view.View["edges"]->java.util.LinkedHashMap["RoadUse"]->uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition["preAggregationFilterFunctions"]->java.util.ArrayList[0]->uk.gov.gchq.gaffer.function.context.ConsumerFunctionContext["selection"]->java.util.ArrayList[0])
This should use an in memory accumulo store.
See Gaffer 0.6.0.
The python shell needs updating due to changes in Gaffer 0.7.4.
Since Gaffer 0.7.4 the DirectedType enum and IncomingOutgoingType enum need updating to be EITHER instead of BOTH.
The EdgeSeed object now has the option for directed to be EITHER, this feature should also be added to the python shell.
See gchq/Gaffer#752.
Develop an Apache Slider application add-on package to allow Gaffer instances to be easily deployed onto YARN-managed clusters.
As discussed in the main Gaffer project (issue gh-888).
The 'jar-shader' tool should be run as part of the build to avoid jackson version issues when Gaffer instances are deployed onto CDH clusters.
This is linked to gchq/Gaffer#817
[ -------- Federated Gaffer REST API ---------- ]
| | |
[Gaffer REST API 1] [Gaffer REST API 2] [Gaffer REST API 3]
| | |
[Accumulo Store 1] [Accumulo Store 2] [HBase/Other Store]
Gaffer operations executed on this federated rest api would be delegated to multiple Gaffer REST APIs
Initially this should be implemented to assume that element groups are not shared across the different REST API. This means there is no need for summarising/merging the results across the REST APIs. The rest api would hold a map of element group to Gaffer url, which is used to delegate the operations to the REST API urls. This map of urls should be able to be updated without the rest api needing to be restarted.
Prior to delegating the operation chains to the URLs the requests need to be heavily validated - to check for things such as:
The Schema for each Store is compatible - e.g Element groups should not be shared across stores.
The functions used in an operation View are available for delegate store
Also, if an Operation View is provided it should be chopped up based on the Element groups in the View and the Element groups in the REST API schemas.
Deserialisation/Serialisation of json operation chains and results should be prevented. This will improve performance and mean the federated REST API will not require all custom Functions and Operations to be present on the classpath. E.g. if [Gaffer REST API 3] has a custom Filter function 'Function3' and a user executes an operation chain including Function3 then if the federated REST API tries to deserialise it into the java Function3 object it would fail if the Function3 wasn't on the class path.
The current python-shell project doesn't meet the python style guidelines and should be updated to meet PEP8 (https://www.python.org/dev/peps/pep-0008/)
#52 only runs AbstractStoreITs against a gaffer-slider instance.
We should also run AccumuloStoreITs.
Doing this requires gchq/Gaffer#686 to be fixed.
Using typescript type definitions properly rather than assigning everything to "any" should create more robust code. Warnings and errors should display if the variables are given values that don't match the interface they are given.
Changing the artifactId from gaffer-slider to slider has broken the 'quick start' scripts
Travis should run some checks to determine whether the Python shell is ready for release (unit tests, pylint/PEP8 checks etc)
Update group ID to uk.gov.gchq.gaffer to match the main project.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.