dataoneorg / slinky Goto Github PK
View Code? Open in Web Editor NEWSlinky, the DataONE Graph Store
License: Apache License 2.0
Slinky, the DataONE Graph Store
License: Apache License 2.0
I gave a demo of using rsdfsurveyor with Slinky on the Nov 4 2021 salmantics call. To do the demo I created deployment and service files for rsdfsurveyor-I'm wondering if there might be some utility in keeping them around for internal use. It's fairly lightweight: an nginx based container with the rdfsurveyor source in www/html
and a service file to expose it.
To process all data that comes into DataONE, we need to be able to get access to the metadata files in an efficient manner, and be notified when new updated revisions or changes to SystemMetadata are available. This is the same access problem from k8s for the MetDIG processor and the DataONE index processor, and can likely use the same solution. We have discussed making the metadata documents available from a known location on a read-only Ceph filesystem that is mounted into appropriate containers. A queue system would be notified when PID changes occurred that need to be processed by various subscribers, and then they would be able to access the data directly from the Ceph filesystem without undertaking a REST call to Metacat (and without making a cached copy).
The Related issue in Metacat is NCEAS/metacat#1436 for designing such a system.
[I'm filing this not because we're moving on to next steps already but just to file it and let people chime in with ideas]
We can always find ways to improve the metadata we have but most metadata are written once, possibly checked and tweaked by a moderation team, and the left fixed in stone. What if we could extend the ways Slinky already improves metadata (ie co-reference resolution, minting/finding party identifiers) beyond what we're doing now?
I got to thinking about this after one of our recent mobilization calls and a recent example got me here to writing this ticket. Take the metadata record at https://search.dataone.org/view/urn%3Auuid%3A84f4e415-53c3-55e9-bb6d-3ee34419595d. It's a JSON-LD record from NPDC. The abstract starts:
Data from Polarstern cruise PS94 in the Arctic in 2015 with chief scientist Ursula Schauer.
There's a few really key elements to this free text description that we could totally extract into linked data and make for a much richer landing page: (1) Polarstern (2) PS94 (3) Arctic (4) 2015, (5) Ursula Shauer (6) Ursula Shauer as a Chief Scientist and (7) the role of Chief Scientist.
Extracting and linking information like this would be a really nice enhancement for a lot of metadata records, but especially our science-on-schema ones which will tend to be more minimal. We might also think about how we preserve any enhancements in our Data Package exports.
Specific things we could build on top:
We should avoid using passwords for the default virtuoso accounts, for example dba:dba
and dav:dav
. We should be setting new passwords during deployment; see this documentation link for more information.
Saw this while testing earlier today
; slinky get doi:10.18739/A23F4KP3J
Traceback (most recent call last):
File "/usr/local/bin/slinky", line 33, in <module>
sys.exit(load_entry_point('d1lod', 'console_scripts', 'slinky')())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/bryce/src/slinky/d1lod/d1lod/cli.py", line 35, in get
model = client.get_model_for_dataset(id)
File "/Users/bryce/src/slinky/d1lod/d1lod/client.py", line 77, in get_model_for_dataset
return processor.process()
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml220_processor.py", line 87, in process
return super().process()
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 70, in process
self.process_publisher()
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 214, in process_publisher
publisher_subject = self.process_party(publisher)
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 129, in process_party
return self.process_organization(party)
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 381, in process_organization
self.process_user_id(party_subject, user_id)
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 228, in process_user_id
self.process_user_id_as_generic(party_subject, user_id)
File "/Users/bryce/src/slinky/d1lod/d1lod/processors/eml/eml_processor.py", line 326, in process_user_id_as_generic
directory = user_id.attrib["directory"]
AttributeError: 'str' object has no attribute 'attrib'
Right now deployment is handled through a make
file. It would be great if developers could use the same tooling across all kubernetes deployments, like helm
.
One hoop to jump through is ordering the deployment process; we want some pods to start after others have started. Helm doesn't have an official way to specify an ordering of deployments, but it looks like there's at least one workaround.
One way is to use chart hooks. Since we have six deployments, we'll have six helm charts. It's possible to run a chart hook before a helm chart is deployed. We're also able to set the hook priority with a hook-weight
. Since hooks with high priorities are deployed first, it can be used as an ordering mechanism. This unfortunately doesn't provide a means to tell the scheduler deployment to wait for the redis deployment (it only allows us to deploy redis first, and immediately the scheduler).
I think we might actually be able to attach an initContainer to the scheduler and worker that pings redis until it finally gets a response. Then the container will pass control to the slinky cli. Doing this, we should be able to start all deployments at once without using chart hooks.
Tasks
We have a few deployment cases for slinky; at the moment we can configure these using an environment. In #43 I added a --prod
flag to the cli which brings us further away from the pattern using environmental variables for this.
Instead, we should be able to configure the hosts in the kubernetes deployment file and the docker-compose definition.
This should involve
The work on https://github.com/DataONEorg/slinky/tree/feature_update_graph_pattern is far along and can be merged to develop
whenever. There will definitely be some cleanup to get the deployment back to a working state so we'll have to do that.
As of #3, @ThomasThelen has gotten Slinky running under minikube so the next thing to work on is to get a real deployment under k8s going so we can test all the odds and ends of getting that done (ie Docker Hub, ingress, etc).
What would be ideal would be hooking things up enough that I can, as a developer, make changes to services like the worker or scheduler, and see my changes reflected in the test cluster relatively quickly.
With the transition over to Virtuoso, we no longer need Apache + Tomcat to expose sesame and the work bench (since they're no longer used). The Apache+Tomcat container is in the web/
folder, while the web stuff is in www/
.
Transform the geolink:Dataset object to, at a high level, conform to the schema,org model (schema:Dataset). For this task, ignore the representation of individuals. This representation should include the following properties, which can then be expanded on.
Some of these aren’t super useful to know through the graph store. Ie how is DataONE going to leverage license information in the graph store? It probably won’t. The reason these are included is to comply with the SOSO recommendation that they be included. They’re also easy fields to include and “shouldn’t take much time”.
Current processing code depends on python 2.x. Need to upgrade to support 3.x.
We have a folder of legacy code and a few unit tests that are tied down to it (test_metadata, test_graph, conftest). We probably want to decouple the legacy code from the tests and remove it from the code base
The Virtuoso service should be set as a ClusterIP so that the ingress controller can manage traffic rather than directly exposing the service to the outside world.
Note: This is using language from the codebase in PR #54
With the ongoing effort around developing a solid deployment, this is a use case that can help drive the solution's direction. In production, Slinky currently uses Virtuoso as the backing graph store. We have unit tests that suggest Blazegraph should also work in a production environment; we should include a way to run Blazegraph in production.
Right now, we're hardcoding which deployment we use in production; each time slinky is deployed, it uses the Virtuoso adapter. To let users choose their graph store, we can support a new flag in the cli to specify which graph store is being used. For example, if the scheduler is going to use Blazegraph-it would be started with slinky schedule --store blazegraph
. We can use this to instantiate the appropriate connector/store class. This solves the is of how to specify the graph, but we still need to be able to specify where.
We use the GRAPH_HOST
environmental variable to specify the network location of virtuoso. I propose that we change this to VIRTUOSO_HOST
, matching the style of BLAZEGRAPH_HOST. If we didn't need to specify the location of all graphs while the unit tests run, we could keep a single environmental variable.
If a user wants to use Blazegraph they must also fill out the appropriate environmental variables. In this case of a user deploying with blazegraph, BLAZEGRAPH_HOST
and BLAZEGRAPH_PORT
must be set. Likewise, if Virtuoso is used the user specifies the store with slinky schedule --store virtuoso
and must fill out the VIRTUOSO_HOST
& VIRTUOSO_PORT
environmental variables.
We had a good discussion on our last salmantics call [1] about applications. I'm filing our notes here for posterity. Please add any ideas you have to this issue..
[1] https://docs.google.com/document/d/1Sc3m526EDkT53s0uJQarV-zAVUVft7ObIii2wazHJrQ/edit
This is to determine how the schema:Dataset
, dcat:Dataset
, and related classes inter-relate, and how they will be tied to People and other critical entities.
On our last weekly salmantics call [1], we agreed we should work towards closing out some of the graph pattern issues and PRs in order to get a solid first pattern down.
We have an open PR: #21 which includes a discussion. The high points include:
I'm going put together a hackpad with a handful of datasets that'll help us find the gaps and issues.
[1] https://docs.google.com/document/d/1Sc3m526EDkT53s0uJQarV-zAVUVft7ObIii2wazHJrQ/edit
At some point we're going to be using ceph storage which means deploying slinky locally will be very involved. There's also the architecture of the DataONE k8 cluster that enforces restrictions on who can create persistent volumes. Because of this, the persistent volume definitions should be moved out of this repository and into the DataONE k8 cluster repository. The development and production clusters will then have persistent volumes for slinky to make claims on.
Transform the representation of people affiliated with datasets to conform to schema,org. This is most likely a large task that involves a few parts...
We need a way to represent individual files. Right now, there isn't a great way to do this with SOSO.
One option is to represent them as individual nodes in the graph and connect the related dataset(s) to them. This would allow us to further annotate the nodes with additional information, such as which variables they describe.
Unknowns:
For a complete stack, we should have some logging resources at hand. The ELK stack is one of the most popular K8 logging frameworks. It was also present in the original d1lod repository.
Within the context of the DataONE Kubernetes stack, we may want to have a central logging system in place that aggregates logs from other namespaces (including slinky).
The output of the CLI's get
command can be a bit hard to read because it doesn't make effective use of namespace prefixes.
To integrate linked open data into existing web views and services, we likely don't want to expose, say, a SPARQL endpoint. If we did, we might at least want to point it at a read replica.
Another approach would be to put something in front of the SPARQL endpoint to constrain the types of queries that can be constructed by users without direct access to the SPARQL endpoint. There must be some solutions already out there, though I think building something specific to our use case isn't out of scope.
One example is https://github.com/UKGovLD/linked-data-api/blob/wiki/Specification.md
Create a DataONE graph store that:
schema:Dataset
and dcat:Dataset
models, and with OAI-OREWe should refactor our PV & PVC definitions to use the ceph-fs facilities which are now available.
DataONE has a Kubernetes cluster that this stack should work nicely in. Migrate the docker swarm over to a Kubernetes Helm Chart and deploy it on the development cluster.
Once complete, do a fully integrated test to make sure that harvesting and graph population works the same (nothing should change from this). With Kubernetes, we should think about a new logging framework that integrates nicely with the whole stack. For now exclude them from the stack
Essential services to migrate:
On the Jan 27'th salmantics call we brought up the idea of versioning the graph and creating a set of API endpoints that are also versioned.
Virtuoso doesn't support versions the same way that GraphDB does. Instead of creating new repositories, subgraphs are used.
Each subgraph name should follow a convention that we define here. We should use the full slinky URI followed by the version.
For example,
The SPAQRL query is sent to https://api.test.dataone.org/slinky/query
to ask for triples in the https://api.test.dataone.org/slinky/v1
graph.
SELECT * WHERE {
GRAPH <https://api.test.dataone.org/slinky/v1/> {
?dataset rdf:type schema:Dataset .
?dataset schema:name ?name .
?dataset schema:creator ?creator .
OPTIONAL {?dataset schema:description ?description . }
}
}
The SPAQRL query is sent to https://api.test.dataone.org/slinky/query
to ask for triples in the https://api.test.dataone.org/slinky/v2
graph.
SELECT * WHERE {
GRAPH <https://api.test.dataone.org/slinky/v2/> {
?dataset rdf:type schema:Dataset .
?dataset schema:name ?name .
?dataset schema:creator ?creator .
OPTIONAL {?dataset schema:description ?description . }
}
}
On the same Jan 27th call it was decided that we want a number of utility endpoints:
Since each one of these will involve a SPARQL query that we construct on the backend, we have complete control over the SPARQL query. This allows us to insert the version based on content from the URL.
For example, the following endpoint URLs query v1 of the graph
https://api.test.dataone.org/slinky/v1/datasets
https://api.test.dataone.org/slinky/v1/persons
The version 2 analogues,
https://api.test.dataone.org/slinky/v2/datasets
https://api.test.dataone.org/slinky/v2/persons
The dataone graph will hopefully have many references to funders through metadata documents that link to global funder identifers from the CrossRef Funder registry, and from the ROR system. Both are made available as data dumps:
I think it would be helpful to:
While the ROR graph isn't in RDF, its highly structured and provides very nicely structured cross-references to other identifier types like WikiData IDs, ISNI, etc.
Being able to report based on funders has long been a goal of ours, and getting all of this accessible in a queryable form would be really helpful.
We should create a few milestones for work in the foreseeable future. From the 03/11/21 call we know that we need at the least, the graph up and running under K8 and that we want a schema,org representation of datasets and people. The work after (measured variable, files, geospatial, etc) is currently unordered.
1.0: Python 3 + Kubernetes Stack
2.0: Graph with schema:Dataset + schema:Person nodes
From the November 4th development call we should create a new email account for dockerhub to store the related project images. This account can be used by other devs when working on the project.
Years ago, back when we set up d1lod, we decided to handle inserting RDF data into whichever triplestore we used in an agnostic fashion so the triplestore could be swapped out without too much work. So we settled on inserting data via SPARQL INSERT statements.
While revisiting d1lod and repurposingn it for Slinky, I've run into two related issues with this approach:
SPARQL may just not be the right thing for this workload. I considered using alternative RDF data loading methods Virtuoso provides but it looks all they have is a system that loads data from a local filesystem via I-SQL commands.
I had been meaning to look at Blazegraph for a few years and I see that it has a nice HTTP bulk data loading REST API where you can just send serialized RDF to an endpoint. We aren't using any special functionality from Virtuoso so this might be a good point to switch.
Feedback or thoughts welcomed. I'll update here with what I figure out.
OpenAlex is launching December 2021 and could be a great complement to the DataONE graph. They'll have ORCIDs, RORs/GRIDs and lots more. See this [https://blog.ourresearch.org/openalex-update-june/](blog post) for more info.
Right now the worker on the dataset queue isn't waiting for virtuoso to start up and will produce errors while trying to communicate with it. Ideally the deployment would wait for Virtuoso to start up before starting the workers. Alternatively the source code of the worker could wait for an available triple store.
With kubectl, I can write a shell script that utilizes kubectl wait
to pause the deployment of deployments until a particular deployment it ready. This will require a ReadinessProbe on the Virtuoso deployment; I think an HTTP GET should be enough of a canary to make the claim that Virtuoso is ready.
I feel like there's an identity crisis in this repository that's between the actual d1lod python library, the deployment of it, and the documentation.
For example, the fact that we need to copy the d1lod
directory into the docker build directory, which then gets copied into the image, means something's not quite right. Doing this will bring about bugs in the future where the requirements.txt in the d1lod folder aren't installed; I'm assuming this is why some of the scheduler requirements are a subset of the d1lod requirements file (since d1lod is installed in the scheduler, the scheduler's requirements file gives us a way to install the d1lod requirements-without ever touching the d1lod requirements file) (and scheduler doesn't use python-dateutil, but has it defined in the requirements, probably so that d1lod has access to it).
If it were up to me, I'd create a separate repository for the python library, and document it as a standalone library which is standard enough. Then I'd create a second repository that's responsible for DataONE's deployment of services to spin up the graph store. One container in that stack uses the d1lod package and could be installed via a simple git clone & pip install, or just a pip install git+
. This would also allow for better placed documentation. Right now we have the docs/
folder, which is probably more appropriate in the python library's documentation (maybe as readthedocs). The architecture of the Slinky K8 stack would belong in the deployment repository.
At the least we should be able to pip install the d1lod folder. At this moment, attempting to do so raises the following
Any thoughts?
We should document the Kubernetes structure along with the interactions between the worker, scheduler, and virtuoso.
Virutoso now (since 2017?) has official images. We should use those I think: https://hub.docker.com/r/openlink/virtuoso-opensource-7/.
We nee to configure the Slinky ingress to route traffic from *.dataone.org/slinky to Virtuoso.
One problem that I've come across is that the links in Virtuoso Conductor are relative to *.dataone.org/, not *.dataone.org/slinky. This results in a bunch of broken links and buttons. This can be seen on the test deployment.
Regardless of the broken links, the sparql endpoint works fine,
curl -G https://api.test.dataone.org/slinky/sparql --data-urlencode query='SELECT DISTINCT * WHERE { ?s ?p ?o} LIMIT 1000'
@amoeba suggested that an alternative is hosting the service at slinky.api.test.dataone.org.
Science on schema has an explicit way of handling spatial data; if the EML document supports it-we should definitely include it in the graph. This can be a strong case for third party integrations (eg KWG).
This is an issue for cleaning up the dependency ordering in the deployment. Right now we're using the makefile that interacts with kubectl to wait for pods to be in the ready state. This works at the moment, but doesn't apply to the docker stack deployment and adds a layer of complexity to the deployment. It also makes things tricky with Helm charts (see #52).
The general idea is to bring the 'waiting' logic into the codebase and remove it from the deployment layer.
The scheduler is deployed in two steps: the first step is initialization of the scheduler (happens in the slinky cli), the second is starting rqschedule
(happens on the command line). These steps can be seen in the deployment file.
Both of the steps require an active instance of redis and should be able to be started independently of each other without issue. Since rqscheduler is effectively moving jobs to different queues, it should be fine if the scheduler from the first step hasn't submitted the update_job
job since it'll pick it up the next time it checks.
Making the scheduler portion of startup wait on redis can be achieved by adding a method that checks for redis with a timeout and threshold. This same code can be used with the workers (see below).
Unfortunately, rqschedule
doesn't have a retry flag, but we can use the same logic above. I'd like to bring the call to rqschedule
inside the Slinky cli. Either in def schedule
or as a separate command. This would enable us to use the blocking code from above and would allow us to manage the dependency in the code.
The workers need to be able to perform database transactions (needs Virtuoso to be online). They also depend on redis.
Redis is easily tackled by using the blocking call from the scheduler solution.
A similar approach can be taken with Virtuoso, along the same lines as the readinessProbe
@ThomasThelen and I touched base today to go over the existing codebase. I wanted to document some of the issues we talked about here for us and others to see. They're all really leftover technical debt from the initial GeoLink work on this:
slinky
concern anymore and is really more of a k8s cluster thing.If you read this and have any questions or additional items to add please do and I'll update this.
Exposing a SPARQL endpoint for Slinky would be fun but it's not all that useful to anyone but ourselves. And even then.
It'd be better if we had a concise showcase of what Slinky is and what it can do and this could similar to what we did for GeoLink: http://data.geolink.org/.
Today we put together a quick list of interesting example queries that could go on this page:
[I don't think we have an issue for this specifically but let me know if that's wrong]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.