Framing about plone.restapi

[lukasgraf]

At the Barcelona Sprint we discussed this, and were leaning towards already changing the layout of the default GET response body in a way that could later be controlled via JSON-LD framing.

Rationale

The basic idea was to move field data out of the top-level namespace / data structure into their own fields dictionary, and using the top-level namespace for things that aren't related to the content type's schema, like @id, @type, items, the parent pointer, and hypermedia links to named contextual endpoints on that resource.

The aim here is not to already implement full JSON-LD framing and using e.g. pyld to apply the framing algorithm, but rather to change the default format of a GET response in a way that addresses some namespacing issues and would mean less radical changes when we later tackle actual JSON-LD framing.

Proposed Format

The proposed format would look somewhat like this:

{
  "@id": "http://nohost/plone/front-page",
  "@type": "Document",
  "UID": "1f699ffa110e45afb1ba502f75f7ec33",
  "fields": {
    "title": "Welcome to Plone",
    "description": "Congratulations! ...",
    "...": ""
  },
  "parent": {
    "@id": "http://nohost/plone",
    "...": ""
    },
  "actions": {"@id": "http://nohost/plone/front-page/@actions/"},
  "versions": {"@id": "http://nohost/plone/front-page/@versions/"},
  "workflow": {"@id": "http://nohost/plone/front-page/@workflow/"}
}

A basic mock of almost exactly that format has been implemented in the barcelona-mock branch (see docs with example).

Open Questions

• Would that mean we also change the format to be used with POST, PATCH and PUT accordingly?
• Would we be strict about only putting field data in fields, and keep other attributes / properties at the top-level?
  You'll notice I left UID and the parent pointer at the top-level in the proposed structure, but I'm not yet clear about what the implications of this would be.

[lukasgraf]

Just so I don't rip my hair out a second time:

JSON-LD Framing Proof of Concept

We start with a fully expanded document that contains everything inline:

from pyld import jsonld

context = {
    '@vocab': 'http://plone.org/vocab#',
    'plone': 'http://plone.org/vocab#',
    'fields': 'plone:fields',
    'workflow': 'plone:workflow',
}

doc = {
    '@context': context,
    '@id': 'http://localhost/front-page',
    '@type': 'Document',
    'fields': {
        '@id': 'http://localhost/front-page#fields',
        'title': 'Front Page',
        'description': None,
    },
    'actions': {
        '@id': 'http://localhost/@actions',
        '@type': 'ActionsInfo',
        'actions': [
            {
                '@id': 'edit',
                'category': '',
                'title': 'Edit'
            },
        ]
    },
    'workflow': {
        '@id': 'http://localhost/@workflow',
        '@type': 'WorkflowInfo',
        'transitions':
            {
                '@id': 'http://localhost/@workflow/publish',
                'title': 'Publish'
            },
    }
}

frame = {
    '@context': context,
    'workflow': {'@embed': False},
    'actions': {'@embed': False},
}


framed = jsonld.frame(doc, frame)
compacted = jsonld.compact(framed, context)

After compaction, this then results in this final document:

{
    '@context': context,
    '@id': 'http://localhost/front-page',
    '@type': 'Document',
    'actions': {'@id': 'http://localhost/@actions'},
    'workflow': {'@id': 'http://localhost/@workflow'},
    'fields': {
        '@id': 'http://localhost/front-page#fields',
        'title': 'Front Page'
    }
}

---

A frame that inlines the actions would then look like this:

frame = {
    '@context': context,
    'workflow': {'@embed': False},
    'actions': {'@embed': True},
}

resulting in

{
    '@context': context,
    '@id': 'http://localhost/front-page',
    '@type': 'Document',
    'fields': fields,
    'actions': {
        '@id': 'http://localhost/@actions',
        '@type': 'ActionsInfo',
        'actions': [
            {
                '@id': 'edit',
                'category': '',
                'title': 'Edit'
            },
        ]
    },
    'workflow': {'@id': 'http://localhost/@workflow'}
}

References:
http://iiif.io/api/annex/notes/jsonld/#frames
http://json-ld.org/spec/latest/json-ld-framing/#dfn-object-embed-flag
https://www.w3.org/TR/json-ld/#default-vocabulary
https://lists.w3.org/Archives/Public/public-linked-json/2011Aug/0078.html
https://lists.w3.org/Archives/Public/public-linked-json/2011Sep/0002.html

As stated above, the goal isn't to implement this anytime soon, but it serves as proof that JSON-LD framing is the right tool for the job, we can use it to accomplish what we need, and we're not barking up the wrong tree (no pun intended).

[lukasgraf]

@tisto @buchi do any of you have any strong opinions on the two open questions above?
(I'm personally leaning towards "format for deserialization should probably also change accordingly" and "separate schema fields from other kinds of properties / attributes").

Because if this still requires some substantial discussion / research, we might want to reconsider and not view it as a blocker for an alpha release.

[tisto]

@lukasgraf I don't see framing as a blocker for an alpha release. Thanks for the detailed analysis/proposal. Will try to find time to reply asap.

[lukasgraf]

Based on a conversation with @buchi, he also doesn't see this as a blocker. I'm good with that as well. I therefore updated issue #111 accordingly.

[bloodbare]

schema may be also a framing option ?

{ '@context': context, '@id': 'http://localhost/front-page', '@type': 'Document', 'schema': }

[lukasgraf]

@bloodbare schema should absolutely be possible I think, once we get the prerequisites in place for doing framing in general.

It would just be another node that could be expanded or compacted, like the actions and workflow nodes in my PoC above.

However, before we attack framing, we need to get our @contexts sorted I think. The JSON-LD Framing spec is still work in progress, and at the moment it seems to be the case that several of the framing algorithms will drop elements that aren't defined in the vocabulary.

---

This is also currently being discussed on the JSON-LD mailing list:

I played with the JSON-LD Frame feature and it is pretty cool. The problem is that it seems to be a "feature in progress" since it only works on some case:

ISSUE 110
Currently, framing allows just to select node definitions based on @type matching or duck typing for included properties. It allows value properties to be explicitly matched based on defining the property and excluding things that are undefined, but it does not allow to be more specific about the types of values selected. Allowing this is currently being discussed.

Yes, it’s likely to remain in progress for a while, until the authors complete current projects, or new ones step forward. In fact, the spec now is at odds with existing implementations in minor ways.

From: https://lists.w3.org/Archives/Public/public-linked-json/2016Jul/0005.html

---

I also ran into this behavior with my PoC using the pyld implementation. So before we play around with framing, we need to make sure all our responses are valid JSON-LD and have @contexts that at least define a default vocabulary.

[tisto]

One other important issue we have to keep in mind is performance when we use framing. I did some research a while ago and it seems the framing algorithms all add a significant performance penalty to the request.

@lukasgraf is there a way we can validate our responses? If so we could include that into our tests.

[bloodbare]

@lukasgraf @tisto We are doing big progress on plone.server and we already implemented a framing option:

• IMHO jsonld framing is overkilling what we need, neither back libraries neither front libraries performance well, its overkilling something simple and its not adding a great functionallity.
• what we need is the option to ask for the content GET with extra information so the front does not need to call 10 GET to be able to render a page
• in plone.server we implemented a simple way where there is a parameter called frame, where you can decide which frames you want on a coma list : GET /site ?frame=schema,actions . Defining named adapters allow us to be able to extend the response dictionary with new parameters that fullfill the needs.

I know that all the autodiscovering of the json+ld is sweet, but after dealing with it on back performance on non json DB, with python implementations and how to work that on the front, I think its not what we need.

[lukasgraf]

@tisto @bloodbare

is there a way we can validate our responses?

I don't know, couldn't find anything after a quick search. Feeding the output through the framing/compaction algorithms using pyld and asserting on the expected results could be an approach though.

IMHO jsonld framing is overkilling what we need

In a way, I tend to agree. JSON-LD framing is basically a way to control how exactly a graph should be serialized to a tree. We (would) barely use any of all the possibilities that offers. We don't want to move nodes around, omit or replicate them. All we basically want is to collapse/expand certain nodes, depending on what the client asks for. That's a very small subset of what JSON-LD framing is capable of.

what we need is the option to ask for the content GET with extra information

Yep. Basically we're aiming to use framing purely as a performance optimization, so the client can already have all the answers it needs inlined in the response instead of having to ask for it piece by piece by following 10 hypermedia links.

GET /site?frame=schema,actions

That's exactly how I would see framing being controlled in plone.restapi.

Defining named adapters allow us to be able to extend the response dictionary with new parameters that fullfill the needs.

I think an approach like that could very well be JSON-LD compliant, as long as every key / every node that can potentially be requested to be included in the response, is always present - but not necessarily in its expanded form.

Also, every "framable" node would need to have a dedicated GET endpoint, that when requested, would return the exact response that would be placed in the node if it was inlined (expanded), and the URI to that endpoint would be used as a reference instead if the node was collapsed.

Whether or not we use a fully JSON-LD compliant library like pyld for that doesn't matter that much I think. It would be feasible to just implement that small subset (collapsing/expanding) ourselves, and still stay compliant with JSON-LD - if that's what we want.

Regarding performance: I agree this is something we need to keep an eye on, I haven't looked into that yet. Can you guys give some more details on that? I can't imagine that an offline transformation of a JSON-LD document with the complexity we're dealing with would add any significant penalty. Are we talking transformations where the algorithm is actually dereferencing IRIs (i.e, doing actual network requests)?

I would rather see performance problems if you were to use the naive approach from my proof of concept above: Prepare everything we could ever need (schema, actions, ...) to generate a fully expanded document, and then collapse most of it and send it to the client.

That would obviously need to work the other way round in an actual implementation. I haven't looked into this too much, but I think a document loader could be used for addressing that.

pyld's default document loader fetches documents via network. We could instead use one that directly calls the needed implementation (the mentioned named adapters for example) instead.

[tisto]

Closing for #149.