Add strict entity mapper about lura

[kpacha]

@joaoqalves , @dhontecillas : I'd rather go with the smallest increment possible in the direction described by @alombarte and I'd try to design some kind of optional validation mode using the fields listed in the whitelist, always keeping an eye on the performance impact.

Once we got that, we can start thinking about complex ways of defining the expected dataset/model.

[joaoqalves]

@kpacha I have a couple of ideas about this, but I'd like these issues to be described more in depth :)

[alombarte]

@joaoqalves Yes, the issues are more like a TODO list right now...

Current behavior:

• A field whatever is defined in the whitelist
• KrakenD goes to the backend and reads the response
• The field whatever is missing from the response
• KrakenD ignores the field, no error of any kind is raised.

Desired behaviour:

• Strict mapper can be enabled if desired (flag)
• When enabled, krakend goes to the backend and readsthe response
• The field whatever is missing from the response
• The mapper returns an error (that could be logged)

@kpacha will give you more details about the implementation, I just described the functionality.

Thanks

[dhontecillas]

The current implementation prunes all elements that are not in the whitelist, without taking into account that the elements exists.

Having had a look at the code, I think there are two approaches:

• Have a separate step that checks that all whitelisted fields exist in the response, after the pruning has done its job (with a O(N) cost, being N the number of 'nodes' of the whitelist - by the way, we already have a linear cost).

• Try to have an early check inside the whitelistPrune function, (in proxy/formatter.go), and change the signature for that function, so it returns an canDelete and an error if it detects that some fields are missing. I would need to deep a little more into this second option to know it its worth the complexity.

I can have a look (but not this week). So, let me know if someone already starts working on this.

[joaoqalves]

@alombarte Yes, I agree with these points. But we're looking at the first solution we got. Apart from implementing this, I suggest to implement a suggestion that imho brings more value to the table: to read JSON schema. The idea behind this is:

1. You touch an API
2. openAPI/whatever generates the schema
3. You run a tool that compares the mapping against the schema and you won't let deploy anything that breaks your system

[kpacha]

After playing around with some ideas, I think the easiest way to do the validation would be as a martian verifier using a json-schema for every backend response to validate.

Delegating this task to the entity combiner could add unrequired overhead

So, this approach requires clients to define the json-schema to validate in the extra_config section of every backend to check (like this example http://json-schema.org/examples.html)

[kpacha]

this two martian verifiers should do the job: https://github.com/kpacha/martian-components/tree/master/body/json-schema.

I'm not sure if these verifiers should be loaded by default into the official krakend-ce binary or if we should keep them for custom builds until we solve the 'plugin' issue

[joaoqalves]

@kpacha Kudos to the JSON schema approach 👍 . I had the same idea, although mine involved a static analysis before deployment and that's out of Krakend's scope.

[kpacha]

please, keep in mind the impact in performance before using it.

My first benchmarks are not too pretty:

pkg: github.com/kpacha/martian-components/body/json-schema
BenchmarkRequestVerifier_ModifyRequest_ok-8     	  300000	      4617 ns/op	    4384 B/op	      47 allocs/op
BenchmarkRequestVerifier_ModifyRequest_ko-8     	  100000	     13176 ns/op	    6610 B/op	      94 allocs/op
BenchmarkResponseVerifier_ModifyResponse_ok-8   	  300000	      4656 ns/op	    4384 B/op	      47 allocs/op
BenchmarkResponseVerifier_ModifyResponse_ko-8   	  100000	     12786 ns/op	    6610 B/op	      94 allocs/op
BenchmarkVerifier_ok-8                          	  500000	      3603 ns/op	    2096 B/op	      42 allocs/op
BenchmarkVerifier_ko-8                          	  200000	     11284 ns/op	    4321 B/op	      89 allocs/op
PASS
ok  	github.com/kpacha/martian-components/body/json-schema	9.992s

I'll benchmark other libs to be sure this is the fastest option

[github-actions]

This issue was marked as resolved a long time ago and now has been automatically locked as there has not been any recent activity after it. You can still open a new issue and reference this link.