Comments (6)
@derekperkins what you're proposing is meant to support this right? (copied from protoc-gen-validate
issue)
Suppose I have an array of messages with field id
.
message Student {
string id = 1;
}
message Class {
// Could your suggestion be used to validate that students[n].id is not repeated within this array?
repeated Student students = 1;
}
Have you found a way to get this behavior with the current API?
from protovalidate.
Yes, this is exactly the use case I'm suggesting. No, I haven't solved this today with protovalidate, but I'm interested to hear back if you find a good workaround
from protovalidate.
You may be able to get away with this using what's available in CEL today on Class.students
:
this.map(student, student.id).unique()
This uses the e.map
CEL standard macro and our unique
custom function.
from protovalidate.
You may be able to get away with this using what's available in CEL today on
Class.students
:
this.map(student, student.id).unique()
This uses the
e.map
CEL standard macro and ourunique
custom function.
I was able to get this to work by specifying the CEL expression on a message, rather than the repeated field. Here is a code sample:
message Environment {
option (buf.validate.message).cel = {
id: "repeated.unique_by_field"
message: "repeated value must contain unique items, identified by the name field"
expression: "this.variables.map(v, v.name).unique()"
};
// Variables to use within the environment. Each variable must have a unique `name` field.
repeated EnvVar variables = 1;
// [(buf.validate.field).cel = {
// id: "repeated.unique_by_field"
// message: "repeated value must contain unique items, identified by the name field"
// expression: "this.map(v, v.name).unique()"
// }];
}
message EnvVar {
// Name of the environment variable. Must be unique.
string name = 1;
// Value of the environment variable.
string value = 2;
}
The buf.validate.message
CEL expression works like a dream, returning the following violation.
"violations": [
{
"fieldPath": "environment",
"constraintId": "repeated.unique_by_field",
"message": "repeated value must contain unique items, identified by the name field"
}
]
However, as you might have noticed. The fieldPath
is not what is wanted in this scenario. While the expression is being ran on environment
, that expression only ever refers to the variables
field, making it more difficult to properly match validation errors to the exact field they are related to.
If we use the buf.validate.field
CEL rule instead of the one on the message, we hit an issue with the way CEL expressions are processed on repeated fields. It seems that when you manually specify a CEL expression on a repeated field, it is processed on each item within the associated array.
compilation error: failed to compile embedded type Environment for UpdateRequest.environment: compilation error: failed to compile expression repeated.unique_by_field: ERROR: \u003cinput\u003e:1:1: expression of type 'EnvVar' cannot be range of a comprehension (must be list, map, or dynamic)
| this.map(v, v.name).unique()
| ^
Assuming that the issue preventing the expression from being evaluated on the repeated field itself is resolved, there is just one other issue. Which is handling what field needs to be unique. The constraint ID can be used for localization, but right now it is just a generic repeated.unique_by_field
which doesn't indicate any information regarding what field was used for the unique validation. There are two issues here, first is the fieldPath
not being specific enough on what indexes failed the unique check (and the associated field potentially), and the constraintId not having a proper way of communicating what field the unique check is being ran on. Right now the best solution is to either have your message contain the field name, or if you need proper localization, to add the field name to the constraint key manually. But ideally there should be a proper solution for handling things like this in the future.
from protovalidate.
After looking at different ways of doing this, could we perhaps have an annotation to mark unique fields:
message Student {
string first_name = 1;
string last_name = 2;
option (buf.validate.message).unique.keys = ["first_name","last_name"];
}
message StudentList {
repeated Student students = 1 [(buf.validate.field).repeated.unique = true];
}
We could set a restriction on the types of the fields in unique.keys. Say it would be simplest if we only allow:
- allow scalar(optional too), Message(provided it has
(buf.validate.message).unique.keys
option)
Note: it's very important to allow optional here as it's crucial for many validations where you need to differentiate between missing and default value - do not allow: oneof, repeated, Message without
(buf.validate.message).unique.keys
option
Do you think this could work?
from protovalidate.
compilation error: failed to compile embedded type Environment for UpdateRequest.environment: compilation error: failed to compile expression repeated.unique_by_field: ERROR: \u003cinput\u003e:1:1: expression of type 'EnvVar' cannot be range of a comprehension (must be list, map, or dynamic)
@matthewpi, I think this is a bug on protovalidate-go's part (likely in the type checking). This should have worked. I'll take a look and see if I can repro and add some tests for it. [Edit: protovalidate-go#80]
There are two issues here, first is the fieldPath not being specific enough on what indexes failed the unique check (and the associated field potentially), and the constraintId not having a proper way of communicating what field the unique check is being ran on.
@matthewpi, there's #17 and #125 which aim to make it easier to get at the underlying info of a violation error. That said, with a custom CEL expression, you have control over the id and error message and can even construct it within the expression to use string formatting against the variables provided:
repeated EnvVar variables = 1 [(buf.validate.field).cel = {
id: "env.vars.unique", // the ID can be very specific to the field in question
expression: "this.map(v, v.name).unique() ? '' : 'variables are expected to be unique, got %s'.format([this.map(v, v.name)])"
}
We intentionally do not include any data in the standard constraint error messages or as part of the violation error as there are security and PII concerns with echoing potentially sensitive data. You are free to do so in custom expressions however, and we're planning to address the linked issues above in the near term.
We could set a restriction on the types of the fields in unique.keys.
@mivanovcpd, a multi-field uniqueness comparison can be problematic for languages that don't have equivalents of Java's equals
and hashCode
overrides. For protobufs in Go, for instance, you cannot use msg1 == msg2
as Go only performs address comparison for pointer types and further the messages contain internal state that may be different between messages even if their fields are identical (so the protobuf library includes a proto.Equal
which is used for comparison). Likewise, Go does not allow overriding the hash function of a type. This means we cannot use a map to make the uniqueness check O(n)
as we'll need to explicitly proto.Equal
each member of the repeated field to the others (which would run in O(n^2)
). Alternatively, especially with a subset of fields, we'd need to write a custom hashing utility explicitly for this purpose if we wanted to maintain an O(n)
runtime.
That said, if we can determine a way to make this universally efficient and not a performance landmine, either in the case of uniqueness across the entire message or a subset of fields (using a FieldMask WKT for this would offer the most flexibility), I'm open to adding such a standard constraint.
from protovalidate.
Related Issues (20)
- [BUG] Conformance test suite has test cases that are unspecified by the proto3 spec HOT 2
- Add package documentation for Protobuf packages
- [Feature Request] PHP Support
- [Feature Request] Ruby Support
- [Question] Deprecated field validation HOT 1
- [Question] I can't generate a validator with buf, is the plugin used by the buf.gen.yaml file incorrect? HOT 1
- Protoc does not generate validation code [BUG] HOT 1
- [Feature Request] Have URI validations for string but also allow empty HOT 1
- [Feature Request]: Add wellknown regex for URL encoded parameters HOT 1
- Cannot resolve import HOT 1
- [Question] Migration from protoc-gen-validate to protovalidate HOT 2
- [Feature Request] Add to scalapb common protos HOT 3
- [Question] Carrying Patches from protoc-gen-validate HOT 1
- How do you use protovalidate on grpc-gateway? HOT 2
- [Feature Request] How to replace error messages in standard constraints? HOT 1
- [BUG] Regex in CEL with non-capturing group issue HOT 1
- [BUG] migrate: tmp on different partition than target [invalid cross-device link] HOT 1
- [BUG] migration tool: repeated strings with ignore_empty doesn't work HOT 2
- Add access to custom options (of host messages).
- [BUG] Migrator uses wrong field path for string ignore HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protovalidate.