Comments (16)
That's funny @DeltaEvolution, that was one of the use cases I wanted to add with tags, file inclusion. That's a decent example you have there for it, in the mean time.
from yaml-rust.
I don't know Rust so forgive me that I can't give more Rust-oriented advice. But I know YAML quite well. What is often overlooked in implementations, beside the intermediate representation graph that @flyx mentions, are Tag Schema. I am not sure a single implementation of YAML currently in the wild actually handles this correctly. A complete implementation would allow for the definition of alternate schema in order to fully control the use of tags in the processes of loading and dumping, and of course include JSON and Core schema out of the box.
I am presently working on adding this support to Crystal, which is built on top of libyaml. When I am done I'll drop a line here if anyone wants to take a look for reference sake.
from yaml-rust.
This library currently support several standard tag directives, e.g. !!str
. But this feature is seldom used, can you give me some typical use cases? And how other YAML libraries support this feature (for example, in python or ruby) ?
from yaml-rust.
Sure, I need to parse a Yaml CloudFormation template, which may or may not have local tags in it. I need to replace some of those tags (and their values) with new values; but some tags I just need to leave alone, so they'll be the same when I dump the Yaml. Here's a really simple example from one of my CloudFormation files:
WebInstance:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Region: !Ref AWS::Region
Timeout: !GetAtt ["a", "b", "c"]
I looked at how Ruby and Python handle custom tags. Tags are used to express the type of next node in the Yaml document, Ruby and Python both expose constructor functions that let the user register a function to be called when a specific tag is encountered, and the result of this function is then used as the value of the tagged node.
In Ruby:
require 'yaml'
YAML.add_domain_type '', 'Ref' do |type,val| { "Reference": val } end
thing = YAML.load_file('../../small-test.yaml')
puts thing.inspect
Which returns
{"WebInstance"=>{"Type"=>"AWS::EC2::Instance", "CreationPolicy"=>{"ResourceSignal"=>{"Region"=>{:Reference=>"AWS::Region"}, "Timeout"=>["a", "b", "c"]}}}}
There are actually four different methods in Ruby's Yaml module: add_builtin_type
, add_domain_type
, add_private_type
, add_ruby_type
Docs. They all do pretty much the same thing though, the major difference is the prefix of the tag (if it's a global tag from yaml.org
, a custom domain, a private tag, or a global tag from ruby.yaml.org
, respectively).
Ruby's implementation seems a bit wonky, they don't explicitly allow you to add local tags (aka !Ref
), so instead you must use add_domain_type
without a domain.
In Python:
import yaml
from pprint import pprint
class CFReference:
def __init__(self, reference):
self.reference = reference
class CFGetAtt:
def __init__(self, attribute):
self.attribute = attribute
def getatt_constructor(loader, node):
return CFGetAtt(loader.construct_sequence(node))
def ref_constructor(loader, node):
return CFReference(loader.construct_scalar(node))
yaml.add_constructor('!Ref', ref_constructor)
yaml.add_constructor('!GetAtt', getatt_constructor)
with open('../../small-test.yaml', 'r') as fh:
pprint(yaml.load(fh))
Which returns
{'WebInstance': {'CreationPolicy': {'ResourceSignal': {'Region': <__main__.CFReference instance at 0x10b4426c8>,'Timeout': <__main__.CFGetAtt instance at 0x10b442710>}},'Type': 'AWS::EC2::Instance'}}
Python also has support for presenters, functions that take a class and return its Yaml equivalent for serialization. I'm not 100% how Ruby does this, perhaps by calling to_yaml on objects.
Unlike Ruby, Python will fail if it encounters a tag it can't construct. The Yaml spec says this about unresolved tags:
If a document contains unresolved tags, the YAML processor is unable to compose a complete representation graph. In such a case, the YAML processor may compose a partial representation, based on each nodeβs kind and allowing for non-specific tags.
A few of the features I think would be useful:
- Generalized tag constructors
Python allows you to specify a prefix, and all tags starting with that prefix get passed to a specific constructor. This would be really useful for me, especially if that prefix can be blank (aka a catchall constructor).
Alternatively, instead of registering constructors, this library exposes a single function that gets called for all unrecognized tags. - Tags are dumped correctly
This is really important for my use case, because I don't want to do anything with some tags. It'd be really nice ifserialize(deserialize(yaml)) == yaml
with respect to tagging. Or at least make this achievable (even if I have to translate tagged values to structs during deserialization, and when dumping those structs explicitly state what tag they should have when they're serialized). - Unresolved tags
I don't think unresolved tags should be an issue for this library. Unlike serde_yaml, this library doesn't deserialize to structs; so unresolved tags should be much less of an issue. Maybe unresolved tags get a special struct that derefs to the value (this way the tag is still available). This way serde_yaml could then fail if it encounters an unresolved tag, but this library wouldn't need to.
from yaml-rust.
I need this functionality too , actualy I copy paste YamlLoader struct and I add this code in the on_event when event is Scalar in the match
I'ts a tag to include yaml tree into another
Exemple:
hello: !include 'hello.yaml'
The code
if style == TScalarStyle::SingleQuoted {
if let Some(TokenType::Tag(ref handle, ref suffix)) = *tag {
if *handle == "!" && *suffix == "include" {
let mut content = String::new();
let node = match File::open(Path::new(self.root.as_str()).join(v)){
Ok(mut f) => {
let _ = f.read_to_string(&mut content);
match YamlLoader::load_from_str(content.as_str() , self.root.clone()) {
Ok(mut docs) => docs.pop().unwrap(),
Err(_) => Yaml::BadValue
}
}
Err(_) => Yaml::BadValue
};
self.insert_new_node((node , aid));
return;
}
}
}
But It will great to have an api for this
(And more generaly an api to customise YamlLoader)
from yaml-rust.
@ioben ;)
@chyh1990 If you want more exemple with another's libraries, I have the !include tag with snakeyaml in java
from yaml-rust.
I have created a very simple implementation here
The parser with this api look like
struct HelloParser;
impl YamlScalarParser for HelloParser {
fn parse_scalar(&self, tag: &TokenType, value: &String) -> Option<Yaml> {
if let TokenType::Tag(ref handle, ref suffix) = *tag {
if *handle == "!" && *suffix == "hello" {
return Some(Yaml::String("Hello ".to_string() + value))
}
}
None
}
}
So you can parse not only tags but all types of scalar values
from yaml-rust.
Hi folks!
Not sure how I ended up here, but since it has been asked how other libraries use this feature, I think I can contribute some thoughts. I am the author of NimYAML, which employs tags by default. Since most other YAML implementations are written for languages with dynamic typing, we might benefit from sharing ideas here.
First, a word on terminology: A tag directive is a prefix before the YAML document; what you are talking about is tag handles:
%YAML 1.2
%TAG !n! tag:yaml.org,2002: # << this is a tag directive
---
!n! tagged scalar # << this is a tag handle
Tag handles are used to explicitly denote the type of a YAML node. The type of a YAML node without a tag is deduced from its content and its ancestors in the same document. Tag handles can be employed for type safety. For example, in NimYAML, I can serialize a sequence of integer values like this (I hope this code is readable for people not knowing Nim; @[]
is a sequence constructor):
import yaml.serialization
echo dump(@[1, 2, 3])
Which will generate:
%YAML 1.2
---
!nim:system:seq(nim:system:int64) [1, 2, 3]
When loading this YAML file with NimYAML, it will check whether it is loaded into a sequence type that matches the tag of the value and raise an error if it doesn't. That way, type errors can be discovered early. Note that the numbers inside the sequence do not need tags because their tag is implicitly given by an ancestor (the sequence).
Now I understand that yaml-rust does not load YAML into native types, but rather into a DOM. This corresponds to the Representation (Node Graph) as defined in the spec. As you can see in the spec, tags are part of that Node Graph, so there is nothing yaml-rust itself is required to do with them if its API barrier is between the Node Graph and native data structures. Its only responsibility should be to collect the tags present in the YAML input and make them accessible in its DOM. NimYAML also has a DOM API, but its use is discouraged because serializing to and from native types is more convenient and automatically includes type checks.
A use-case which might be relevant for yaml-rust is tag handles in sequences or mappings with heterogeneous value types. An example from NimYAML documentation:
%YAML 1.2
---
- this is a string
- 42
- false
- !!str 23
- !nim:demo:Person {name: Trillian}
- !!null
To process this sequence, the type of each value must be evaluated. For scalar values, this could be done by guessing the type (this is a string
-> string, 42
-> integer). But if you have multiple complex types that can occur inside this sequence, it can be very impractical (or downright impossible) to infer their type from the value. Therefore, using tags here is very helpful. Since I do not want to clutter this conversation with code too much, I will simply link to the NimYAML examples to show how this data is loaded with NimYAML.
So, to sum up: My advice would be to simply make tags queryable from your DOM interface and let the caller decide what to do with them. They are very relevant for implementing automatic (de-)serialization of native types, and perhaps you want to add that some day to yaml-rust.
Cheers!
from yaml-rust.
Thanks for the detailed discussion!
Here's my options:
- Handling tags with hooks (in the same way in Python or Ruby) is easy, maybe it can be added in the next major release. @ioben @DeltaEvolution
- Unfortunately, yaml-rust does not implement a full DOM containing all metadata in the original text file (e.g. tags, mark, etc.) yet. This makes
serialize(deserialize(yaml)) == yaml
impossible. The serializer is preliminary, we need contribution for that. @ioben - tag directives for standard types are already supported by yaml-rust (also type guessing), but for a strongly typed language like Rust, maybe it's not easy to do auto type casting? @flyx
@DeltaEvolution Can you generalize your patch and submit a PR?
from yaml-rust.
@chyh1990 The root type must be known at compile time and it can only be checked whether the YAML has the correct tag on its root element for strongly typed languages, that is correct. However, if you have
enum Message {
Quit,
ChangeColor(i32, i32, i32),
Move { x: i32, y: i32 },
Write(String),
}
(forgive me, I do not really know Rust and just copy things from the documentation), you can use Vec<Message>
as root type, which would make it possible to deserialize this to a native type:
- !Quit # no value
- !ChangeColor [1, 2, 3] # sequence as child
- !Move {x: 23, y: 42] # mapping as child
- !Write some string # scalar as child
If you have exactly one value for each enum item, and all these values have different types, you can match the tag (or the guessed type) against the value types instead of using the enum items as tags. This makes it possible to deserialize complex heterogeneous structures. You do not lose type safety and do not do any type casting.
from yaml-rust.
@DeltaEvolution propose a new API for tag parsing (see #37 ). It looks OK for me. Any opinion for the PR? @flyx @ioben
from yaml-rust.
@trans I don't think that it's overlooked. The whole idea of having programming language independent tags never really took off, because YAML is being used more for serialization/deserialization within the same application and for configuration. In popular YAML implementations, you can specify custom tags in addition to those supported out of the box, and the spec advices that other schemas are based on the core schema, so I don't really see how that is not correct.
from yaml-rust.
What's the status of this. I just surveyed the rust yaml ecosystem and this crate looks to be the most complete despite missing features like this that would make it complete or at least on par with other language ecosystems support for yaml tools.
Have others been able to work around this yet? I'm also needing to parse cloudformation yaml templates that leverages several !Foo tags https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html
from yaml-rust.
yaml-rust/tests/specexamples.rs.inc
Lines 55 to 73 in 9f7d848
from yaml-rust.
Is there an interface on master for registering tags?
from yaml-rust.
FWIW I've merged #135 (a rebased version of #37) into my fork: https://github.com/davvid/yaml-rust
from yaml-rust.
Related Issues (20)
- Add support for YAML 1.1 HOT 1
- Support alternate line endings
- Bug in anchor handling? HOT 2
- Panic uninitialised linked hash map HOT 2
- can not part ref HOT 1
- Doesn't seem possible to iteratate over results HOT 3
- misleading? HOT 1
- is it wasm compatible? HOT 1
- Dupliacte keys are not detected as invalid YAML
- Add API to emit YAML strings (i.e. `need_quotes` and `escape_str`)
- tabs are not allowed as the first character of a block scalar
- `Parser` parses empty scalar as '~' HOT 1
- Support no-std with alloc crate
- Integration with google oss-fuzz fuzzing service
- Maintainers HOT 8
- unsafe-libyaml looks... unsafe? HOT 1
- Dynamically choosing a YAML document path? HOT 1
- `True` and `False` boolean literals seem to be parsed as strings HOT 2
- Is this crate maintained? HOT 4
- Read This First ~ Switch to the actively maintained yaml-rust2 fork
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yaml-rust.