miracl / conflate Goto Github PK

45.0 4.0 6.0 5 MB

Library providing routines to merge and validate JSON, YAML and/or TOML files

License: Apache License 2.0

Go 100.00%

configuration config configuration-management golang golang-library json json-schema yaml toml merge

conflate's Introduction

MIRACL

What is MIRACL? Multiprecision Integer and Rational Arithmetic Cryptographic Library – the MIRACL Crypto SDK – is a C software library that is widely regarded by developers as the gold standard open source SDK for elliptic curve cryptography (ECC).

Why is it different? While many other cryptographic SDKs are focused on PC use, MIRACL also enables developers to build security into highly constrained environments, including embedded, mobile apps and SCADA.

Full documentation can be accessed here:

MIRACL enables

Reduced program code
Greatly simplified program development
Developer-designed APIs
Rapid implementation, using inline code wrappers, example programs and other innovations

These unique qualities are the reason MIRACL, and the solutions and services built using it, are in use in hundreds of organizations across the world, including BAE Systems, Hitachi, Intel, Panasonic, Toyota and many others.

Features and Benefits: why MIRACL is the right choice

MIRACL delivers a wide and unique range of benefits, enabling developers to secure even the most constrained environments quickly, easily and effectively. It features, amongst others:

An inline C++ wrapper – greatly simplifying program development
Over 25 example programs in C and C++, covering a wide range of applications, to give development a head start
Optimization of both embedded processors and RAM, to help developers overcome device and memory constraints
Compatibility with industry security technologies including AES encryption, RSA public key cryptography, Diffie-Hellman key exchange, DSA digital signature, and others
A set of tools that enable any new number-theoretic technique to be implemented quickly

The MIRACL library consists of well over 100 routines that cover all aspects of multi-precision arithmetic. Two new data-types are defined - big for large integers and flash (short for floating-slash) for large rational numbers. The large integer routines are based on Knuth’s algorithms, described in Chapter 4 of his classic work ‘The Art of Computer Programming’. Floating-slash arithmetic, which works with rounded fractions, was originally proposed by D. Matula and P. Kornerup. All routines have been thoroughly optimised for speed and efficiency, while at the same time remaining standard, portable C. However optional fast assembly language alternatives for certain time-critical routines are also included, particularly for the popular Intel 80x86 range of processors. A C++ interface is also provided. Full source code is included.

Bug Tracker

MIRACL Ltd. uses JIRA for bug and feature tracking which is integrated with our development system. If you find a bug, you should report bugs into the MIRACL bug tracker . You can check that the bug hasn't already been reported by searching for it. If you find the bug already reported, you can add a comment of your own about it, or change its status to "Confirmed". If the bug hasn't been reported, you can file a new bug report.

Community

MIRACL Ltd. is most of all a community of like-minded information security professionals who believe that cryptography is a necessary tool to advance individual freedom and safeguard privacy. MIRACL Ltd. acts on that belief by providing tools that can be used to secure information, guard privacy and advance individual freedom.
Anyone who uses MIRACL Ltd. code or services is part of this global community, and we invite you to help shape MIRACL to better meet your needs. To make it yours!
Keep track of development and community news.

Follow @MIRACL on https://twitter.com/MIRACL
Read our blog for the latest security and crypto news first
Have a question that's not a feature request or bug report? Contact Us.

Contributing:

MIRACL Ltd. provides an Open Source suite of solutions for data security. The MIRACL Ltd. team firmly believes that our solutions and the organizations and users who benefit by them all derive value from active contributions from the community.
You can contribute to help shape and improve our MIRACL Ltd. products. If you have ideas and suggestions on new features and improvements that you would like to see and help bring to MIRACL Ltd., please fork the public available code on GitHub.

Authors:

MIRACL Ltd.

Copyright and License:

MIRACL SDK provides developers with an extensive and efficient set of cryptographic functions. For further information about its features and functionalities please refer to https://miracl.com.

MIRACL SDK is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

MIRACL SDK is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You can be released from the requirements of the license by purchasing a commercial license. Buying such a license is mandatory as soon as you develop commercial activities involving MIRACL without disclosing the source code of your own applications, or shipping MIRACL with a closed source product.

For full details regarding our MIRACL Ltd. terms of service please refer to the following links:

conflate's People

Contributors

Stargazers

Watchers

Forkers

ewocker runz0rd kamahl19 axelarnetwork nicorikken zeitlinger

conflate's Issues

Add support for additional headers/options for a remote uri include

In order to support consul authentication the user can configure the uri with a ?token=123 querystring. However this has been deprecated in favour of a X-Consul-Token header. We need to support the ability to add arbitrary headers for any uri in the includes list.

https://www.consul.io/api/

support $ref in applyDefaults

We could do this by expanding the schema out to replace al the $ref with the appropriate concrete schemas. This would be the simplest implementation (as it would leave the applyDefaults code intact, but it would bloat the schema document.

handling list as entity

First off, thanks for this package! In my use case, it works almost too well.
Say i have two sources, each with a list property:
source.A:

group_members:
  - alice

source.B:

group_members:
  - bob

what i end up with by default is a list ["alice", "bob"]. I'm wondering if there is any way to treat lists as a whole, such that if i merge A then B, that source.B list is exclusive (ie just bob)?

cli tool should handle lowercase format specification

e.g.

conflate --data data.json --format json

cli tool doesnt work in windows

./conflate_windows -data data.json -format JSON
parse file://C:\mypath/: invalid character "\\" in host name

example should demonstrate expanding environment variables

Support arbitrary marshallers/unmarshallers

We should add a list of marshallers to each Conflate instance. This should be editable by the client so it can edit the marshallers that should be used to parse the config... i.e. a client might not want to support TOML, or might want to support XML, or some other custom format...

`AddData()` does not recurse the includes

Unlike AddURLs and AddFiles() from data does not recurse through all the descendent includes and merge the data.

Metaschema validation fails when loading metaschema from string

Currently schema validation against the metaschema requires downloading the metaschema remotely using validateSchemaTemporary. This is due to a bug is gojson schema (xeipuuv/gojsonschema#170 and transitively to xeipuuv/gojsonreference#8). When this bug is fixed the code can be reverted to call validateSchema

panic: reflect: call of reflect.flag.mustBeExported on zero Value

Hi everyone,

I'm in the progress of migrating a project from perl to go and have problems with a previously working JSON schema file with conflate. I first tried gojsonschema and the schema was valid there, but had no default value integration, so I turned to conflate.

I ran into a panic (with my code and the conflate binary)

$ ~/go/bin/conflate -data testdata/data.yaml -schema schema/data.schema.json -validate -defaults
panic: reflect: call of reflect.flag.mustBeExported on zero Value

goroutine 1 [running]:
reflect.flag.mustBeExportedSlow(0x0)
        /usr/local/Cellar/go/1.13.4/libexec/src/reflect/value.go:222 +0xad
reflect.flag.mustBeExported(...)
        /usr/local/Cellar/go/1.13.4/libexec/src/reflect/value.go:216
reflect.Value.Set(0x147fbe0, 0xc000267c50, 0x194, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.13.4/libexec/src/reflect/value.go:1532 +0x56
github.com/miracl/conflate.applyDefaultsRecursive(0xc00026a850, 0xb, 0x14816e0, 0xc0001dab10, 0x1453340, 0xc000267c50, 0x14816e0, 0xc0001dade0, 0x0, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:208 +0xa1f
github.com/miracl/conflate.applyObjectDefaults(0xc00026a780, 0x6, 0x14816e0, 0xc0001dab10, 0x14816e0, 0xc0001da0f0, 0xc0001dad80, 0x0, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:247 +0x294
github.com/miracl/conflate.applyDefaultsRecursive(0xc00026a780, 0x6, 0x14816e0, 0xc0001dab10, 0x1453340, 0xc0002678e0, 0x14816e0, 0xc0001dad80, 0x0, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:215 +0x959
github.com/miracl/conflate.applyObjectDefaults(0x150ad41, 0x1, 0x14816e0, 0xc0001dab10, 0x14816e0, 0xc0001da060, 0xc0001dab10, 0x10bd, 0x12bd)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:247 +0x294
github.com/miracl/conflate.applyDefaultsRecursive(0x150ad41, 0x1, 0x14816e0, 0xc0001dab10, 0x1453340, 0xc0000ba980, 0x14816e0, 0xc0001dab10, 0xc000188417, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:215 +0x959
github.com/miracl/conflate.applyDefaults(0x1453340, 0xc0000ba980, 0x14816e0, 0xc0001dab10, 0x0, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:160 +0x6c
github.com/miracl/conflate.(*Schema).ApplyDefaults(0xc000267820, 0x1453340, 0xc0000ba980, 0x0, 0x0)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/schema.go:77 +0x54
github.com/miracl/conflate.(*Conflate).ApplyDefaults(...)
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/conflate.go:112
main.main()
        /Users/andreas/go/pkg/mod/github.com/miracl/[email protected]/conflate/main.go:72 +0x7f2

with this schema properties:

...
"host" : {
          "type": [ "string", "null" ],
          "description": "Hostname",
          "pattern": "^([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])$",
          "default": null
        },
...

If I change it to use only string (and allow the empty string in the validation), it works:

"host" : {
          "type": "string",
          "description": "Hostname",
          "pattern": "^([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9]|)$",
          "default": ""
        },

It's not a big limitation, but I need to change my previously working schema file in order to get it to work with conflate.

Is this a bug worth noting?

Best,
Andreas

update travis to push statically built binary to releases

Workaround for issue 5 fails if there is no network connectivity

Issue #5 depends on a PR xeipuuv/gojsonreference#8 that has not yet been merged. While we wait for this, we have a temporary workaround/hack to load the meta schema from http://json-schema.org/draft-04/schema, but this requires outbound network connectivity. Alternatively, we can use a file uri to a temporary file where we place the meta schema file.

idea: allow for environment variable substitution in config files

We could support substitution of environment variables of the form ${MYVAR} within config file values. This would be optional and would require a new function in the conflate interface.

e.g. if MYVAR=myvalue in the environment, then conflate.ApplyEnvironment() would transform this :

{
  "x": "$MYVAR"
}

Would resolve to :

{
  "x": "myvalue"
}

Since it would also be useful to expand environment variables into non-string fields, the expansion should be performed on the raw data, not on the individual fields.

e.g. if MYVAR=123 then the following is not valid JSON :

{
  "x": $MYVAR
}

However it does expand into valid json

{
  "x": 123
}

In a similar way environment variables can be used to inject whole sections of data into the JSON file if required, although probably not recommended.

Cant read from pipes

i.e. the following command fails

$conflate --data <(echo '{"x": 123}') --format JSON
Failed to load url : 500 : file:///dev/fd/63

Failure to load v4 meta schema occasionally fails

Its not clear why this fails but if you turn off your internet access and then create schema.json containing only {}, then the following command occasionally fails :

conflate --schema schema.json --validate

Whilst it is a mystery why this fails only occasionally, we never want conflate to retrieve the meta schema from the internet as we have it hard-coded.

Removing the id attribute of the metaSchema in schema.go appears to prevent this occurring. Seems to be a consequence of schema/metaschema validation deep in gojsonschema.

default object created always

For a schema such as

{
  "title": "test",
  "type": "object",
  "properties": {
    "obj": {
      "type": "object",
      "properties": {
        "x" : {
          "type": "integer",
          "default": 1
        }
      }
    }    
  }
}

If you supply a file consisting of {} then you should not expect {"obj":{"x":1}}, as there is no default for obj, only a default for obj.x.

You should need to either explicitly add an obj in the data i.e. {"obj": {}} or add a default for obj in the schema :

{
  "title": "test",
  "type": "object",
  "properties": {
    "obj": {
      "type": "object",
      "default": {},      <------------- HERE
      "properties": {
        "x" : {
          "type": "integer",
          "default": 1
        }
      }
    }    
  }
}

Currently this doesnt work as expected.

apply basic indentation formatting to JSON output

Currently the MarshalJSON output generates JSON in an unformatted form. This makes it difficult to read without using a third-party formatter. It is preferable therefore to use some kind of default formatting.

support anyOf, allOf, oneOf, and not, in applyDefaults

This is subtle, e.g. with anyOf it is possible that the data matches several subschemas each with different default values. Do we apply (a) the first default, (b) the defaults in turn (ordering of map is not necessarily the same as the original schema data file), (c) the first default providing there are no other schemas providing defaults, (d) no defaults (unless there is a single matching subschema).

Similar issue for allOf, do we apply the defaults of each subschema in turn, overriding where necessary?

For oneOf, we must check that the data matches a unique subschema, if it matches more than one then we cannot reliably determine which default (from which subschema) to apply.

not is easy, since the data must never match any of the subschemas we know never to apply any of these defaults.

e.g.
https://spacetelescope.github.io/understanding-json-schema/reference/combining.html

Expand() expands unknown/unset env vars to blank strings

In some json there may be data which contains strings like $somename that do not represent environment variables e.g. data that represents code such as variables in golang templates. Secondly, there might be a case where the variable replace takes place in two phases under two different environments, where one environment contains some of the variables and the other environment contains the other variables. Hence, it is sensible that we should only perform replacements where we have the variable defined in the environment, else we should ignore it.

empty value not fllowing the priority rule

according to the priority rule, files at right should take highest priority, but empty values in right file don't.

empty yaml file in includes list causes panic

An empty yaml file is deserialized as a nil map[string]interface{} and this causes a panic in the merge with the parent file.

parent_blank.yaml

includes:
- blank.yaml

blank.yaml

---

Conflate should define its own schema for the `includes` section

Conflate should define a schema, build against it statically, and validate all data files passed to it, against this schema.

Metaschema validation fails when loading metaschema from string

Currently schema validation against the metaschema requires downloading the metaschema remotely using validateSchemaTemporary. This is due to a bug is gojson schema (xeipuuv/gojsonschema#170 and transitively to xeipuuv/gojsonreference#8). When this bug is fixed the code can be reverted to call validateSchema

Expand should recursively expand env vars

If replacement of an env var sets a new env var then that too should be replaced, until no further expansions occur. Need to limit this to prevent possibility of infinite recursion.

support supplementary oneOf, anyOf, allOf, noneOf section in schema with valid 'type'

If you have a schem with a valid type and also oneOf anyOf, allOf or noneOf, then defaults are not applied.

A typical use case is when you have an object schema that is required to have one field or another, but not both, (see https://stackoverflow.com/questions/24023536/json-schema-how-do-i-require-one-field-or-another-or-one-of-two-others-but-no). You can do this kind of thing with the following type of schema schema :

{
    "title": "test",
    "type": "object",
    "properties": {
      "obj1": {
        "type": "object",
        "properties": {
          "prop1": {
            "type": "string",
            "default": "val1"
          }
        }
      },
      "obj2": {
        "type": "object",
        "properties": {
          "prop2": {
            "type": "string",
            "default": "val2"
          }
        }
      }
    },
    "oneOf": [
      {
        "required": [
          "obj1"
        ]
      },
      {
        "required": [
          "obj2"
        ]
      }       
    ]
}

However, conflate does not apply the object property defaults in the case of json input data such as :

{
  "obj1": {} 
}

Conflate should output :

{
  "obj1": {
    "prop1": "val1"
  }
}

Need to remove "includes" from merged data

Currently the merge process leaves the "includes" entry in the merged output file. We should remove this, as currently it gives misleading results.

Add support for additional headers/options for a remote uri include

In order to support consul authentication the user can configure the uri with a ?token=123 querystring. However this has been deprecated in favour of a X-Consul-Token header. We need to support the ability to add arbitrary headers for any uri in the includes list.

https://www.consul.io/api/

apply order based on schema

Can we reorder output data based on order of schema as an option?

Needs to provide a --version flag

conflate --version

Can we sponsor a feature request?

We are planning on using this in our production app, but require a feature. Can we sponsor it and give it back to the community?

merging POD value over an array or map should fail

Merging { "x": 123 } over { "x": {} } or { "x": [] } should fail with a type error but currently it doesn't.

Allow user to specify name of key for includes array

i.e. conflate --includes 'using' --data test.json --format JSON

{
  "using": [
     "other.json"
  ]
}

If no --data arguments are specified then use stdin

This should work

cat test.json | ./conflate --format json

$ echo  | ./conflate --defaults --schema ../testdata/test.schema.json --data stdinN
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/miracl/conflate.applyDefaultsRecursive(0x84ac91, 0x1, 0x7b0e40, 0xc420085620, 0x7e1960, 0xc420231170, 0x0, 0x0)
	/home/andy/Documents/projects/GO/src/github.com/miracl/conflate/schema.go:119 +0x645
github.com/miracl/conflate.applyDefaults(0x7b0e40, 0xc420085620, 0x7e1960, 0xc420231170, 0x0, 0x0)
	/home/andy/Documents/projects/GO/src/github.com/miracl/conflate/schema.go:56 +0x5e
github.com/miracl/conflate.(*Conflate).ApplyDefaults(0xc420085620, 0x0, 0x1c)
	/home/andy/Documents/projects/GO/src/github.com/miracl/conflate/conflate.go:146 +0x56
main.main()
	/home/andy/Documents/projects/GO/src/stash.certivox.com/scm/msso/conflate/conflate/main.go:51 +0x617

allow the cli tool to read from stdin

Support reading from stdin. We need to be able to specify where in the list of other files it should be inserted though i.e.

echo '{"x": 123}' | ./conflate ---data firstfile.json --data stdin --alaterfile.json --format JSON