beached / daw_json_link Goto Github PK

With a causal glance, most things are ok. Some things that might come up are
use of std::uint32_t/std::uint64_t to help with parsing, along with utf8 decoding. May have to require 32bit unsigned type

There are some helpers for std::optional/std::unique` pointer around nullable types. This may work better with a concept of sorts. But could be removed too.

I have no investigated the float parsing in daw_json_link, but I wanted to make sure you were aware that we have packaged the fast number parsing routine from the simdjson library into its own single-header library: https://github.com/lemire/fast_double_parser
This library provides exact parsing at high speed (under Linux, freeBSD, macOS, Visual Studio).

Feel free to close this issue if it is not relevant.

The Range name no longer fits as it is the general parse state now, not a range of characters

Allow to call from_json with a second argument that is a preparsed version of the JSON Path string to speed up parsing

Allow an event based parsing function.

Currently the project is installed in /usr/local/include/include/daw/...

The error comes from,

When reading the docs... https://cmake.org/cmake/help/v3.13/command/install.html#directory

The directory structure is copied verbatim to the destination. The last component of each directory name is appended to the destination directory but a trailing slash may be used to avoid this because it leaves the last component empty.

CMake provides the example:

install(DIRECTORY icons scripts/ DESTINATION share/myproj
        [...])

will install the icons directory to share/myproj/icons and the scripts directory to share/myproj.

The code in questions adds an include folder to the include destination (this is a very confusing CMake feature 😕 )

This is more generally said that a single sub member can describe the structure of the others.

struct ConfigV1 {
  string option1;
};

struct ConfigV2 {
  vector<string> option1;
};

The JSON document can have a mapped/non-mapped member that allows us to know which alternative to parse to.

Possibly a dependency-chain omission, but it was necessary to explicitly run make check in order for make test to work.

To reproduce, it was enough to run make clean and then make test will fail, as it does not find the necessary binaries to run.

The first immediate error when trying to import this library is the use of CMAKE_SOURCE_DIR instead of CMAKE_CURRENT_SOURCE_DIR.

Second, the CMakeLists.txt doesn't seem to export anything at all so there is nothing to include.

As a sidenote, CMakeLists.txt seems to be in a very chaotic state, maybe it could use a clean-up anyway.

For example allow a class that takes a single string as it's ctor to map to a json_string

Currently only the json iterator supports settings the parse start via json paths. Add support to normal parsing too

Currently there is a push parser(the event driven one). This has an internal loop and state, events are handled by a callback handler. This can be extended such that the loop can be iterated by an external call to a next( ) method

Related to issue #177, Generate JSON Schema from JSON Link bindings

It is desirable to have a tool to generate the data structures and bindings from a JSON schema file.

There seems to be some issue with json_custom types - or I am missing something. I wanted to try daw_json for the public data from binance.com . It seemed to work fine at first - but when I implemented the custom type Fixed8 (replaced by double in the code below) the order of the fields "a" and "b" in the json suddenly seemed to make a difference. When the order in the json matches the oder in the json_data_contract declaration it works fine, if the two arrays are swapped an exception is thrown instead.

The order of keys in the input are not meant to make a difference, right?

const std::string testJson = R"({"e":"depthUpdate","E":1609884707320,"s":"BTCBUSD","U":2544556159,"u":2544556219,"a":[["34198.19000000","0.00000000"],["34198.23000000","0.00000000"],["34198.25000000","0.00000000"],["34198.27000000","0.00000000"],["34198.30000000","0.00000000"],["34198.32000000","0.00958500"],["34198.40000000","0.01232200"],["34198.41000000","0.01000000"],["34198.87000000","0.00000000"],["34199.12000000","0.00000000"],["34199.16000000","0.00000000"],["34199.42000000","0.00000000"],["34200.25000000","0.00000000"],["34200.71000000","0.03199900"],["34201.27000000","0.03100000"],["34201.62000000","0.00000000"],["34202.58000000","0.00000000"],["34204.45000000","0.00952700"],["34207.64000000","0.00000000"],["34207.74000000","0.00000000"],["34209.77000000","0.00000000"],["34209.81000000","0.20400000"],["34225.94000000","0.20200000"],["34226.60000000","0.91050000"],["34236.08000000","0.30000000"]],"b":[["34198.31000000","0.00000000"],["34196.54000000","0.00453200"],["34193.34000000","0.00000000"],["34189.89000000","0.00000000"],["34188.82000000","0.00000000"],["34185.32000000","0.00000000"],["34184.84000000","0.06350200"],["34184.83000000","0.20000000"],["34180.61000000","0.08622700"],["34180.60000000","0.00000000"],["34180.59000000","0.19200000"],["34180.02000000","0.00000000"],["34180.01000000","0.00000000"],["34176.88000000","0.00000000"],["34166.48000000","0.00000000"],["34166.47000000","0.00000000"],["34159.85000000","0.03317500"],["34159.24000000","0.09394900"],["34158.29000000","1.00000000"],["34154.86000000","0.00000000"]]})";

struct Fixed8JsonConverter {
	double operator( )( std::string_view sv ) const {
		return stod(std::string(sv));
	}
	
	template<typename OutputIterator>
	constexpr OutputIterator operator()(OutputIterator _it, double _f) const {
		return daw::json::utils::copy_to_iterator(_it, std::to_string(_f));
	}
};

template<JSONNAMETYPE name>
using json_fixed8 = daw::json::json_custom<name, double, Fixed8JsonConverter, Fixed8JsonConverter>;

struct Change {
	double rate;
	double amount;
};

namespace daw::json {
	template<>
	struct json_data_contract<Change> {
		using type = json_ordered_member_list<
						json_fixed8<no_name>, 
						json_fixed8<no_name>
					>;
	};
}

class DepthUpdateJson {
public:
	int64 time; // E
	std::string pairName; // s
	int64 idTo; // u
	int64 idFrom; // U
	std::vector<Change> bid; // b
	std::vector<Change> ask; // a
};

namespace daw::json {
	template<>
	struct json_data_contract<DepthUpdateJson> {
		using type = json_member_list<
			json_number<"E", int64>,
			json_string<"s">,
			json_number<"u", int64>,
			json_number<"U", int64>,
			json_array<"b", Change>,
			json_array<"a", Change>
		>;
	};
}

int main (const int argC, char* argV[] ) {
	try {
		auto parsed = daw::json::from_json<DepthUpdateJson>(testJson);
	} catch (daw::json::json_exception e) {
		std::cout << "daw error: " << e.reason() << " near: '" << e.parse_location() << "'"  << std::endl;
	}
}

The code produces the output

daw error: A value of known size was accessed past the end near: '["34198.23000000","0.00000000"],["34198.25000000","0.00000000"],["34198.27000000","0.00000000"],["34198.30000000","0.00000000"],["34198.32000000","0.00958500"],["34198.40000000","0.01232200"],["34198.41000000","0.01000000"],["34198.87000000","0.00000000"],["34199.12000000","0.00000000"],["34199.16000000","0.00000000"],["34199.42000000","0.00000000"],["34200.25000000","0.00000000"],["34200.71000000","0.03199900"],["34201.27000000","0.03100000"],["34201.62000000","0.00000000"],["34202.58000000","0.00000000"],["34204.45000000","0.00952700"],["34207.64000000","0.00000000"],["34207.74000000","0.00000000"],["34209.77000000","0.00000000"],["34209.81000000","0.20400000"],["34225.94000000","0.20200000"],["34226.60000000","0.91050000"],["34236.08000000","0.30000000"]],"b":[["34198.31000000","0.00000000"],["34196.54000000","0.00453200"],["34193.34000000","0.00000000"],["34189.89000000","0.00000000"],["34188.82000000","0.00000000"],["34185.32000000","0.00000000"],["34184.84000000","0.06350200"],["34184.83000000","0.20000000"],["34180.61000000","0.08622700"],["34180.60000000","0.00000000"],["34180.59000000","0.19200000"],["34180.02000000","0.00000000"],["34180.01000000","0.00000000"],["34176.88000000","0.00000000"],["34166.48000000","0.00000000"],["34166.47000000","0.00000000"],["34159.85000000","0.03317500"],["34159.24000000","0.09394900"],["34158.29000000","1.00000000"],["34154.86000000","0.00000000"]]}'

Hi everyone,
Sorry i'm quite new with C++ and i can't figure out how to install this library

i've tried
add_library(daw STATIC) target_include_directories(daw PUBLIC ${PROJECT_SOURCE_DIR}/external/daw) add_library(third_party STATIC) target_include_directories(third_party PUBLIC ${PROJECT_SOURCE_DIR}/external/third_party) target_link_libraries(${LIBRARY_NAME} PUBLIC doctest daw third_party)

but in my main.cpp i get this error when "make"
fatal error: daw/json/daw_json_link.h: No such file or directory 29 | #include "daw/json/daw_json_link.h"

Does anyone have any idea how to install it easily?

JSON Benchmark

If you want to benchmark your code against more JSON libraies:
https://github.com/Loki-Astari/JsonBenchmark

See the section: https://github.com/Loki-Astari/JsonBenchmark#adding-a-new-json-library

This may be a per file or per member, but the interface should be examined

Hello,
thank for the library which works perfectly, i was wondering :
is there anyway to map struct member to json property?
example:
json { ts: 45435235 } to struct { dateCreated: 45435235 }

Add an optional bool data member in the json_data_contract specialization that indicates that the user would like to opt into outputting JSON to operator<<

The Readme mentions a difference between checked (from_json) and unchecked (from_json_unchecked) calls https://github.com/beached/daw_json_link#using-mapped-data-types . The latter don't seem to exist (anymore?). Instead the unchecked version should probably be daw::json::from_json<MyClass, daw::json::NoCommentSkippingPolicyUnchecked>, right?

When playing around with this, I was very surprised to find, that the "checked" version does not in fact check whether the input is valid json at all. Unused invalid numbers (even very blatantly wrong numbers {"a":1.0fsdf3, ...}) and especially unterminated json is not reported if all required keys have been found before the end of the string. (A behavior I expected for the trusted / "unchecked" version, but not for the "checked" version)

Is there any way to add these checks or to have a FullyChecked version or something like that, that checks whether the input is valid json?

Some context: When receiving json via websockets we occasionally receive faulty json strings that consist of a valid json appended to the incomplete beginning of another one (see example below). They are very obviously invalid - but relying on daw jsons parsing we might miss these issues and treat them as valid instead.

Below code does not throw an exception:

std::string test = R"({"e":"aggTrade","E":1610729466077,"s":"BTCUSDT","a":516119850,"p":"36062.5500{"e":"aggTrade","E":1610732218225,"s":"BTCUSDT","a":516218642,"p":"35943.45000000","q":"0.08800100","f":574701726,"l":574701727,"T":1610732218224,"m":false,"M":true})";
std::cout << daw::json::from_json<int, daw::json::NoCommentSkippingPolicyChecked>(test, "a") << std::endl;

Allow to call from_json with a second argument that is a preparsed version of the JSON Path string to speed up parsing

Show examples of dealing with Dictionary like types that have more than one value for a key

Hi, just want to make sure i am not doing anything wrong

versions it happens on: (release and dev branches)
c++20 with g++ 10.2.1

struct test_complex works.
struct test_complex1 works.
struct test_complex2 fails to compile.

struct test_complex {
    std::vector<std::vector<double>> a;
};
namespace daw::json {
    template<>
    struct json_data_contract<test_complex>
    {
        using type = json_member_list<
            json_array<"a", std::vector<std::vector<double>> >
        >;
    };
}

struct test_complex1 {
        std::vector<std::vector<std::vector<double>>> a;
};
namespace daw::json {
    template<>
    struct json_data_contract<test_complex1>
    {
        using type = json_member_list<
            json_array<"a", std::vector<std::vector<std::vector<double>>> >
        >;
    };
}
struct test_complex2 {
        std::optional<std::vector<std::vector<double>>> a;
};
namespace daw::json {
    template<>
    struct json_data_contract<test_complex2>
    {
        using type = json_member_list<
            json_array_null<"a", std::optional<std::vector<std::vector<double>>> >
        >;
    };
}

the error i get is

response.hpp:299:75: error: no type named ‘parse_to_t’ in ‘std::conditional<false, std::optional<std::vector<std::vector<double> > >, daw::json::missing_json_data_contract_for<std::optional<std::vector<std::vector<double> > > > >::type’ {aka ‘struct daw::json::missing_json_data_contract_for<std::optional<std::vector<std::vector<double> > > >’}
  299 |      json_array_null<"a", std::optional<std::vector<std::vector<double>>> >
      |                                                                           ^
response.hpp:299:75: error: template argument 3 is invalid
response.hpp:299:75: error: template argument 4 is invalid
response.hpp:300:2: error: template argument 1 is invalid
  300 |  >;
      |  ^

Any ideas?

Currently a dot path is supported for specifying members to start parsing with. A statically split path should yield better performance

Any plan to support CBOR(RFC 8949)?

I think that if daw_json_link support CBOR, daw_json_link will be best C++ serialization/reflection library!!!

Similar to Iterators, so that the overload specifies the minimum requirements and those with that or more can call it unless there is a more specific one available.

This will allow structured bindings with it

Currently the member selection in functions like from_json use a custom format that uses . as the member separator and [Idx] as array indices.

https://tools.ietf.org/html/rfc6901

Allow one to take a JSON document and transform values. An interface can look like, but not exactly,

json_transform( json_doc, out_it, {
  { "path.to.member", []( json_value const & jv ) { ... } }, 
  { "path.to.array_item[1]", []( int const & i ) { return 2*i; }} } );

Right now the error handling is a compile time option with either abort( ) or a json_exception being thrown. Make it a policy item so that the user can choose and mix.

Hi again,

when trying to build my code, i got a really verbose error that I can't past entirely, it told me first some about pedantic (which i disabled), then i got a much bigger one that i paste here:

/usr/include/c++/9/bits/shared_ptr.h:388:5: note: template argument deduction/substitution failed: In file included from /home/tom/Projects/cproject/build/_deps/daw_json_link-src/include/daw/json/impl/daw_json_parse_policy.h:12, from /home/tom/Projects/cproject/build/_deps/daw_json_link-src/include/daw/json/impl/daw_json_iterator_range.h:11, from /home/tom/Projects/cproject/build/_deps/daw_json_link-src/include/daw/json/daw_json_link.h:11, from /home/tom/Projects/cproject/app/main.cpp:29: /home/tom/Projects/cproject/build/_deps/daw_json_link-src/include/daw/json/impl/daw_json_parse_common.h:265:56: note: ‘const string_view’ {aka ‘const daw::basic_string_view<char>’} is not derived from ‘const std::shared_ptr<_Tp>’ 265 | inline constexpr bool is_no_name = ( JsonMember::name == no_name ); | ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~ In file included from /usr/include/c++/9/memory:81, from /usr/local/include/boost/system/detail/std_interoperability.hpp:12, from /usr/local/include/boost/system/error_code.hpp:963, from /usr/local/include/boost/beast/core/error.hpp:14, from /usr/local/include/boost/beast/core/detail/bind_handler.hpp:13, from /usr/local/include/boost/beast/core/bind_handler.hpp:14, from /usr/local/include/boost/beast/core/async_base.hpp:14, from /usr/local/include/boost/beast/core.hpp:15, from /home/tom/Projects/cproject/app/main.cpp:19: /usr/include/c++/9/bits/shared_ptr.h:393:5: note: candidate: ‘template<class _Tp> bool std::operator==(std::nullptr_t, const std::shared_ptr<_Tp>&)’ 393 | operator==(nullptr_t, const shared_ptr<_Tp>& __a) noexcept | ^~~~~~~~

any idea what is happening here?

(here is my code)
`
struct WeatherUpdate
{
string type; // e

};

namespace daw::json
{
template <>
struct json_data_contract
{
using type = json_member_list<
json_string<"e">>;
};
} `

https://github.com/jk-jeon/dragonbox
The benchmarks show better conversion times for the general (smaller number of digits) case compared to Ryu.

I'm in the process of assisting moving it to CMake (jk-jeon/dragonbox#4), so maybe this could be something to consider for v3.0

The name isn't reflective that this is used for converting to a JSON string and isn't compatible with std::to_string

Currently the daw_json_exception has an optional pointer to the position of the parser where the error occurred. Using the sax interface to the parsing, build a stack trace to the position. This may require using both class_start/first from Range as errors can occur that cause the position to be null.