evestera / json_typegen Goto Github PK

Hey, very useful crate!

The CLI works fine, but the current web version produces invalid code with missing generic params.

Eg:

struct Root {
  a: Vec,
  b: Option,
}

For my use-case, I need to be able to, on receiving an already existing type, create a type alias.

I've implemented it locally, and I'm going to open a PR. I'm not sure if this is functionality you want, and it's likely not up to the quality standards of the rest of the crate, but perhaps it works as a basis.

The exact issue can be seen on ritz078/transform#93

Add an option to do this

#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct RootInterface {
    some_object_with: String,
    a_lot_of_fields: String,
    which_quickly_becomes: String,
    hard_to_read: String,
}

instead of this

#[derive(Serialize, Deserialize)]
struct RootInterface {
    #[serde(rename = "someObjectWith")]
    some_object_with: String,
    #[serde(rename = "aLotOfFields")]
    a_lot_of_fields: String,
    #[serde(rename = "whichQuicklyBecomes")]
    which_quickly_becomes: String,
    #[serde(rename = "hardToRead")]
    hard_to_read: String,
}

Currently we generate "$schema": "http://json-schema.org/schema#". This is not a valid version of JSON Schema. Use a valid version, e.g. "$schema": "http://json-schema.org/draft-07/schema#".

When reading sample from file using the procedural macro (e.g. json_typegen!("Point", "point.json")), if the sample file changes the compiler will not be aware that anything relevant has changed and so will not recompile.

There is a hack we can do to let the compiler know we are using a file. Output code like this in addition to the actual code:

const _: &'static str = include_str!("point.json");

Tracking issue for better way to do this (from which the snippet above was taken): rust-lang/rust#73921

Hello!

First things first: thank you for the great library!

I am the author of schema_analysis, a crate that allows anyone to infer the schema of any self-describing format using Serde.
Our projects are somewhat similar, but with different focuses and strengths, which is why I have tried to integrate the two together.

As mentioned here, I would be interested in working out a way to more seamlessly access the internals of json_typegen_shared to link directly to your library instead of the somewhat ugly workaround of forking and adding pub to what I need.

To clarify the context: I currently expose access to the shape, options, and generation modules in a fork of json_typegen_shared. You can see here how I am using them.

If you were interested, I would also be happy to use directly a codegen function that takes a Shape directly, along the lines of this one. This would also reduce my imports to the shape and options modules.

In any case, thank you again for your OS work!

Best,
Quartz

E.g.

{
  "/hits/hits/-/_source": {
    "type_parameter": "T"
  }
}

or

{
  "/hits/hits/-/_source": {
    "use_type": "type_parameter"
  }
}

For typescript/typealias this is already possible with just adding <T> to the type name and using

{
  "/hits/hits/-/_source": {
    "use_type": "T"
  }
}

I tried building on my own but it keeps breaking. will you like to distribute such a build that can be directly consumed using JS?

I use json_typegen_wasm in my tauri App， it work fine in dev mode，when build it throw error

Uncaught (in promise) CompileError: WebAssembly.instantiateStreaming(): Refused to compile or instantiate WebAssembly module because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self'

Generating the type zero from the following sample:

{
  "one": {
    "two": {},
    "three": {}
  },
  "four": {}
}

Currently generates types in this order, that we then reverse.

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Two {}
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Three {}
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct One {
    two: Two,
    three: Three,
}

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Four {}
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Zero {
    one: One,
    four: Four,
}

Ideally, types should be output in the order they appear in the sample.

Kotlin does not allow empty data classes, so we should not generate code like this for inner object types:

data class People(
)

Looks like a great tool. I can go in and fix this by grabbing the source and changing it but thought I'd just note it here. Let me know if you want me to try to create a PR (a bit new at all this).

when trying to use cargo install json_typegen_cli

>cargo install json_typegen_cli
    Updating crates.io index
  Downloaded json_typegen_cli v0.4.0
  Downloaded 1 crate (14.9 KB) in 0.89s
  Installing json_typegen_cli v0.4.0
error: failed to compile `json_typegen_cli v0.4.0`, intermediate artifacts can be found at `C:\Users\jas\AppData\Local\Temp\cargo-installRFMGPW`

Caused by:
  failed to select a version for the requirement `clap = "~2.19.0"`
  candidate versions found which didn't match: 2.33.3, 2.33.2, 2.33.1, ...
  location searched: crates.io index
required by package `json_typegen_cli v0.4.0`

I have a 130 MB JSON file (from llvm-tblgen --dump-json) but unfortunately it just seems to freeze when I run it through json_typegen. I don't even get a progress bar (which I think the CLI is supposed to do?)

Any hints on how to debug this?

I've been using json_typegen_wasm and it's great! Noticed the npm package is a bit behind the repo. Any chance we could get a new release? It'd be awesome to have the recent additions, especially around pydantic.

Just add a minimal Gradle project to the CI folder, use the CLI to generate a Kotlin file that is written to that folder and run ./gradlew test testing roundtripping and such.

For JSON Schema, null and an optional field / undefined is not the same, so the currently generated schemas fail for JSON where explicit nulls are used instead of omitting optional fields.

Code like this should preferably always be legal:

json_typegen!("<typename>", <valid source>);
// ...
let foo: <typename> = <parsed JSON with same shape as source>;

Cases where that is currently not the case:

The source JSON is a collection. The typename will be used as the type of the elements of the collection. E.g. json_typegen!("Foobar", ... will require you to parse to Vec<Foobar>.

Code generation tries to be "helpful" and create a more suitable type name, e.g.

json_typegen!("point", r#"{ "x": 1, "y": 2 }"#); will create the type Point

json_typegen!("foo_bar", r#"{ "x": 1, "y": 2 }"#); will create the type FooBar

json_typegen!("Points", r#"[{ "x": 1, "y": 2 }]"#); will create the type Point, so you get a Vec<Point>, but will not define Points

For all cases where the requested type name is legal it should be possible to just reserve the name up front, and then add an alias. Just have to make sure to allow a named root type to use the requested type (e.g. type Point = Point1; struct Point1 { ... } would be a bit silly`).

Amazing work!

I noticed this behavior:

{
  "aaAaaAaa": 0,
  "bbBbbBbb": 0,
  "ccCccCcc": 0
}

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct ZalandoArticle {
    #[serde(rename = "aaAaaAaa")]
    aa_aaa_aaa: i64,
    #[serde(rename = "bbBbbBbb")]
    bb_bbb_bbb: i64,
    #[serde(rename = "ccCccCcc")]
    cc_ccc_ccc: i64,
}

This would be more friendly using rename_all instead:

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct ZalandoArticle {
    aa_aaa_aaa: i64,
    bb_bbb_bbb: i64,
    cc_ccc_ccc: i64,
}

The supported renames are "lowercase" (which you won't need, mostly for enums), "PascalCase", "camelCase", "snake_case" (also for enum variants), "SCREAMING_SNAKE_CASE", "kebab-case".

https://github.com/evestera/json_typegen/blob/master/json_typegen_shared/src/generation/kotlin.rs#L185

The wasm bundle used in json_typegen_web and the npm package is quite big, currently ~1.3 MB. A large portion of that is the regex crate. A quick test of removing it (and the Inflector crate, as it depends on it) reduced the bundle size to ~400 KB.

If it turns out to be too hard to completely remove, a quick test with regex = { version = "1.3", default-features = false, features = ["std"] } yielded a wasm bundle of ~800 KB. This solution would only require a simple fork of Inflector, to match versions and features of regex.

Places where regex is currently used:

• lib.rs handle_pub_in_name - Has been considered removed already
• generation/typescript.rs is_ts_identifier - Could easily be rewritten without regex
• generation/* Inflector::to_singular - Somewhat hard to replace, but it's already a slight issue that we don't have control over how field names are converted to type names.

Prompted by a reddit comment. The following JSON:

{ 
    "layouts": {
        "1900x200": "1",
        "1900*200": "2",
        "1900_200": "3"
    }
}

Results in renamed fields:

1900x200 -> n1900_x200
1900*200 -> n1900200
1900_200 -> n19002002

A suggested improvement:

1900x200 --> n1900x200
1900*200 --> n1900_200
1900_200 --> n1900_200_2

We should avoid creating multiple structurally identical types. E.g. for the following JSON:

{
  "pointA": {
    "x": 3,
    "y": 5
  },
  "pointB": {
    "x": 3,
    "y": 5
  }
}

We currently generate this:

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Rect {
    #[serde(rename = "pointA")]
    point_a: PointA,
    #[serde(rename = "pointB")]
    point_b: PointB,
}

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct PointB {
    x: i64,
    y: i64,
}

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct PointA {
    x: i64,
    y: i64,
}

But we want something like this:

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Rect {
    #[serde(rename = "pointA")]
    point_a: Point,
    #[serde(rename = "pointB")]
    point_b: Point,
}

#[derive(Default, Debug, Clone, Serialize, Deserialize)]
struct Point {
    x: i64,
    y: i64,
}

The form is complicated enough as is. Hide/disable options that are not relevant to an output mode

Awesome works! Thank you for creating this crates.

I notice this false behavior which only happen in typescript:

{
  "states": {
    "alphaGo": {
      "on": {
        "": "betaRust"
      }
    },
    "nightl'y update": {
      "on": {
        "every 4 weeks": "channel:beta"
      }
    },
    "b": {
      "on": {
        "ON": "initial D",
        "FALSE_ALARM": "c"
      }
    }
  }
}

interface Root {
    states: States;
}
interface States {
    alphaGo: AlphaGo;
    nightl'y update: NightlYUpdate;
    b: B;
}
interface AlphaGo {
    on: On;
}
interface On {
    "": string;
}
interface NightlYUpdate {
    on: On2;
}
interface On2 {
    every 4 weeks: string;
}
interface B {
    on: On3;
}
interface On3 {
    ON: string;
    FALSE_ALARM: string;
}

The empty string key parsed correctly but some keys that have white-space or quote were not surrounded with double-quote. At least the interface name is correct 🤔

Currently, JSON parsing is done with e.g. serde_json::de::from_reader(File::open(path)?) to a serde_json::Value. This is OK for small and medium-sized JSON documents, but does way too many allocations to be efficient when the JSON gets above a few hundred MBs (not a fault of serde, just the reality of parsing unknown JSON to memory). I see at least two alternatives here:

• Rewrite the inference code to work with streaming JSON data, rather than parsing the entire structure into memory at once. (Proof of concept written. Does the trick. Just a big refactoring to do in the actual crate.)
• Replace the use of serde_json::Value with a mostly compatible type that uses a string pool, as repeated object keys are a very large part of the memory used.

The output from the json_typegen CLI contains non-public types (and fields).
When saving it to a module, as shown in the Readme, these types are not exported and are therefore unusable.

This tool should add a pub modifier to all types and fields generated, possibly behing a flag.

In the docs for json_typegen_shared, I read the following:

If you want to use this crate directly, and thus care about its stability, please open an issue to let me know.

And here I am 🙂

I'm working on a project where in addition to a step to generate Rust types from JSON, I need to also generate the JSON itself. This lends itself to doing the generation in a build script so that I can orchestrate both steps, without having to bring in a Makefile or something to drive the CLI.

So far the json_typegen_shared::codegen interface has been suitable for my needs, so I don't have any requests or problems to report. Thank you for the great library!

Currently tuples are only used when all source arrays have the same length and combining the shapes of the array would lose type information. E.g. you can end up with [["a", 1], ["b", 2]] is inferred to Array<[string, number]>, but [[1, 2], [3, 4]] is inferred as number[][].

We already have the hint use_type: "map". We should similarly have use_type: "tuple" and with that make it possible to end up with types like [string, string] and [number, number, number?].

{"TRY": 10} parsing throws "found reserved keyword try"

Useful if you need to preserve all data on roundtrips. E.g. if you are fetching something, changing it and sending it back/somewhere else.

Jackson/Kotlin

    @JsonAnySetter
    @get:JsonAnyGetter
    val additionalFields: Map<String, Any> = mutableMapOf()

Rust/Serde

    #[serde(flatten)]
    additional_fields: HashMap<String, Value>,

Not relevant for TypeScript, but maybe for JSON Schema.