vitineth / kaitai_struct_compiler

Kaitai Struct: compiler to translate .ksy => .cpp / .cs / .dot / .go / .java / .js / .lua / .nim / .php / .pm / .py / .rb

Kaitai Struct: compiler

This project is an official reference compiler for Kaitai Struct project.

Kaitai Struct is a declarative language used to describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc.

The main idea is that a particular format is described in Kaitai Struct language (.ksy files) only once and then can be compiled with this compiler into source files in one of the supported programming languages. These modules will include the generated code for a parser that can read described data structure from a file / stream and give access to it in a nice, easy-to-comprehend API.

Fork Status

This fork integrates a few tweak on the HTML renderer to make the documentation slightly more functional for a reference viewer. These are:

• Support for Markdown in any doc: component
  • Including tables
• Wrap data types in code blocks for increased readability
• Where possible, output field sizes so tables can be followed more easily
• Documents valid: blocks from the specification in the table for seq elements and types tables.

Further information

If you're looking for information on:

• Kaitai Struct language itself (.ksy files, general usage patterns) — refer to the user guide.
• How to download and install Kaitai Struct — see the downloads.
• How to build the compiler, run the test suite, and join the development — see the developer memo.

Licensing

Main code

Kaitai Struct compiler itself is copyright (C) 2015-2022 Kaitai Project.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

FastParse

Portions of Kaitai Struct compiler are loosely based on pythonparse from FastParse and are copyright (c) 2014 Li Haoyi ([email protected]).

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

XMLUtils code

Portions of Kaitai Struct compiler are based on scala/xml/Utility.scala from Scala XML.

Copyright (c) 2002-2017 EPFL
Copyright (c) 2011-2017 Lightbend, Inc.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of the EPFL nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Libraries used

Kaitai Struct compiler depends on the following libraries:

• scopt — MIT license
• fastparse — MIT license
• snakeyaml — Apache 2.0 license

---

Note that these clauses only apply only to compiler itself, not .ksy input files that one supplies in normal process of compilation, nor to compiler's output files — that consitutes normal usage process and you obviously keep copyright to both.

kaitai_struct_compiler's Issues

Render `if` conditions in HTML output

Currently conditionals are not represented at all in the spec. For this input

      - type: b5
        if: enable_dhcp == true
        doc: Used to skip 5 bits if enable_dhcp is set to true

      - id: program_default_gateway
        type: b1
        doc: Program Default gateway
        if: enable_dhcp == false

It produces

Which does not inform at all about the if. This needs some sort of representation but may need some thoughts on the best way to show this

Display validation specification as hex rather than binary

Currently the contents field for this data structure

  - id: first_property_address
    contents: [0x41, 0x72, 0x74, 0x2D, 0x4E, 0x65, 0x74, 0x00]

would produce

which, while correct, does not follow the usual standard use of hex within protocols.

I am proposing that we switch to a hex representation within the html output, with a possible extension flag enabling the user to adjust this if desired.

Add support for validation rules in html documentation

This only needs to be limited, even in the form of generated expressions however at minimum there should be support for structures like

valid:
    min: 1
    max: 4

which produces nice output. Potentially something at the bottom of the description that says

Validation Rule
0 <= port <= 4