kaitai-io / kaitai_struct_visualizer

Kaitai Struct: visualizer and hex viewer tool

Kaitai Struct: visualizer

This is a console visualizer for the Kaitai Struct project.

Kaitai Struct is a declarative language used for describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc.

The main idea is that a particular format is described in Kaitai Struct language (.ksy file) and then can be compiled with ksc into source files in one of the supported programming languages. These modules will contain generated code for a parser that can read the described data structure from a file / stream and provide access to it using a nice, easy-to-understand API.

See the Kaitai Struct homepage for details on .ksy files and general usage patterns.

Downloading and installing

Requirements

• ksc — kaitai-struct-compiler
• Java
• Ruby (latest Ruby 3.x recommended, at least Ruby 2.4 required)

From RubyGems repository

Kaitai Struct visualizer is written in Ruby and is available on RubyGems. Thus, you'll need Ruby installed on your box and then you can just run:

gem install kaitai-struct-visualizer

---

You can use ksv --version to check what versions of ksv and its dependencies are installed, for example:

$ ksv --version
kaitai-struct-visualizer 0.7
kaitai-struct-compiler 0.11
kaitai-struct 0.11 (Kaitai Struct runtime library for Ruby)

The versions of kaitai-struct-compiler and kaitai-struct should match. If not, see https://kaitai.io/#download for instructions how to install the latest version of kaitai-struct-compiler and/or use gem update kaitai-struct to update the kaitai-struct gem if needed.

Source code

If you're interested in developing the visualizer itself, you can check out the source code from the kaitai_struct_visualizer GitHub repository:

git clone https://github.com/kaitai-io/kaitai_struct_visualizer.git

Then run bundle install to install dependencies. After that, you can run bin/ksv or bin/ksdump right away (without having to install the kaitai-struct-visualizer gem first), which makes development easier.

Usage

There are two executables provided by this package:

• ksv — interactive console visualizer with GUI
• ksdump — command-line tool for dumping parsed data in JSON, XML or YAML format to standard output (stdout)

The basic usage is similar for both programs:

ksv <binary-file> <ksy-file>... | <rb-file>

For ksdump, it may be useful to change the output format with the -f option (the default is yaml) and redirect the output to a file so that your terminal is not flooded with thousands of lines (for larger input files):

ksdump -f json <binary-file> <ksy-file>... > output.json

Running with Docker

This project is also available via the kaitai/ksv image on Docker Hub. The default entrypoint is ksv (the interactive visualizer):

docker run --rm -it -v "$(pwd):/share" kaitai/ksv <binary-file> <ksy-file>

You can specify ksdump as the entrypoint like this (and note that we don't need the -it flags anymore because ksdump is not interactive — omitting them in fact allows you to distinguish the ksdump's output to stdout and stderr, see this comment):

docker run --rm -v "$(pwd):/share" --entrypoint ksdump kaitai/ksv -f json <binary-file> <ksy-file> > output.json

---

Building the Docker image locally:

docker build . --tag docker.io/kaitai/ksv

Licensing

Kaitai Struct visualizer is copyright (C) 2015-2023 Kaitai Project.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

---

Note that it applies only to visualizer itself, not .ksy input files that one supplies in normal process of compilation, nor to compiler's output files — that constitutes normal usage process and you obviously keep copyright to both.

kaitai_struct_visualizer's Issues

Jump to end of array

This is similar to #12 but it would be great to just be able to jump to the end of some long array. With how slow scrolling is, it's currently very painful. Perhaps using the vim shortcuts of "G" and "gg"

Win32API is not in the ruby standard library since version 1.9.2

According to the developer of win32-api, The current Win32API library that ships with the standard library has been slated for removal from Ruby 2.0. I installed the latest version of ruby (3.0) to run ksv. Running ksv I get this error: <internal:C:/Program Files/Ruby/Ruby30-x64/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in 'require': cannot load such file -- Win32API (LoadError) . It is a requires of console_windows.rb . We can use Fiddle instead.

Instances do not highlight types as groups of values

Types used in seq will display the type, when selected by itself, and when components are highlighted, as a group. That is, it'll highlight the entire type structure, with the particular thing you have selected being highlighted brighter than everything else. Instances do not do this.

Is this a bug, or just a currently NI feature?

Also semi related: selecting the type itself (and not any of the values/arrays contained within) does not highlight the entire type. This also happens with seq stuff.

On-screen help (on F1, `?` or something) is needed to be displayed

There is quite a few misunderstandings on keybindings. There should be some way to show them all.

Why is Rake declared as a development dependency?

Cc @GreyCat

See .gemspec:

kaitai_struct_visualizer/kaitai-struct-visualizer.gemspec

Line 34 in 4e273ad

s.add_development_dependency 'rake', '~> 12.3.3'

When I just run rake in the console, it fails because there is no "rakefile":

$ rake
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)

(See full trace by running task with --trace)

I'm not familiar with Rake at all, I just have a hunch that it's like writing Makefiles in Ruby (and I saw a few gems that were using a Rakefile to build native extensions, for example). But I think we don't use Rake in any way here?

unable to run

$ ksv 1194614.gif gif.ksy

error message

/eta: unknown key found, expected: enums, instances, meta, seq, types
Compilation OK
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': no implicit conversion of nil into String (TypeError)
	from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:31:in `block in compile_format'
	from /usr/lib/ruby/2.3.0/tmpdir.rb:89:in `mktmpdir'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:10:in `compile_format'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer.rb:18:in `initialize'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `new'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `<top (required)>'
	from /usr/local/bin/ksv:23:in `load'
	from /usr/local/bin/ksv:23:in `<main>'

Find out what Ruby versions ksv is compatible with on Windows

Ruby installations come from https://rubyinstaller.org/downloads/, everything is run in Windows Sandbox. kaitai-struct-visualizer-0.7.gem was built manually from #49 by using gem build kaitai-struct-visualizer.gemspec with the humdogm:master branch checked out.

Ruby 1.8

C:\Users\WDAGUtilityAccount>ruby --version
ruby 1.8.7 (2013-06-27 patchlevel 374) [i386-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
1.8.24

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Fetching: kaitai-struct-0.10.gem (100%)
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
2 gems installed
Installing ri documentation for kaitai-struct-0.10...
Installing ri documentation for kaitai-struct-visualizer-0.7...
Installing RDoc documentation for kaitai-struct-0.10...
Installing RDoc documentation for kaitai-struct-visualizer-0.7...

C:\Users\WDAGUtilityAccount>ksv --help
C:/Ruby187/lib/ruby/gems/1.8/gems/kaitai-struct-visualizer-0.7/bin/ksv:9: undefined method `realpath' for File:Class (NoMethodError)
        from C:/Ruby187/bin/ksv:23:in `load'
        from C:/Ruby187/bin/ksv:23

kaitai_struct_visualizer/bin/ksv

Line 9 in 5b3ec3d

full_bin_path = File.realpath($PROGRAM_NAME)

Ruby 1.9

C:\Users\WDAGUtilityAccount>ruby --version
ruby 1.9.3p551 (2014-11-13) [i386-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
1.8.29

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Fetching: kaitai-struct-0.10.gem (100%)
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
2 gems installed
Installing ri documentation for kaitai-struct-0.10...
Installing ri documentation for kaitai-struct-visualizer-0.7...

RDoc::Parser::Ruby failure around line 16 of
lib/kaitai/tui.rb

Before reporting this, could you check that the file you're documenting
has proper syntax:

  C:/Ruby193/bin/ruby.exe -c lib/kaitai/tui.rb

RDoc is not a full Ruby parser and will fail when fed invalid ruby programs.

The internal error was:

        (RDoc::Error) unknown type of %string "i"

ERROR:  While generating documentation for kaitai-struct-visualizer-0.7
... MESSAGE:   unknown type of %string "i"
... RDOC args: --ri --op C:/Ruby193/lib/ruby/gems/1.9.1/doc/kaitai-struct-visualizer-0.7/ri lib --title kaitai-struct-visualizer-0.7 Documentation --quiet

C:\Users\WDAGUtilityAccount>ruby -c ksv-src\lib\kaitai\tui.rb
ksv-src/lib/kaitai/tui.rb:16: unknown type of %string
        @highlight_colors = %i[white aqua blue green white]
                               ^
ksv-src/lib/kaitai/tui.rb:16: syntax error, unexpected $end
        @highlight_colors = %i[white aqua blue green white]
                               ^

kaitai_struct_visualizer/lib/kaitai/tui.rb

Line 16 in 5b3ec3d

@highlight_colors = %i[white aqua blue green white]

Ruby 2.0

C:\Users\WDAGUtilityAccount>ruby --version
ruby 2.0.0p648 (2015-12-16) [i386-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
2.0.14.1

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: kaitai-struct-visualizer requires kaitai-struct (~> 0.4)

C:\Users\WDAGUtilityAccount>gem install kaitai-struct
ERROR:  Could not find a valid gem 'kaitai-struct' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.org/latest_specs.4.8.gz)

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-0.10.gem
Successfully installed kaitai-struct-0.10
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
1 gem installed

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
WARNING:  Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.org/specs.4.8.gz)
1 gem installed

C:\Users\WDAGUtilityAccount>ksv --help
C:/Ruby200/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require': C:/Ruby200/lib/ruby/gems/2.0.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/hex_viewer.rb:128: syntax error, unexpected '.' (SyntaxError)
          @tree&.do_exit
                 ^
	from C:/Ruby200/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from C:/Ruby200/lib/ruby/gems/2.0.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:7:in `<top (required)>'
	from C:/Ruby200/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from C:/Ruby200/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from C:/Ruby200/lib/ruby/gems/2.0.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/parser.rb:5:in `<top (required)>'
	...

C:\Users\WDAGUtilityAccount>ruby -c C:/Ruby200/lib/ruby/gems/2.0.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/hex_viewer.rb
C:/Ruby200/lib/ruby/gems/2.0.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/hex_viewer.rb:128: syntax error, unexpected '.'
          @tree&.do_exit
                 ^

kaitai_struct_visualizer/lib/kaitai/struct/visualizer/hex_viewer.rb

Line 128 in 5b3ec3d

@tree&.do_exit

https://stackoverflow.com/a/36812667/12940655:

It is called the Safe Navigation Operator. Introduced in Ruby 2.3.0, it lets you call methods on objects without worrying that the object may be nil(Avoiding an undefined method for nil:NilClass error), similar to the try method in Rails.

Ruby 2.3

C:\Users\WDAGUtilityAccount>ruby --version
ruby 2.3.3p222 (2016-11-21 revision 56859) [x64-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
2.5.2

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
Done installing documentation for kaitai-struct, kaitai-struct-visualizer after 0 seconds
WARNING:  Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
2 gems installed

C:\Users\WDAGUtilityAccount>ksv --help
Usage: ksv [options] <file_to_parse.bin> <format.ksy>...|<format.rb>

    -I, --import-path [DIRECTORIES]  .ksy library search path(s) for imports (see also KSPATH env variable)
        --opaque-types [BOOLEAN]     opaque types allowed, default: false
    -r, --require [PATH]             load ("require") .rb file into Ruby process
        --version                    show versions of ksv and ksc

C:\Users\WDAGUtilityAccount>ksv example.cdr.unpk cdr_unpk.ksy
Compilation OK
... processing cdr_unpk.ksy 0
...... loading file_streams.rb
...... loading coreldraw_cdr.rb
...... loading cdr_unpk.rb
Classes loaded OK, main class = CdrUnpk
C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/console_windows.rb:97:in `goto': undefined method `unpack1' for "\x00\x00\x00\x00":String (NoMethodError)
Did you mean?  unpack
	from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/console_windows.rb:92:in `clear'
	from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:163:in `block in redraw'
	from C:/Ruby23-x64/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
	from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:162:in `redraw'
	from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/visualizer.rb:20:in `run'
	from C:/Ruby23-x64/lib/ruby/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/bin/ksv:59:in `<top (required)>'
	from C:/Ruby23-x64/bin/ksv:22:in `load'
	from C:/Ruby23-x64/bin/ksv:22:in `<main>'

I was actually trying out #49, so line 97 is the following:

kaitai_struct_visualizer/lib/kaitai/console_windows.rb

Line 97 in 2e19269

coord = [x, y].pack('vv').unpack1('V')

And I already know that String#unpack1 was introduced in Ruby 2.4: kaitai-io/kaitai_struct_ruby_runtime#8

Ruby 2.4

C:\Users\WDAGUtilityAccount>ruby --version
ruby 2.4.10p364 (2020-03-31 revision 67879) [x64-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
2.6.14.4

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
Done installing documentation for kaitai-struct, kaitai-struct-visualizer after 0 seconds
2 gems installed

C:\Users\WDAGUtilityAccount>ksv --help
Usage: ksv [options] <file_to_parse.bin> <format.ksy>...|<format.rb>

    -I, --import-path [DIRECTORIES]  .ksy library search path(s) for imports (see also KSPATH env variable)
        --opaque-types [BOOLEAN]     opaque types allowed, default: false
    -r, --require [PATH]             load ("require") .rb file into Ruby process
        --version                    show versions of ksv and ksc

C:\Users\WDAGUtilityAccount>ksv 8f.bin hello_world.ksy

It seems to work fine (with changes from #49), the hex viewer finally shows up.

meta:
  id: hello_world
seq:
  - id: one
    type: u1

[-] [root]                                 00000000: 8f                                              | .
  [.] one = 143

Ruby 2.7

C:\Users\WDAGUtilityAccount>ruby --version
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x64-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
3.1.6

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
Done installing documentation for kaitai-struct, kaitai-struct-visualizer after 0 seconds
2 gems installed

C:\Users\WDAGUtilityAccount>ksv --help
Usage: ksv [options] <file_to_parse.bin> <format.ksy>...|<format.rb>

    -I, --import-path [DIRECTORIES]  .ksy library search path(s) for imports (see also KSPATH env variable)
        --opaque-types [BOOLEAN]     opaque types allowed, default: false
    -r, --require [PATH]             load ("require") .rb file into Ruby process
        --version                    show versions of ksv and ksc

C:\Users\WDAGUtilityAccount>ksv 8f.bin hello_world.ksy

Works the same as Ruby 2.4.

Ruby 3.0

For Ruby 3.0+ support on Windows (since the version from RubyGems fails with in `require': cannot load such file -- Win32API (LoadError) on line lib/kaitai/console_windows.rb:2 since Ruby 3 - see #48), I checked out #49 and built kaitai-struct-visualizer-0.7.gem from sources using gem build kaitai-struct-visualizer.gemspec.

C:\Users\WDAGUtilityAccount>ruby --version
ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x64-mingw32]

C:\Users\WDAGUtilityAccount>gem --version
3.2.33

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
Done installing documentation for kaitai-struct, kaitai-struct-visualizer after 0 seconds
2 gems installed

C:\Users\WDAGUtilityAccount>ksv 8f.bin hello_world.ksy

Works, same as Ruby 2.4 and 2.7.

Ruby 3.1

C:\Users\WDAGUtilityAccount>ruby --version
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x64-mingw-ucrt]

C:\Users\WDAGUtilityAccount>gem --version
3.3.7

C:\Users\WDAGUtilityAccount>gem install kaitai-struct-visualizer-0.7.gem
Fetching kaitai-struct-0.10.gem
Successfully installed kaitai-struct-0.10
Successfully installed kaitai-struct-visualizer-0.7
Parsing documentation for kaitai-struct-0.10
Installing ri documentation for kaitai-struct-0.10
Parsing documentation for kaitai-struct-visualizer-0.7
Installing ri documentation for kaitai-struct-visualizer-0.7
Done installing documentation for kaitai-struct, kaitai-struct-visualizer after 0 seconds
2 gems installed

C:\Users\WDAGUtilityAccount>ksv 8f.bin hello_world.ksy

Same as Ruby 2.4, 2.7 and 3.0.

key pressing error

ubuntu 12.04

 /usr/lib/ruby/gems/1.9.1/gems/kaitai-struct-visualizer-0.5/lib/kaitai/console_ansi.rb:94:in `read_char': undefined method `cooked!' for #<IO:<STDIN>> (NoMethodError)

[Bug] Broken cyrillic input after exit.

Notes

System: Linux x86_64, vte 0.40.2, TERM is xterm-256color

Steps to reproduce

• Start terminal.
• Type in some cyrillics to verify that input is correct.
• Run ksv with some file and corresponding template. Running without parameters doesn't trigger the bug.
• Exit ksv.
• Type in more cyrillics.

To fix

Aforementioned behaviour can be fixed by executing stty -brkint -istrip -echonl after running ksv.
Correct way to resolve this probably is saving terminal settings (e.g. output of system('stty -g')) on init and restoring them on exit or exception.

Fails to start

Can't get the visualizer to start. Tried several versions of ruby, no success... Here is the console output :

/home/dave/.rvm/rubies/ruby-2.4.0/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': no implicit conversion of nil into String (TypeError)
	from /home/dave/.rvm/rubies/ruby-2.4.0/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /home/dave/.rvm/gems/ruby-2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:31:in `block in compile_format'
	from /home/dave/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/tmpdir.rb:89:in `mktmpdir'
	from /home/dave/.rvm/gems/ruby-2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:10:in `compile_format'
	from /home/dave/.rvm/gems/ruby-2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer.rb:18:in `initialize'
	from /home/dave/.rvm/gems/ruby-2.4.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `new'
	from /home/dave/.rvm/gems/ruby-2.4.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `<top (required)>'
	from /home/dave/.rvm/gems/ruby-2.4.0/bin/ksv:22:in `load'
	from /home/dave/.rvm/gems/ruby-2.4.0/bin/ksv:22:in `<main>'
	from /home/dave/.rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in `eval'
	from /home/dave/.rvm/gems/ruby-2.4.0/bin/ruby_executable_hooks:15:in `<main>'

Suggestion: Configure GitHub RuboCop Action

Please configure the GitHub RuboCop Action.
Reference: https://github.com/marketplace/actions/github-rubocop-action

PS:
With this change, there will be a need to have a strict guideline for new code to run RuboCop locally first before committing the changes.

Regards!

Windows: command injection via the input `.ksy` file name

Recently (at the time of writing in April 2024) there has been a report about improper escaping of command-line arguments on Windows when executing batch files affecting many programming languages: https://flatt.tech/research/posts/batbadbut-you-cant-securely-execute-commands-on-windows/

Most notably, Rust released a patch version 1.77.2 just for the fix of this security issue and published a security advisory describing the problem. For reference, here's the diff between Rust 1.77.1 and 1.77.2, which shows how they fixed it: rust-lang/rust@1.77.1...1.77.2

But it's not specific to Rust. It's more of a general problem on Windows that is not very well known. The problem is that batch files (.bat or .cmd) are executed via cmd.exe, which has different escaping rules than most programs, and this distinction is apparently often overlooked by standard library implementations in many programming languages. As a result, executing batch files via these standard APIs from the particular programming language and passing arguments originating from user input has been found to be often unsafe, because these APIs will fail to escape some special characters properly for cmd.exe and thus there's a good chance that the user can achieve command injection.

I did a quick test to check if KSV is affected (because I knew that kaitai-struct-compiler uses a .bat launcher on Windows). For testing purposes, I added a debug print of ARGV here first, so that it's clear that the improper escaping we'll see really occurs inside KSV, not in the shell:

--- i/bin/ksdump
+++ w/bin/ksdump
@@ -70,6 +70,8 @@ if ARGV.size < 2
   exit 1
 end

+pp ARGV
+
 c = Kaitai::Struct::Visualizer::KSYCompiler.new(options)
 app = Kaitai::Struct::Visualizer::Parser.new(c, ARGV[0], ARGV[1..-1], options)
 exc = app.load

We can also add some more debug prints in ksy_compiler.rb:

1. before the invocation of KSC

   --- i/lib/kaitai/struct/visualizer/ksy_compiler.rb
   +++ w/lib/kaitai/struct/visualizer/ksy_compiler.rb
   @@ -110,6 +111,7 @@ module Kaitai::Struct::Visualizer
          # on Windows.
          args.unshift('--') unless Kaitai::TUI.windows?
   
   +      pp args
          log_str, err_str, status = Open3.capture3('kaitai-struct-compiler', *args)
          unless status.success?
            if status.exitstatus == 127

2. after the invocation of KSC to print the JSON output

   --- i/lib/kaitai/struct/visualizer/ksy_compiler.rb
   +++ w/lib/kaitai/struct/visualizer/ksy_compiler.rb
   @@ -83,6 +83,7 @@ module Kaitai::Struct::Visualizer
        # @return [String] Main class name, or nil if were errors
        def compile_and_load(fns, code_dir)
          log = compile_formats_to_output(fns, code_dir)
   +      pp log
          load_ruby_files(fns, code_dir, log)
        end

---

If you run this in a cmd.exe session (obviously, you need ruby and kaitai-struct-compiler in PATH), the Calculator app will appear, so we indeed have command injection:

C:\temp\kaitai_struct\visualizer>ruby bin\ksdump sample.bin 'hello" & calc.exe .ksy'
["sample.bin", "hello\" & calc.exe .ksy"]
["--ksc-json-output", "--debug", "-t", "ruby", "-d", "C:/Users/pp/AppData/Local/Temp/d20240411-20136-nb8x10", "hello\" & calc.exe .ksy"]
{"hello\""=>{"errors"=>[{"file"=>"hello\"", "path"=>[], "message"=>"hello\" (The filename, directory name, or volume label syntax is incorrect)"}]}}
C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:151:in `block in load_ruby_files': undefined method `[]' for nil:NilClass (NoMethodError)

        if log_fn['errors']
                 ^^^^^^^^^^
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:149:in `each'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:149:in `each_with_index'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:149:in `load_ruby_files'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:87:in `compile_and_load'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:63:in `block in compile_formats'
        from C:/Ruby32-x64/lib/ruby/gems/3.2.0/gems/tmpdir-0.2.0/lib/tmpdir.rb:99:in `mktmpdir'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:63:in `compile_formats'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:35:in `compile_formats_if'
        from C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/parser.rb:25:in `load'
        from bin/ksdump:77:in `<main>'

Note that KSC clearly only received hello" as the file name - the & calc.exe .ksy part was interpreted by cmd.exe, even though the pp args debug print proves that the Open3.capture3 Ruby function was called with expected arguments.

If we invoke KSC directly, no command injection occurs (so the problem is not in the .bat launcher, for example):

C:\Users\WDAGUtilityAccount>kaitai-struct-compiler.bat --ksc-json-output --debug -t ruby "hello"" & calc.exe .ksy"
{"hello\" & calc.exe .ksy": {"errors": [{"file": "hello\" & calc.exe .ksy","path": [],"message": "hello\" & calc.exe .ksy (The filename, directory name, or volume label syntax is incorrect)"}]}}

After checking the Ruby docs, it seems that Open3.capture3 simply isn't designed to handle any escaping or sanitization at all. Following the links in the documentation starting from Open3.capture3, we get to Kernel#system, which has even a warning about command injection:

1. Open3.capture3:

   The arguments env, cmd and opts are passed to Open3.popen3 except (...)

2. Open3.popen3:

   The parameters env, cmd, and opts are passed to Process.spawn.

3. Process.spawn:

   This method is similar to Kernel#system but it doesn’t wait for the command to finish.

4. Kernel#system:

   This method has potential security vulnerabilities if called with untrusted input; see Command Injection.

So we should either use a different function (I don't know if there is a safe alternative with escaping in the Ruby standard library), or do the escaping/sanitization ourselves.

How to close ksv on Windows?

This might sound stupid, but how do I close ksv? I tried STRG+C, +D, +Z an it's always just repainting. The only way to close it I found was the "X" of the shell Window. The problem with this is that one looses the commandline to open ksv and it's a bit annoying to always need to retype paths.

Not working with latest kaitai struct ver for windows.

Make visualizer more compatible with long value names on Windows.

I have the following KSY:

seq:
  - id:       control_info_check_hdr_none
    contents: [0x78]
    if:       is_header_none
    doc:      TODO docs

leading to the following error message:

[.] control_info_check_hdr_short = C:/Program Files (x86)/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/node.rb:114:in draw': undefined method +' for nil:NilClass (NoMethodError)
from C:/Program Files (x86)/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/tree.rb:190:in `block in draw_rec'

I can workaround the problem simply be shortening the name. But 1. it took me a while to understand the problem, because I have longer (instance) names in the same type without this problem, and 2. I think the visualizer could do this on it's own as well. It could either simply reduce the whole string it wanted to show to the absolute maximum so that the ... can still be added or reduce the name a bit and add a bit of the value.

[.] control_info_check_hdr_sh...
vs.
[.] control_info_c... = 78

Would only be great if it wouldn't crash. :-) Thanks!

Support for creating a new console/shell to render text in?

The problem.

Running the viewer within an already existing console or shell is easy and text is simply rendered within that. But I'm running the viewer using launch configs of Eclipse as well and in that scenario things either get output to the console provided by Eclipse or one needs to take care of creating an additional console manually.

The problem with the Eclipse-console.

The problem with the console of Eclipse seems to be that while it is able to print STDOUT, STDERR etc., it doesn't seem to support real textual UIs like used by ksv. After executing a launch config only calling ruby itself directly, some debugging output gets printed and the process seem to halt afterwards. It doesn't terminate as well, it simply doesn't seem to do anything anymore, which can be seen at the red terminate-button on the right.

The problem with creating a custom console in the launch config.

While creating an additional own console within the launch config is possible, things get complicated if one wants do use that launch config under Linux and Windows at the same time, because both use different default shells. What I'm currently using on Windows is the following:

That starts a new cmd.exe to actually only start another cmd.exe to start the process with all its arguments in a new console. Two cmd.exe were necessary to forward all individual args properly. Additionally, ComSpec maps to cmd.exe on Windows, but is not known on Linux, while SHELL is known on Linux but not on Windows and stuff like that.

Some ideas for workarounds.

1. Two launch-configs of course, but especially because the arguments, settings for environment variables etc. are OS-independent already, would be great if one could avoid this.

2. Would be great if one could tell ruby.exe itself to create a console. But didn't find any arguments in this direction.

3. Issuing a system call using ruby -e '', because system maps to local shells mostly. The problem is that when using paths like I have in my Eclipse launch config, that those sometimes map to \ and sometimes to /, with the former introducing syntax errors for the system call. Those would need to be \\ in code.

4. Add creating a new console in console_*.rb, but I don't know if that is possible at all in non-Windows. Windows supports the following calls to create a new console:

   def initialize
   Win32API.new('kernel32', 'FreeConsole', []).call()
   Win32API.new('kernel32', 'AllocConsole', []).call()

But rendering output to that new console didn't work properly yet, I guess because of problems with what STDIN/STDOUT is when.

5. Using -r to load some text actually creating a console only, which makes things easier than to use -e and a long system-call.

6. Adding a new argument like --new-console to recall the app using system with all given arguments, but without --new-console. Pretty similar to --version is handled currently, but system seems to only create a shell in case of a command line and that to build might be OS-dependent again because of how cmd.exe handles quotes and stuff. Didn't get a quick PoC to work as well:

   opts.on("--new-console [BOOLEAN]", "Recall with a newly created console. MUST BE FIRST ARG!") do |v|
   if v
   system($0.prepend('"').concat('" "').concat(ARGV.join('" "')).concat('"'))
   exit 0
   end
   end

Any other ideas?

Avoid reading whole file

Code generated by KS does a fairly good job at not reading the whole file in memory and even doing stuff lazily where possible, but HexViewer component of ksv currently works with a byte array buffer, which is always filled up like that:

  def hv_update_io
    io = @cur_node.io
    if io != @cur_io
      @cur_io = io
      io.seek(0)
      buf = io.read_bytes_full
      @hv.buf = buf
    end
  end

So, effectively it means that huge files would be read fully into memory before doing any visualizations at all. Partially, it depends on kaitai-io/kaitai_struct#44, but even without that probably we could do a IO-based instead of preread byte array based HexViewer.

Screenshots

Would be nice to see how it looks like before installing.

Support forward slashes (not just backslashes) in .ksy paths on Windows

Right now, it's effectively only possible to use backslashes (C:\path\to\spec.ksy) to specify the full path to a .ksy file on Windows, which is quite restrictive - basically all programs on Windows accept paths with forward slashes, so you get used to being able to use them.

I also don't think that the Windows users who tried to use forward slash .ksy paths and got the following cryptic error because of it:

C:/temp/kaitai_struct/visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb:84:in `block (2 levels) in compile_formats':
	undefined method `[]' for nil:NilClass (NoMethodError)

...had any clue what's going on and that it's caused by the use of forward slashes - this just typically doesn't occur to you when you can use forward slashes pretty much everywhere. So they would probably conclude that ksv and ksdump can't be used on Windows and maybe resort to installing it in WSL.

This bug is caused by the fact that the compiler's --ksc-json-output uses backslash paths as JSON keys on Windows, so when KSV tries to use the forward slash path (directly from the CLI argument provided by the user) as the key on this line:

kaitai_struct_visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb

Line 83 in 77252f5

log_fn = log[fn]

..., it can't find anything and crashes.

The solution is to convert the path to use backslashes and also try using it as the key (in case the direct use doesn't find anything).

Feature Request: __str__ and __repr__ for python generated classes

the methods are equivalent to java's toString method, having them generated by kaitai would greatly ease the debugging.

First time running ksv on a simple fmt breaks?

4046010217 (of class java.lang.Long)
Compilation OK
C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': no implicit conversion of nil into String (TypeError)
        from C:/Ruby24-x64/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:31:in `block in compile_format'
        from C:/Ruby24-x64/lib/ruby/2.4.0/tmpdir.rb:89:in `mktmpdir'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer_main.rb:10:in `compile_format'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer.rb:18:in `initialize'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `new'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `<top (required)>'
        from C:/Ruby24-x64/bin/ksv:22:in `load'
        from C:/Ruby24-x64/bin/ksv:22:in `<main>'

meta:
  id: fmt
  application: fmt
  endian: le
seq:
  - id: magic
    contents: [0xdeadbeef]
  - id: static
    contents: [0x56000000, 0x00000000, 0x33000000, 0xdeadcode]
  - id: BaseChunk
    type: u4

Am I missing something?

Crashes upon user input

https://asciinema.org/a/MF2m6DMT5UoLNdSxo7QVnKYxp (crash occurs after pressing <enter>)

  [.] magic = Traceback (most recent call last):
	16: from /usr/local/bin/ksv:23:in `<main>'
	15: from /usr/local/bin/ksv:23:in `load'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/bin/ksv:53:in `<top (required)>'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/visualizer.rb:21:in `run'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:39:in `run'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:39:in `loop'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:40:in `block in run'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:165:in `redraw'
	 8: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:168:in `block in redraw'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:196:in `draw_rec'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:196:in `each'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:197:in `block in draw_rec'
	 3: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:188:in `draw_rec'
	 2: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
	 1: from /usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/tree.rb:190:in `block in draw_rec'
/usr/local/lib/ruby/gems/2.5.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/node.rb:114:in `draw': undefined method `+' for nil:NilClass (NoMethodError)

▶ ruby --version
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin17]

~
▶ ksv --version
kaitai-struct-visualizer 0.7

~
▶ kaitai-struct-compiler --version
kaitai-struct-compiler 0.8

Go to array entry by index

It would be very convenient to be able to position the cursor at array entry with specified index, especially when your array is large enough. How I see it: an user types a number which displayed at bottom and then presses Enter or G or any other key which sets a new position.

Add ability to view local offsets

It would be nice to be able to view "local offsets" in a repeat block. Take for example, the following YAML markup:

meta:
  id: scp
  application: TiTS Engine - Sprite Format
  endian: le
seq:
  - id: header
    type: header
  - id: frame_entries
    type: frame_entries
types:
  header:
    seq:
      - id: frame_count
        type: u1
      - id: magic_zero
        contents: [0]
  frame_entries:
    seq:
      - id: frame_entry
        repeat: eos

        # The pages are always 512 byte blocks / pages
        # for some reason... not sure why considering a lot of
        # the data is wasted...
        size: 512

It would be nice when examing the data to have "local offsets" toggalable, this would allow me to quickly deal with offsets without needing to do subtraction. :) For example, for each frame_entry

[...]kaitai/struct/struct.rb:314:in `ensure_fixed_contents': wrong number of arguments (given 1, expected 2) (ArgumentError)

I tried the zip KSY of your formats repo with some ordinary zip and got the following error:

C:\Users\tschoening>ksv "Desktop\zip.zip" "Desktop\zip.ksy"
Compilation OK
Class loaded OK
C:/Users/tschoening/.gem/ruby/2.3.0/gems/kaitai-struct-0.5/lib/kaitai/struct/struct.rb:314:in `ensure_fixed_contents': wrong number of arguments (given 1, expected 2) (ArgumentError)
		from C:/Users/tschoening/AppData/Local/Temp/d20161124-10844-nvzwha/zip.rb:167:in `_read'
		from C:/Users/tschoening/AppData/Local/Temp/d20161124-10844-nvzwha/zip.rb:39:in `_read'
		from C:/Users/tschoening/.gem/ruby/2.3.0/gems/kaitai-struct-visualizer-0.5/lib/kaitai/struct/visualizer/visualizer.rb:29:in `initialize'
		from C:/Users/tschoening/.gem/ruby/2.3.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `new'
		from C:/Users/tschoening/.gem/ruby/2.3.0/gems/kaitai-struct-visualizer-0.5/bin/ksv:19:in `<top (required)>'
		from C:/Users/tschoening/.gem/ruby/2.3.0/bin/ksv:23:in `load'
		from C:/Users/tschoening/.gem/ruby/2.3.0/bin/ksv:23:in `<main>'

I'm pretty sure the Zip itself is valid and I tried different ones as well, all with the same result.

Additionally:

Is there some arg to tell ksv to not delete the temporarily compiled classes? If not, interested to add one?

Currently those temp folders seem to be deleted on program exit, which is fine in the default case of course. But in cases like these they might be of value for debugging. My main environment isn't Ruby, so I would need to do things to compile the classes again, which in theory are unnecessary because ksv did all those things already. I would just need to tell it to not remove the files and at least could have a first look myself.

README: Add installation instructions

How to install ksv? Please mention it in the README.

Is this how?

gem install specific_install
gem specific_install -l https://github.com/kaitai-io/kaitai_struct_visualizer.git

visualizer: adding search support

Is there any plans to add search support to the visualizer? I doubt your goal is to make a fully featured hex editor, but just having vim-like search support would be pretty neat, I think.

I might be able to supply a pull request for this feature if approved, as well, would just need an hour of someone's time to talk implementation.

I often use Kaitai to map out a file when I am reverse-engineering and like to explore the data surrounding a particular string. I often go to my hex editor to find the offset but it would be nice if I could just do "/Estelle" for example, and jump to the first occurrence of the string "Estelle"

Enable support for opaque types?

I'm currently working to get my KSYs usable using an opaque type in the Ruby visualizer and by default it doesn't seem to work. I need to activate support for opaque types in the compiler:

args = ['--debug', '-t', 'ruby', fn, '-d', code_dir]
to
args = ['--debug', '-t', 'ruby', '--opaque-types=true', fn, '-d', code_dir]

I don't see any way providing this on the shell or such. So, do you want to provide some way or simply enable this feature like you hard code e.g. debug as well?

instance field array crashes

The visualizer crashes when you use it to expose an array based property in instance fields. It seems to work fine for user-defined structs, I believe it's related not being able to figure out the associated part filestream to display.

Feature request: Hex editing

KSC crashes immediately due to unknown option --

The java command line does not accept -- as an option, stopping before even executing KSC.

Java version: Oracle JDK 1.8.0_152
OS: macOS 10.13.2

Things work again after commenting line 34 in kaitai_struct_visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb

I have not opened a PR for this yet, as I am not sure if this was recently changed or has always been like this

Invalid byte sequence crashes ksdump 0.7

I an seeing ksdump crash with the following:

ksdump prescription-sample-1.0-le.eygl kaitai/eyeglass_little_endian.ksy 
Compilation OK
... processing kaitai/eyeglass_little_endian.ksy 0
...... loading eyeglass_format.rb
Classes loaded OK, main class = EyeglassFormat
/usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:268:in `visit_String': invalid byte sequence in UTF-8 (ArgumentError)
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:136:in `accept'
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:330:in `block in visit_Hash'
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:328:in `each'
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:328:in `visit_Hash'
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:136:in `accept'
	from /usr/lib/ruby/3.0.0/psych/visitors/yaml_tree.rb:118:in `push'
	from /usr/lib/ruby/3.0.0/psych.rb:513:in `dump'
	from /usr/lib/ruby/3.0.0/psych/core_ext.rb:13:in `to_yaml'
	from /var/lib/gems/3.0.0/gems/kaitai-struct-visualizer-0.7/bin/ksdump:122:in `<top (required)>'
	from /usr/local/bin/ksdump:25:in `load'
	from /usr/local/bin/ksdump:25:in `<main>'

Example file.

Example ksy.

Via: https://github.com/ross-spencer/eyeglass/tree/main

I'm not clear on if this is a user error, or if there's a newer version of ksdump I should be looking at?

Remove legacy (pre-imports) feature of accepting multiple .ksy files

Currently, ksv's usage help looks like this:

$ ksv --help
Usage: ksv [options] <file_to_parse.bin> <format.ksy>...|<format.rb>

Note the ellipsis in <format.ksy>..., indicating that multiple .ksy files are accepted. Indeed, ksv does accept multiple .ksy specs - the first one is treated as the main one:

kaitai_struct_visualizer/lib/kaitai/struct/visualizer/ksy_compiler.rb

Lines 166 to 170 in 3ec1dce

	# Is it main ClassSpecs?
	if idx.zero?
	main = log_classes[log_fn['firstSpecName']]
	main_class_name = main['topLevelName']
	end

I wondered what this is good for: personally, I've always only specified the root .ksy spec, which contained a /meta/imports section with all the dependencies. So I searched some old commits and discussions on this topic.

This commit from March 2016 introduced ksv's support for accepting multiple .ksy files:

c5871c9

Changed CLI arguments from "<format> <binary>" to "<binary>
 <formats>..." and implemented support for compilation and loading of multiple
 files

---

And these @GreyCat's comments from November 14, 2016 explain the purpose of this feature:

kaitai-io/kaitai_struct#49 (comment)

As for "having 50s of them in one file vs having 50 different files" - it's generally a matter of preference. Kaitai Struct allows you to have several different files for different structures and allows you to reference types in different files. If you'll be using ksv, then you can load up several ksy files at once into it like that: ksv your-binary.dat first-file.ksy second-file.ksy third-file.ksy, etc.

kaitai-io/kaitai_struct#49 (comment)

In the end all of those hacks would remain in one KSY file, right? At least as long as I want your compiler to do the reading work. Or is there any include/module approach I'm missing?

I guess I'm kind of answered that question, but let me elaborate some more. Right now, it's perfectly fine to have stuff like these:

meta:
  id: top_level
seq:
  - id: child
    type: child

and

meta:
  id: child
seq:
  - id: something
    type: u1 # or anything else

in 2 distinct files. It's not possible to separate a single type in 2 different types (at least now). If you feel like that we need to add some sort of include mechanism - let's discuss that - it shouldn't be that hard to implement. Right now we've implemented somewhat working IP networking stack using this approach alone (i.e. some files referencing types in other files).

---

If we combine this knowledge with 0.7 release notes, it all starts making sense:

2017-03-22: Kaitai Struct v0.7 released

• New ksy features:
  • Type importing system: meta/imports can be used to import other
    types as first-class citizens in current compilation unit; "opaque
    types" are now disabled by default (see below)

• Command-line compiler options:
  • --opaque-types=true to enable opaque types (disabled by default,
    i.e. using unknown type would be treated as error)

In short, the ksv's feature of accepting multiple .ksy files pre-dates version 0.7, when imports were added to Kaitai Struct. Before 0.7, the compiler always worked in the "opaque types" mode - if you referenced a type that wasn't defined in the current .ksy spec, it was assumed to be defined somewhere else (either in another .ksy spec, or directly implemented in the target language as an "opaque type" as we understand it today).

So when using a .ksy spec with other .ksy specs as dependencies, you were expected to know what these dependencies are and provide them all to ksv after the main spec. This approach would still work with current ksv, provided you opt into the opaque types mode (which is disabled by default):

opaque_external_type.ksy

meta:
  id: opaque_external_type
  ks-opaque-types: true
seq:
  - id: hw
    type: hello_world

hello_world.ksy

meta:
  id: hello_world
seq:
  - id: one
    type: u1

pp@DESKTOP-89OPGF3 MINGW64 /c/temp/kaitai_struct/tests/formats (serialization)
$ /c/temp/kaitai_struct/visualizer/bin/ksdump -f json ../src/term_strz.bin opaque_external_type.ksy hello_world.ksy
{
  "hw": {
    "one": 102
  }
}

However, I don't see any point in this feature anymore - to me, it's a legacy of pre-0.7 versions which had no imports. I think only one .ksy spec should be accepted (i.e. ksv [options] <file_to_parse.bin> <format.ksy>) and all dependencies on other .ksy files should be specified via /meta/imports - providing more than one .ksy file would cause an error.

What do you think?

KSV ubuntu 12.04 not started

/var/lib/gems/1.8/gems/kaitai-struct-visualizer-0.5/bin/ksv:6: undefined method realpath' for File:Class (NoMethodError) from /usr/local/bin/ksv:19:in load'
from /usr/local/bin/ksv:19

I dont understand ruby.

Not worked with grammars with parameters

If ksy contains key params at top-level, then visualizer crashed with error:

$ ksv msg kaitai-struct/ddc.ksy 
Compilation OK
... processing kaitai-struct/ddc.ksy 0
...... loading ddc.rb
Classes loaded OK, main class = Ddc
/tmp/d20200109-15900-1lqithx/ddc.rb:12:in `initialize': wrong number of arguments (given 1, expected 2..4) (ArgumentError)
	from /var/lib/gems/2.3.0/gems/kaitai-struct-0.8/lib/kaitai/struct/struct.rb:26:in `new'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-0.8/lib/kaitai/struct/struct.rb:26:in `from_file'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/parser.rb:28:in `load'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/visualizer.rb:13:in `run'
	from /var/lib/gems/2.3.0/gems/kaitai-struct-visualizer-0.7/bin/ksv:53:in `<top (required)>'
	from /usr/local/bin/ksv:23:in `load'
	from /usr/local/bin/ksv:23:in `<main>'

Generated struct has following definition:

class Ddc < Kaitai::Struct::Struct
  def initialize(_io, _parent = nil, _root = self, has_mac)
    super(_io, _parent, _root)
    @has_mac = has_mac
    _read
  end
  ...

Original ksv:

meta:
  id: ddc
params:
  - id: has_mac
    type: bool
seq:
  ...

Import issue with KSV

I'm attempting to import my 6th ksy file into my main ksy, but when I run ksv, the interface hangs. When I interrupt it with ctrl-c, it shows the following stack:

^CTraceback (most recent call last):
        11: from /usr/local/bin/ksv:23:in `<main>'
        10: from /usr/local/bin/ksv:23:in `load'
         9: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/bin/ksv:53:in `<top (required)>'
         8: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/visualizer.rb:13:in `run'
         7: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/parser.rb:25:in `load'
         6: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/ksy_compiler.rb:19:in `compile_formats'
         5: from /usr/lib/ruby/2.7.0/tmpdir.rb:89:in `mktmpdir'
         4: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/ksy_compiler.rb:39:in `block in compile_formats'
         3: from /usr/lib/ruby/2.7.0/open3.rb:101:in `popen3'
         2: from /usr/lib/ruby/2.7.0/open3.rb:219:in `popen_run'
         1: from /var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/ksy_compiler.rb:40:in `block (2 levels) in compile_formats'
/var/lib/gems/2.7.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/ksy_compiler.rb:40:in `value': Interrupt

Is there a limit to the number of imports supported?

u8 values not displayed properly

$ xxd foo
00000000: ffff ffff ffff ffff                      ........
$ cat foo.ksy
meta:
  id: foo
  application: foo
  endian: le
seq:
  - id: foo
    type: u8

When expanding the tree of ksv foo foo.ksy:

[-] [root]                    00000000: ff ff ff ff ff ff ff ff                         | ........        
  [-] @foo
    [?] &
    [?] *
    [?] +
    [?] -
    [?] /
    [?] ^
    [?] |
    [?] ~
    [?] **
    [?] <<
    [?] >>
    [?] []
    [?] size
    [?] to_f
    [?] bit_length

treat single-byte fields as byte/integer rather than an array:

array size too big (ArgumentError)

I was trying to parse the data with ksv and then I get this error

Compilation OK
... processing osu_file.ksy 0
...... loading timing_point.rb
...... loading datetime.rb
...... loading vlq_base128_le.rb
...... loading osu_file.rb
...... loading osu_string.rb
...... loading int_double_pair.rb
Classes loaded OK, main class = OsuFile
Traceback (most recent call last):
        10: from C:/Ruby26-x64/bin/ksv:23:in `<main>'
         9: from C:/Ruby26-x64/bin/ksv:23:in `load'
         8: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/kaitai-struct-visualizer-0.7/bin/ksv:53:in `<top (required)>'
         7: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/visualizer.rb:13:in `run'
         6: from C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/kaitai-struct-visualizer-0.7/lib/kaitai/struct/visualizer/parser.rb:32:in `load'
         5: from C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:41:in `_read'
         4: from C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:41:in `times'
         3: from C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:44:in `block in _read'
         2: from C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:157:in `_read'
         1: from C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:157:in `new'
C:/Users/User/AppData/Local/Temp/d20190527-5748-f6tn1m/osu_file.rb:157:in `initialize': array size too big (ArgumentError)

I really don't know what to do.
The ksy files are at here

If you want to try parsing it yourself, you can request the file from me and I will email you the file.

Make ksdump output only JSON on success

Hi, i think ksdump -f json would be easier to use with jq etc and be more what i would expect if only outputted JSON on stdout. Currently it also outputs some log messages on stdout making the output invalid JSON. Maybe on error it could print logs on stderr?

Reproduction:

$ docker run -v "$PWD:/share" -it --entrypoint=ksdump kaitai/ksv -f json /dev/zero some.ksy
Compilation OK
... processing substring_diff.ksy 0
...... loading substring_diff.rb
Classes loaded OK, main class = SubstringDiff
{
  "a": ""
}

Workaround i use at the moment is to pipe thru tail ksdump ... | tail +5 | jq ....

If good ide I guess same would apply for XML and YAML output.

I also noticed that there is no trailing new line for the last JSON output line, making it a bit awkward in a shell.