kaitai-io / kaitai_struct_formats Goto Github PK

View Code? Open in Web Editor NEW

696.0 38.0 201.0 1.5 MB

Kaitai Struct: library of binary file formats (.ksy)

Home Page: http://formats.kaitai.io

Shell 0.06% Ruby 0.95% HTML 1.60% Python 0.63% C# 0.01% C++ 0.01% Kaitai Struct 96.75%

kaitai-struct file-format protocol-parser

kaitai_struct_formats's People

Contributors

Stargazers

Watchers

Forkers

chubbymaggie dgladkov heruix adventureloop moritonal koczkatamas techlord-rce hameloid darklinkxxxx jedahan davidhicks michaelsproul sorgloomer mrexodia filipdominec plepers r00tdaemon mnakamura1337 goodboycc mewbak mewpull chinshou berenm johnwbyrd boxashu sbambach jimregan houcy filippobiga ninelhodzic johnbeard rossburton stennie ikv arlorean snowflakey theorbtwo ykankaya silviupopescu polz113 mhils vpmanske pebsconsulting elliottecton jameshabben meguerreroa skyplabs opt9 kolen bungeman frankpolte chemag gh0st3rs kerrigan29a brunchboy mariuszlitwin bia10 volo-zyko bengardiner pedoc gregcopenhaver dgelessus djkoberstein rjt-gupta lusceu lwerdna argondesign fishilico ildar ciarant pachoo moreati takase1121 vic4key joelkevinjones nsoft jimigrok jeffli678 generalmimon mzehren fechan maurisanfi h4rryk4ne phoenixstar7 trodery leeefang kk49 smcculle petr-briza cudiconsulting kolanich-specs jiangzhhhh nn--- bigboze vineg arbruijn mblsha julienblitte vogelsgesang thatskriptkid

kaitai_struct_formats's Issues

RFC 4122 UUID/GUID

meta:
  id: uuid
  title: UUID/GUID
  xref:
    forensicswiki: Universally Unique Identifier
    iso: "9834-8:2005"
    justsolve: UUID
    rfc: 4122
    wikidata: Q195284
  endian: be
  license: Unlicense
doc: a UUID according to RFC 4122
doc-ref: https://www.ietf.org/rfc/rfc4122.txt

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/uuid/common/uuid.ksy

com.android.opengl.shaders_cache

meta:
  id: android_opengl_shaders_cache
  title: com.android.opengl.shaders_cache
  application: Android apps

doc: |
  Android apps using directly or indirectly OpenGL cache something to com.android.opengl.shaders_cache file.

doc-ref:
  - https://android.googlesource.com/platform/frameworks/native/+/master/opengl/libs/EGL/FileBlobCache.cpp
  - https://android.googlesource.com/platform/frameworks/native/+/master/opengl/libs/EGL/BlobCache.cpp

WiP: https://github.com/kaitai-io/kaitai_struct_formats/blob/shaders_cache/media/android_opengl_shaders_cache.ksy

OpenAL HRIR/HRTF

meta:
  id: openal_hrir
  title: OpenAL Head-Related Impulse Response
  application:
    - OpenAL
  file-extension: mhr
  xref:
    wikidata:
      - Q1432854
      - Q910775
  endian: le

doc: |
  The format to store head-related impulse responses to have binaural spatial sound in the apps using OpenAL.
  The samples can be found by the link: https://github.com/kcat/openal-soft/tree/master/hrtf
doc-ref: https://github.com/kcat/openal-soft/blob/master/docs/hrtf.txt

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/openal_hrir/media/openal_hrir.ksy
issues:
  block: [291]

Document list of supported encodings

https://github.com/kaitai-io/kaitai_struct/wiki/Expressions has this example:

seq:
  - id: filename_len
    type: u4
  - id: filename
    type: str
    size: filename_len
    encoding: UTF-8

Other than UTF-8, which encodings are supported? Would be useful to have this list documented somewhere :)

Ethernet doesn't account for 802.1Q

If 2 Bytes starting at byte 13 is x8100, everything gets thrown off. x8100 is for 802.1Q, and there are an additional 4 Bytes added before the IP packet. I've been trying to fix this, but the compiler seems to cache old .ksy file, and frankly I'm about to pull my hair out. I would love a little bit of help from the community.

tortoisegit.data

meta:
  id: tortoise_git_log_cache
  title: TortoiseGit git log cache
  application: TortoiseGit
  -filename-regexp: "^tortoisegit\\.data$"
  xref:
    wikidata: Q178572
  license: GPL-3.0+
  endian: le
  encoding: utf-16le

doc: |
  TortoiseGit is a GUI frontend for git SCM. It caches some data to `tortoisegit.data` file in `.git` dir.

doc-ref:
  - https://gitlab.com/tortoisegit/tortoisegit/raw/master/src/TortoiseProc/gitlogcache.h
  - https://gitlab.com/tortoisegit/tortoisegit/raw/master/src/TortoiseProc/GitLogCache.cpp

Can't open some gifs

There are some gifs that cannot be opened by the parser obtained from gif.ksy, these can be opened in the browser, on desktop and also by Pillow. The below gif is one example taken from the Pillow repository.

I'm using python 3.5 and the error is -

>>> a = Gif.from_file('iss634.gif')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/s4chin/.virtualenvs/kaitai/lib/python3.5/site-packages/kaitaistruct.py", line 28, in from_file
    return cls(KaitaiStream(f))
  File "/home/s4chin/img/gif.py", line 35, in __init__
    self.blocks.append(self._root.Block(self._io, self, self._root))
  File "/home/s4chin/img/gif.py", line 146, in __init__
    self.block_type = self._root.BlockType(self._io.read_u1())
  File "/usr/lib/python3.5/enum.py", line 241, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.5/enum.py", line 476, in __new__
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 166 is not a valid BlockType

QEMU Copy-on-Write 2

meta:
  id: qcow2
  title: QEMU Copy-on-Write 2
  application:
    - QEMU
    - VirtualBox
    - VMWare Workstation
    - libqcow
  file-extension: qcow
  xref: 
    forensicswiki: QCOW_Image_Format 
    wikidata: Q1052000
  endian: be
  encoding: utf-8
doc: |
  A native QEMU image file format.
  A qcow2 image file is organized in units of constant size, which are called (host) clusters. A cluster is the unit in which all allocations are done, both for actual guest data and for image metadata.
  Likewise, the virtual disk as seen by the guest is divided into (guest) clusters of the same size.
  If the image has a backing file then the backing file name should be stored in the remaining space between the end of the header extension area and the end of the first cluster. It is not allowed to store other data here, so that an implementation can safely modify the header and add extensions without harming data of compatible features that it doesn't support. Compatible features that need space for additional data can use a header extension.

doc-ref:
  - https://github.com/qemu/qemu/blob/master/docs/interop/qcow2.txt
  - https://github.com/qemu/qemu/blob/master/block/qcow2.h
  - https://github.com/qemu/qemu/blob/master/block/qcow2.c
  - https://www.virtualbox.org/svn/vbox/trunk/src/VBox/Storage/QCOW.cpp

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/qcow/filesystem/qcow2.ksy
issues:
  block: [ null ] # there used to be one, but I don't remember which one

xdelta

meta:
  id: vcdiff
  title: VCDiff / xdelta
  application: xdelta
  xref:
    rfc: 3284
    wikidata: Q619133

doc: |
  a binary diff format used in incremental updates

doc-ref:
  - https://tools.ietf.org/html/rfc3284
  - https://www.w3.org/TR/NOTE-gdiff-19970901
  - https://github.com/jmacd/xdelta
  - https://github.com/mantlik/xdeltaencoder

Qualcomm Multiple Partition

meta:
  id: qualcomm_mbn
  title: Qualcomm Multiple Partition
  application: Qualcomm SOC
  file-extension: mbn
doc: |
  A format used by Qualcomm SOC to store firmware.

doc-ref:
  - https://github.com/laginimaineb/unpack_bootloader_image/blob/master/unpack_bootloader_image.py
  - https://bits-please.blogspot.ru/2015/08/exploring-qualcomms-trustzone.html
  - https://forum.xda-developers.com/showthread.php?t=2641245
  - https://github.com/Surge1223/ida-mbn-sbl-loader
  - https://github.com/vtsingaras/qcom-mbn-ida-loader
  - https://github.com/ralekdev/mbn_ida_loader

Array Index error in Websocket messages

Right now, the websocket format uses _root.dataframes[0] to figure out the payload type. The issue is that _root.dataframes[0] doesn't exist for the first dataframe, so when the constructor gets called to read (which is eagerly evaluated), there's an array index error.

Given that messages are guaranteed to be at least 1 dataframe long, should the format be updated to have an initial frame, and then all frames following?

EDIT: That doesn't actually work either, because now there's a NPE accessing _root.first_frame because the constructor runs too quickly.

Windows Prefetch File

meta:
  id: windows_prefetch_file
  title: Windows Prefetch File 
  application: pf
  file-extension: Microsoft Windows Prefetcher
  xref:
    forensicswiki: Windows_Prefetch_File_Format
    wikidata: Q4047325

doc: |
  Windows Prefetcher service collects info about apps launches to optimize their startup.

doc-ref:
  - https://github.com/libyal/libscca/blob/master/documentation/Windows%20Prefetch%20File%20(PF)%20format.asciidoc

Microsoft CogNitive ToolKit

meta:
  id: cntk
  title: Microsoft CogNitive ToolKit Computation NeTworK
  file-extension: model
  xref:
    wikidata: Q23680120
  endian: le
  encoding: UTF-16le
  license: MIT

doc: |
  A native CNTK format for serialization of a computation graph. These graphs are usually used to store machine-learning models, like neural networks and gradient boosted decision trees ensembles.
  The list of pretrained models is here: https://github.com/Microsoft/CNTK/blob/master/PretrainedModels/download_model.py
  Only some of them have this format.

doc-ref:
  - https://github.com/Microsoft/CNTK

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/CNTK/scientific/data_science/cntk.ksy
issues:
  block: [ 435, 459 ]

To overcome KSC limitations currently a patch file is used.

Zeno IMproved

meta:
  id: zim
  title: "(Open) Zeno IMproved"
  application: 
    - Kiwix
    - zimlib
  file-extension: zim
  xref:
    wikidata: Q784695
  license: CC-BY-SA-3.0
  encoding: utf-8
  endian: le
doc: |
  A file format to store encyclopaedias of articles written in MediaWiki markup language. 
  Files for test: https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/

doc-ref:
  - https://www.openzim.org/wiki/ZIM_file_format
  - https://wiki.openzim.org/wiki/OpenZIM

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/OpenZIM/media/openzim.ksy

Add JPEG Interchange Format (JIF) file format

Documentation on the file format: https://www.w3.org/Graphics/JPEG/itu-t81.pdf

Branch containing draft specification: https://github.com/davidhicks/kaitai_struct_formats/tree/jif

Sample/test files: https://www.itu.int/net/itu-t/sigdb/speimage/ImageForm-s.aspx?val=1010083

Acronis True Image tib

meta:
  id: acronis_tib
  title: Acronis True Image Backup
  application: Acronis True Image
  file-extension: tib
  xref:
    wikidata: Q2422684

doc: |
  Acronis True Image is a popular backup solution.

doc-ref:
  - https://github.com/cgsecurity/TestDisk/blob/master/src/file_tib.c
  - https://geektimes.ru/post/241086/

Windows Platform Binary Table

meta:
  id: windows_platform_binary_table
  title: Windows Platform Binary Table
  application: Microsoft Windows

doc: |
  A table in a BIOS containing a PE executable to be executed by Windows. Used by some vendor-preinstalled bacodoors.

doc-ref:
  - https://download.microsoft.com/download/8/A/2/8A2FB72D-9B96-4E2D-A559-4A27CF905A80/windows-platform-binary-table.docx

Add 7-zip (.7z) file format

Documentation on the file format: see /DOC/7zFormat.txt within the latest SDK available for download at http://www.7-zip.org/sdk.html

Branch containing draft specification: https://github.com/davidhicks/kaitai_struct_formats/tree/7z

WebAssembly

@sophoslabs have started a WebAssembly parser:

meta:
  id: webassembly
  title: Web Assembly parser
  file-extension: wasm
  endian: le
  license: CC0-1.0
  file-extension:
    - wasm
  imports:
    - vlq_base128_le
doc: |
  WebAssembly is a web standard that defines a binary format and a corresponding
  assembly-like text format for executable code in Web pages. It is meant to
  enable executing code nearly as fast as running native machine code.
doc-ref: https://github.com/WebAssembly/design/blob/master/BinaryEncoding.md

Apple AWDL

meta:
  id: apple_awdl
  title: Apple Wireless Direct Link
  application:
    - AirDrop
  xref:
    wikidata: Q406655

doc: | 
  Apple Wireless Direct Link is a proprietaty protocol used in Apple software and devices for peer-to-peer connections. It utilizes Wi-Fi for data transfer and Bluetooth Low Energy for sending metainformation to reduce power consumption.
doc-ref:
  - "https://arxiv.org/abs/1808.03156"
  - "https://github.com/seemoo-lab/owl"
  - "https://github.com/seemoo-lab/wireshark-awdl"
  - "https://github.com/Tarlab/tarAWDL"
  - "https://github.com/seemoo-lab/opendrop"

How can IPv4 header be described in kaitai with options?

I saw example kaitai for IPv4 header. But options extention in not mentioned in it.
Can IPv4 header be fully describer in kaitai?

CERN ROOT

meta:
  id: root
  title: CERN ROOT
  application: CERN ROOT
  file-extension: root
  xref:
    justsolve: ROOT
    wikidata: Q947171
  endian: be
  imports:
    - /common/uuid
doc: | 
  File format used by scientific software CERN ROOT.
doc-ref:
  - "https://root.cern.ch/doc/master/classTFile.html"
  - "https://github.com/root-mirror/root/blob/master/documentation/users-guide/InputOutput.md"
  - "https://root.cern.ch/root/htmldoc/guides/users-guide/ROOTUsersGuide.html#inputoutput"

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/root/scientific/root.ksy

Add MPEG-TS format

Basic parsing is possible (@pavja2 has POC), but for a more effective parser it depends on kaitai-io/kaitai_struct#196.

Add ISO/IEC 8211 (ASCII variant) file format

Documentation on the file format: see https://www.wikidata.org/wiki/Q29644119

Branch containing draft specification: https://github.com/davidhicks/kaitai_struct_formats/tree/iso_iec_8211_ascii

Wiki entry for `contents:`?

What are the various ways of specifying contents? I couldn’t find info on contents in the wiki.

The example in the README has this:

types:
  header:
    seq:
      - id: magic
        contents: 'GIF'

Which made me wonder, does it only accept string values or can you specify byte sequences as well? E.g.

contents: 0xCA 0xFE

Does something like this work?

Update: Ah, java_class.ksy partially answers the question:

contents: [0xca, 0xfe, 0xba, 0xbe]

Still, it would be nice to have the different formats documented on the wiki.

jpeg.ksy doesn't have all segments declared

So I'm using this version which solves a lot of problems - https://github.com/koczkatamas/kaitai_struct_formats/tree/fix/compile-workarounds.
As all the appn segments are not declared there, it gives an error when it encounters one or more undeclared segments. Should I add them and submit a PR?

bsdiff

meta:
  id: bsdiff
  title: bsdiff
  application: bsdiff
  file-extension: bsdiff

doc: |
  a format for diffs of binary files. Used in incremental updates.

doc-ref:
  - https://github.com/mendsley/bsdiff
  - https://github.com/synalysis/Grammars/blob/master/bsdiff.grammar

PeaZip pea

meta:
  id: pea
  title: PeaZIP archive file
  file-extension: pea
  application:
    - PeaZIP
    - PEA
    - UnPEA
  xref:
    wikidata: Q1275912
  endian: le
  license: Unlicense
doc: |
  Format for PeaZIP archiver. WARNING: don't design formats like this!
doc-ref:
  - https://github.com/giorgiotani/PeaZip/releases

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/peazip

Origin opj

meta:
  id: origin_opj
  title: Origin project
  application:
    - Origin
    - OriginPro
    - Origin Viewer
  file-extension: opj
  xref:
    wikidata: Q1307239
  endian: be
  license: ?

doc: |
  Origin is a scientific spreadsheet editor. It uses an own proprietary format.
  
  Test files:
    https://github.com/aperbook/APER/tree/master/Origin
    https://github.com/choderalab/bayesian-itc/tree/master/data/Mg2-EDTA/Mg2EDTA
    https://github.com/aitap/Ropj/blob/master/inst/test.opj
    https://github.com/p4db/p4db-data/raw/master/icnp.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/76/Photocurrent%20spectra%20of%20InSe-Indium%20oxide%20p-n%20junction.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/76/Effect%20of%20thermal%20annealing%20with%20increasing%20temperature.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/69/Grafting%20from%20%28thermal%29%20v%20time.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/69/Grafting%20to%20IR%20PDMSvThiol.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/69/Grafting%20to%20IR%20results%20bar.opj
    https://rdmc-test.nottingham.ac.uk/bitstream/handle/internal/69/Polymer%20kinetics%20solution.opj
doc-ref:
  - https://github.com/jgonera/openopj/blob/master/docs/opj_format.markdown
  - https://github.com/Saluev/python-liborigin2
  - https://github.com/swharden/PyOriginTools
  - https://sourceforge.net/p/liborigin/git/ci/master/tree/
  - https://github.com/vpaeder/terapy/blob/master/terapy/files/origin.py
  - https://sourceforge.net/p/scidavis/svn/HEAD/tree/trunk/qtfrontend/origin/OpjImporter.cpp

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/origin_opj

Windows Spider Solitaire save file

meta:
  id: spider_sav
  title: Windows Spider Solitaire save file
  application:
    - Windows Spider Solitaire
  file-extension: sav
doc-ref: http://www.genialitaet.de/fw/Spider.sav
doc:|
  A file format used by Spider Solitaire shipped with Windows XP for saving games.

WiP: https://github.com/KOLANICH/kaitai_struct_formats/blob/spider_sav/game/spider_sav.ksy

Smile serialization format

meta:
  id: smile
  title: Smile
  file-extension: sml
  xref:
    mime: application/x-jackson-smile
    wikidata: Q17081599
  endian: le
  encoding: UTF-8
  -version: "1.0.4 (12-May-2013)"
  license: BSD-2-Clause

doc: |
  Efficient JSON-compatible binary format: "Smile"
  "Smile" is codename for an efficient JSON-compatible binary data format, initially developed by Jackson JSON processor project team.
  Its logical data model is same as that of JSON, so it can be considered a "Binary JSON" format.
doc-ref: https://github.com/FasterXML/smile-format-specification

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/smile/serialization/smile.ksy

Alcohol / DaemonTools .MDF

meta:
  id: mdf
  title: "DaemonTools and Alcohol 120% proprietary format"
  application:
    - Alcohol 120%
    - Alcohol 62%
    - DaemonTools
  file-extension: mdf
  xref:
    justsolve: MDF_and_MDS
    wikidata: Q3853674
  endian: le
  license: LGPL-3.0 # or maybe MIT: https://github.com/discimagechef/DiscImageChef/blob/master/LICENSE.MIT
doc: |
  Native file format of Alcohol and DaemonTools. Stores a disk image.
doc-ref:
- https://github.com/discimagechef/DiscImageChef/tree/master/DiscImageChef.DiscImages/Alcohol120

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/alcohol_daemontools_mdf/filesystem/mdf.ksy

Android Parcel

meta:
  id: android_parcel
  title: Android Parcel
  application:
    - Android
    - Android Binder
  xref:
    wikidata: Q17121613
  license: Apache-2.0
  encoding: utf-16
  endian: le # in fact machine endianess, but most of kachines are LE

doc: |
  Container for a message (data and object references) that can be sent through an IBinder.  A Parcel can contain both flattened data that will be unflattened on the other side of the IPC (using the various methods here for writing specific types, or the general Parcelable interface), and references to live IBinder objects that will result in the other side receiving a proxy IBinder connected with the original IBinder in the Parcel.
  Parcel is **not** a general-purpose serialization mechanism.  This class (and the corresponding Parcelable API for placing arbitrary objects into a Parcel) is designed as a high-performance IPC transport.  As such, it is not appropriate to place any Parcel data in to persistent storage: changes in the underlying implementation of any of the data in the Parcel can render older data unreadable.


doc-ref:
  - https://github.com/android/platform_frameworks_base/blob/master/core/java/android/os/Parcel.java
  - https://developer.android.com/reference/android/os/Parcel

WiP:
  - https://github.com/KOLANICH/kaitai_struct_formats/blob/android_parcel/serialization/android_parcel.ksy

Tcl cookfs

meta:
  id: tcl_cookfs
  title: Tcl cookfs
  application: tcl

doc: |
  A Tcl virtual file system for serialization

doc-ref: # MANDATORY.
  - https://sourceforge.net/p/cookit/code/HEAD/tree/cookfs/generic/cookfs.c
  - https://tclcommunityassociation.org/wub/proceedings/Proceedings-2010/WojciechKocjan/Cookfs-Paper/Cookfs-Paper.pdf

JVM pack200

meta:
  id: jvm_pack
  application: 
    - pack200
    - unpack200
  file-extension: pack
  xrefs:
    wikidata: Q1227683
doc: |
  A format to compress JVM JAR files.

doc-ref: https://github.com/netroby/jdk9-dev/blob/master/jdk/src/jdk.pack200/share/native/unpack200/main.cpp

Metakit

meta:
  id: metakit
  title: Metakit
  application: Metakit
  file-extension: mk
  xref:
    wikidata: Q1924669

doc: |
  An embeddable database format.

doc-ref:
  - https://equi4.com/metakit/
  - https://git.jeelabs.org/jcw/metakit
  - https://github.com/electric-cloud/metakit/tree/master/src

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/metakit4

ZX Spectrum TZX tape image

meta:
  id: zx_spectrum_tzx
  title: ZX Spectrum TZX tape image
  file-extension: tzx
  xref:
    justsolve: TZX
    pronom: fmt/1000

doc: |
  The TZX tape image format for tapes for the ZX Spectrum (and compatibles/derivatives/related machines). Compared to the very basic TAP format, which can only store standard-format tape data as used by the ROM load/save routines, the TZX format can also store various kinds of turbo loader and copy protection data, as well as arbitrary audio data if necessary. The TZX format also supports various kinds of metadata, including human-readable information about the tape, machine-readable information about the hardware that the tape is intended for, and control instructions to affect how the tape is loaded by an emulator.

doc-ref:
  - https://www.worldofspectrum.org/TZXformat.html
  - http://www.zx-modules.de/fileformats/tzxformat.html

Lytro Light Field Picture

meta:
  id: lytro_lfp
  title: Lytro Light Field Picture
  application:
    - Lytro Cameras
    - Lytro Desktop
  file-extension: lfp
  xref:
    wikidata: Q2984629

doc: |
  Native file format of Lytro light field cameras.

doc-ref:
  - https://github.com/nrpatel/lfptools
  - https://github.com/behnam/python-lfp-reader
  - https://github.com/hahnec/plenopticam
  - https://github.com/imageio/imageio/blob/master/imageio/plugins/lytro.py
  - https://eclecti.cc/computervision/reverse-engineering-the-lytro-lfp-file-format
  - http://optics.miloush.net/lytro/TheFileFormat.aspx

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/lytro_lfp

NI LabView vi

meta:
  id: labview_vi
  title: LabView virtual instrument
  application: LabView
  file-extension: vi
  xref: 
    wikidata: Q746261

doc: |
  NI LabView is a popular IDE for visual programming. Usages are from children toys like LEGO Mindstorms to ~~children toys~~experiment automation, digital signal processing and machine learning pipelines.

doc-ref:
  - https://github.com/tomsoftware/VI-Explorer
  - https://github.com/tomsoftware/VI-Explorer-VI

FTDI232 USB to RS232 protocol

meta:
  id: ftdi232
  title: FTDI232 protocol
  endian: le
  license: MIT
  xref:
    wikidata: Q2561478
doc: |
  Protocol for FTDI232 chips.
  Most of the stuff is set via control transactions, data is usually passed as control transaction arguments: value and index. Obviously, Kaitai Struct cannot describe protocols where the data is transfered in fields of other protocols, we need a separate Verilog-based DSL for that.
  But sometimes the stuff is set up as control commands payload.

doc-ref:
  - https://github.com/felHR85/UsbSerial

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/ftdi

zchunk

meta:
  id: zchunk
  title: zchunk
  application:
    - unzck
    - zck
    - zckdl
    - zck_read_header
  file-extension: zck
doc: |
  A format for incremental distribution of software updates.

doc-ref: https://github.com/zchunk/zchunk/blob/master/zchunk_format.txt

FreeArc arc

meta:
  id: freearc_arc
  title: FreeArc
  application: FreeArc
  file-extension: arc
  xref:
    wikidata: Q3495962

doc: |
  Native archive format of FreeArc archiver.

doc-ref:
  - https://github.com/Bulat-Ziganshin/FA

Pepakura PDO

meta:
  id: pepakura_pdo
  endian: le
  file-extension: pdo
  title: Pepakura PDO v3
  application:
   - Pepakura Designer
   - Pepakura Reader
  encoding: UTF-8
doc: |
  A proprietary format of popular software for creation of (ki|o)rigami.
  Files for testing: https://github.com/daeken/PepakuraReverse/tree/master/pdos
doc-ref:
  - https://github.com/daeken/PepakuraReverse

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/pepakura_pdo

Add Adobe Photoshop (PSD) file format

Official specification of the Adobe Photoshop (.psd) file format: https://www.adobe.com/devnet-apps/photoshop/fileformatashtml/

Work in progress branch for implementing a Kaitai Struct format specification for .psd: https://github.com/davidhicks/kaitai_struct_formats/tree/psd

Where are the docs?

I'm trying to understand doom_wad.ksy, but I'm having trouble because the language is almost completely undocumented except for the GIF example. Doom_wad.ksy uses several features of the KSY language that aren't used in the GIF example and aren't documented anywhere.

The commit log mentions a change that was made "per spec". Where is this spec? I couldn't find it on kaitai.io or in any of the repositories.

Turi Create SFrame

meta:
  id: turi_create_sframe
  title: Turi Create SFrame
  application: Turi Create

doc: |
  Turi Create is an external memory machine learning framework. SFrame is an external memory tahle of data similar to pandas.DataFrame

doc-ref: https://github.com/apple/turicreate/blob/master/src/core/storage/sframe_data/sframe_saving.cpp

.NET Remoting Binary Format (NRBF)

meta:
  id: nrbf
  endian: le
  application: .NET
  imports:
    - vlq_base128_le

doc: |
  .NET binary serialisation format (BinaryFormatter)

@trejkaz started WiP here: https://github.com/trejkaz/nrbf.ksy

Cannot compile JavaClass.ksy

I downloaded the latest compiler (0.3) from bintray.
I ran it as so: kaitai-struct-compiler kaitai_struct\formats\executable\java_class.ksy --target java
I get the code below, which has a compile error on line 445.
The problem line wants to call JavaClass.Attribute(KaitaiStream, JavaClass.MethodInfo, JavaClass)
But the closest constructor is JavaClass.Attribute(KaitaiStream, JavaClass.FieldInfo, JavaClass)

Not sure if it's a bug in the compiler or the ksy. Very cool project idea :)

//This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

import io.kaitai.struct.KaitaiStruct;
import io.kaitai.struct.KaitaiStream;

import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;

public class JavaClass extends KaitaiStruct {
 public static JavaClass fromFile(String fileName) throws IOException {
     return new JavaClass(new KaitaiStream(fileName));
 }

 public JavaClass(KaitaiStream _io) throws IOException {
     super(_io);
     this._root = this;
     _parse();
 }

 public JavaClass(KaitaiStream _io, KaitaiStruct _parent) throws IOException {
     super(_io);
     this._parent = _parent;
     this._root = this;
     _parse();
 }

 public JavaClass(KaitaiStream _io, KaitaiStruct _parent, JavaClass _root) throws IOException {
     super(_io);
     this._parent = _parent;
     this._root = _root;
     _parse();
 }
 private void _parse() throws IOException {
     this.magic = _io.ensureFixedContents(4, new byte[] { -54, -2, -70, -66 });
     this.versionMinor = _io.readU2be();
     this.versionMajor = _io.readU2be();
     this.constantPoolCount = _io.readU2be();
     constantPool = new ArrayList<ConstantPoolEntry>((int) ((constantPoolCount() - 1)));
     for (int i = 0; i < (constantPoolCount() - 1); i++) {
         this.constantPool.add(new ConstantPoolEntry(_io, this, _root));
     }
     this.accessFlags = _io.readU2be();
     this.thisClass = _io.readU2be();
     this.superClass = _io.readU2be();
     this.interfacesCount = _io.readU2be();
     interfaces = new ArrayList<Integer>((int) (interfacesCount()));
     for (int i = 0; i < interfacesCount(); i++) {
         this.interfaces.add(_io.readU2be());
     }
     this.fieldsCount = _io.readU2be();
     fields = new ArrayList<FieldInfo>((int) (fieldsCount()));
     for (int i = 0; i < fieldsCount(); i++) {
         this.fields.add(new FieldInfo(_io, this, _root));
     }
     this.methodsCount = _io.readU2be();
     methods = new ArrayList<MethodInfo>((int) (methodsCount()));
     for (int i = 0; i < methodsCount(); i++) {
         this.methods.add(new MethodInfo(_io, this, _root));
     }
 }
 public static class MethodRefCpInfo extends KaitaiStruct {
     public static MethodRefCpInfo fromFile(String fileName) throws IOException {
         return new MethodRefCpInfo(new KaitaiStream(fileName));
     }

     public MethodRefCpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public MethodRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public MethodRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.classIndex = _io.readU2be();
         this.nameAndTypeIndex = _io.readU2be();
     }
     private int classIndex;
     private int nameAndTypeIndex;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int classIndex() { return classIndex; }
     public int nameAndTypeIndex() { return nameAndTypeIndex; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 public static class FieldInfo extends KaitaiStruct {
     public static FieldInfo fromFile(String fileName) throws IOException {
         return new FieldInfo(new KaitaiStream(fileName));
     }

     public FieldInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public FieldInfo(KaitaiStream _io, JavaClass _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public FieldInfo(KaitaiStream _io, JavaClass _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.accessFlags = _io.readU2be();
         this.nameIndex = _io.readU2be();
         this.descriptorIndex = _io.readU2be();
         this.attributesCount = _io.readU2be();
         attributes = new ArrayList<Attribute>((int) (attributesCount()));
         for (int i = 0; i < attributesCount(); i++) {
             this.attributes.add(new Attribute(_io, this, _root));
         }
     }
     private int accessFlags;
     private int nameIndex;
     private int descriptorIndex;
     private int attributesCount;
     private ArrayList<Attribute> attributes;
     private JavaClass _root;
     private JavaClass _parent;
     public int accessFlags() { return accessFlags; }
     public int nameIndex() { return nameIndex; }
     public int descriptorIndex() { return descriptorIndex; }
     public int attributesCount() { return attributesCount; }
     public ArrayList<Attribute> attributes() { return attributes; }
     public JavaClass _root() { return _root; }
     public JavaClass _parent() { return _parent; }
 }
 public static class NameAndTypeCpInfo extends KaitaiStruct {
     public static NameAndTypeCpInfo fromFile(String fileName) throws IOException {
         return new NameAndTypeCpInfo(new KaitaiStream(fileName));
     }

     public NameAndTypeCpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public NameAndTypeCpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public NameAndTypeCpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.nameIndex = _io.readU2be();
         this.descriptorIndex = _io.readU2be();
     }
     private int nameIndex;
     private int descriptorIndex;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int nameIndex() { return nameIndex; }
     public int descriptorIndex() { return descriptorIndex; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 public static class Utf8CpInfo extends KaitaiStruct {
     public static Utf8CpInfo fromFile(String fileName) throws IOException {
         return new Utf8CpInfo(new KaitaiStream(fileName));
     }

     public Utf8CpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public Utf8CpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public Utf8CpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.strLen = _io.readU2be();
         this.value = _io.readStrByteLimit(strLen(), "UTF-8");
     }
     private int strLen;
     private String value;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int strLen() { return strLen; }
     public String value() { return value; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 public static class InterfaceMethodRefCpInfo extends KaitaiStruct {
     public static InterfaceMethodRefCpInfo fromFile(String fileName) throws IOException {
         return new InterfaceMethodRefCpInfo(new KaitaiStream(fileName));
     }

     public InterfaceMethodRefCpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public InterfaceMethodRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public InterfaceMethodRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.classIndex = _io.readU2be();
         this.nameAndTypeIndex = _io.readU2be();
     }
     private int classIndex;
     private int nameAndTypeIndex;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int classIndex() { return classIndex; }
     public int nameAndTypeIndex() { return nameAndTypeIndex; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 public static class Attribute extends KaitaiStruct {
     public static Attribute fromFile(String fileName) throws IOException {
         return new Attribute(new KaitaiStream(fileName));
     }

     public Attribute(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public Attribute(KaitaiStream _io, FieldInfo _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public Attribute(KaitaiStream _io, FieldInfo _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.attributeNameIndex = _io.readU2be();
         this.attributeLength = _io.readU4be();
         this.info = _io.readBytes(attributeLength());
     }
     private int attributeNameIndex;
     private long attributeLength;
     private byte[] info;
     private JavaClass _root;
     private JavaClass.FieldInfo _parent;
     public int attributeNameIndex() { return attributeNameIndex; }
     public long attributeLength() { return attributeLength; }
     public byte[] info() { return info; }
     public JavaClass _root() { return _root; }
     public JavaClass.FieldInfo _parent() { return _parent; }
 }
 public static class ClassCpInfo extends KaitaiStruct {
     public static ClassCpInfo fromFile(String fileName) throws IOException {
         return new ClassCpInfo(new KaitaiStream(fileName));
     }

     public ClassCpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public ClassCpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public ClassCpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.nameIndex = _io.readU2be();
     }
     private ConstantPoolEntry name;
     public ConstantPoolEntry name() throws IOException {
         if (name != null)
             return name;
         name = _root.constantPool().get((nameIndex() - 1));
         return name;
     }
     private int nameIndex;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int nameIndex() { return nameIndex; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 public static class ConstantPoolEntry extends KaitaiStruct {
     public static ConstantPoolEntry fromFile(String fileName) throws IOException {
         return new ConstantPoolEntry(new KaitaiStream(fileName));
     }

     public ConstantPoolEntry(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public ConstantPoolEntry(KaitaiStream _io, JavaClass _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public ConstantPoolEntry(KaitaiStream _io, JavaClass _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.tag = Tag.byId(_io.readU1());
         if (tag() == Tag.UTF8) {
             this.utf8CpInfo = new Utf8CpInfo(_io, this, _root);
         }
         if (tag() == Tag.CLASS) {
             this.classCpInfo = new ClassCpInfo(_io, this, _root);
         }
         if (tag() == Tag.NAME_AND_TYPE) {
             this.nameAndTypeCpInfo = new NameAndTypeCpInfo(_io, this, _root);
         }
         if (tag() == Tag.FIELD_REF) {
             this.fieldRefCpInfo = new FieldRefCpInfo(_io, this, _root);
         }
         if (tag() == Tag.METHOD_REF) {
             this.methodRefCpInfo = new MethodRefCpInfo(_io, this, _root);
         }
         if (tag() == Tag.INTERFACE_METHOD_REF) {
             this.interfaceMethodRefCpInfo = new InterfaceMethodRefCpInfo(_io, this, _root);
         }
     }
     private Tag tag;
     private Utf8CpInfo utf8CpInfo;
     private ClassCpInfo classCpInfo;
     private NameAndTypeCpInfo nameAndTypeCpInfo;
     private FieldRefCpInfo fieldRefCpInfo;
     private MethodRefCpInfo methodRefCpInfo;
     private InterfaceMethodRefCpInfo interfaceMethodRefCpInfo;
     private JavaClass _root;
     private JavaClass _parent;
     public Tag tag() { return tag; }
     public Utf8CpInfo utf8CpInfo() { return utf8CpInfo; }
     public ClassCpInfo classCpInfo() { return classCpInfo; }
     public NameAndTypeCpInfo nameAndTypeCpInfo() { return nameAndTypeCpInfo; }
     public FieldRefCpInfo fieldRefCpInfo() { return fieldRefCpInfo; }
     public MethodRefCpInfo methodRefCpInfo() { return methodRefCpInfo; }
     public InterfaceMethodRefCpInfo interfaceMethodRefCpInfo() { return interfaceMethodRefCpInfo; }
     public JavaClass _root() { return _root; }
     public JavaClass _parent() { return _parent; }

     public enum Tag {
         LONG(5),
         METHOD_REF(10),
         UTF8(1),
         DOUBLE(6),
         FIELD_REF(9),
         NAME_AND_TYPE(12),
         CLASS(7),
         INTEGER(3),
         INVOKE_DYNAMIC(18),
         METHOD_TYPE(16),
         INTERFACE_METHOD_REF(11),
         STRING(8),
         FLOAT(4),
         METHOD_HANDLE(15);

         private final long id;
         Tag(long id) { this.id = id; }
         public long id() { return id; }
         private static final Map<Long, Tag> byId = new HashMap<Long, Tag>(14);
         static {
             for (Tag e : Tag.values())
                 byId.put(e.id(), e);
         }
         public static Tag byId(long id) { return byId.get(id); }
     }
 }
 public static class MethodInfo extends KaitaiStruct {
     public static MethodInfo fromFile(String fileName) throws IOException {
         return new MethodInfo(new KaitaiStream(fileName));
     }

     public MethodInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public MethodInfo(KaitaiStream _io, JavaClass _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public MethodInfo(KaitaiStream _io, JavaClass _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.accessFlags = _io.readU2be();
         this.nameIndex = _io.readU2be();
         this.descriptorIndex = _io.readU2be();
         this.attributesCount = _io.readU2be();
         attributes = new ArrayList<Attribute>((int) (attributesCount()));
         for (int i = 0; i < attributesCount(); i++) {
             this.attributes.add(new Attribute(_io, this, _root));
         }
     }
     private int accessFlags;
     private int nameIndex;
     private int descriptorIndex;
     private int attributesCount;
     private ArrayList<Attribute> attributes;
     private JavaClass _root;
     private JavaClass _parent;
     public int accessFlags() { return accessFlags; }
     public int nameIndex() { return nameIndex; }
     public int descriptorIndex() { return descriptorIndex; }
     public int attributesCount() { return attributesCount; }
     public ArrayList<Attribute> attributes() { return attributes; }
     public JavaClass _root() { return _root; }
     public JavaClass _parent() { return _parent; }
 }
 public static class FieldRefCpInfo extends KaitaiStruct {
     public static FieldRefCpInfo fromFile(String fileName) throws IOException {
         return new FieldRefCpInfo(new KaitaiStream(fileName));
     }

     public FieldRefCpInfo(KaitaiStream _io) throws IOException {
         super(_io);
         _parse();
     }

     public FieldRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent) throws IOException {
         super(_io);
         this._parent = _parent;
         _parse();
     }

     public FieldRefCpInfo(KaitaiStream _io, ConstantPoolEntry _parent, JavaClass _root) throws IOException {
         super(_io);
         this._parent = _parent;
         this._root = _root;
         _parse();
     }
     private void _parse() throws IOException {
         this.classIndex = _io.readU2be();
         this.nameAndTypeIndex = _io.readU2be();
     }
     private int classIndex;
     private int nameAndTypeIndex;
     private JavaClass _root;
     private JavaClass.ConstantPoolEntry _parent;
     public int classIndex() { return classIndex; }
     public int nameAndTypeIndex() { return nameAndTypeIndex; }
     public JavaClass _root() { return _root; }
     public JavaClass.ConstantPoolEntry _parent() { return _parent; }
 }
 private byte[] magic;
 private int versionMinor;
 private int versionMajor;
 private int constantPoolCount;
 private ArrayList<ConstantPoolEntry> constantPool;
 private int accessFlags;
 private int thisClass;
 private int superClass;
 private int interfacesCount;
 private ArrayList<Integer> interfaces;
 private int fieldsCount;
 private ArrayList<FieldInfo> fields;
 private int methodsCount;
 private ArrayList<MethodInfo> methods;
 private JavaClass _root;
 private KaitaiStruct _parent;
 public byte[] magic() { return magic; }
 public int versionMinor() { return versionMinor; }
 public int versionMajor() { return versionMajor; }
 public int constantPoolCount() { return constantPoolCount; }
 public ArrayList<ConstantPoolEntry> constantPool() { return constantPool; }
 public int accessFlags() { return accessFlags; }
 public int thisClass() { return thisClass; }
 public int superClass() { return superClass; }
 public int interfacesCount() { return interfacesCount; }
 public ArrayList<Integer> interfaces() { return interfaces; }
 public int fieldsCount() { return fieldsCount; }
 public ArrayList<FieldInfo> fields() { return fields; }
 public int methodsCount() { return methodsCount; }
 public ArrayList<MethodInfo> methods() { return methods; }
 public JavaClass _root() { return _root; }
 public KaitaiStruct _parent() { return _parent; }
}

ZStandard compression

meta:
  id: zstd
  title: ZStandard compression format
  file-extension: zst
  xref: 
    mime: application/zstd
    rfc: 8478
    wikidata: Q26737171
  license: Unlicense
  endian: le
doc: |
  ZStandard compressed frame.
doc-ref:
  - https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md
  - https://tools.ietf.org/html/rfc8478

WiP: https://github.com/KOLANICH/kaitai_struct_formats/tree/zstd

Andor's Trail

meta:
  id: andors_trail
  title: Andor's Trail save
  application: Andor's Trail

doc: | 
  Andor's Trail is an RPG for Android

doc-ref: https://github.com/Zukero/andors-trail/blob/master/AndorsTrail/src/com/gpl/rpg/AndorsTrail/savegames/Savegames.java