Coder Social home page Coder Social logo

kaitai_struct_javascript_runtime's Introduction

Kaitai Struct: runtime library for JavaScript

npm@latest npm@next downloads

This library implements Kaitai Struct API for JavaScript.

Kaitai Struct is a declarative language used for describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc.

Further reading:

Licensing

Copyright 2012-2016 Ilmari Heikkinen
Copyright 2016-2022 Kaitai Project

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

kaitai_struct_javascript_runtime's People

Contributors

cherue avatar generalmimon avatar greycat avatar koczkatamas avatar mfalkvidd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kaitai_struct_javascript_runtime's Issues

Entires limit 50

Hello.
I am using kaitai to structure iso file using this article (javascript). But I have a problem with a folder that has more than 50 files. If i use code like
var parsedIso = new Iso9660(new KaitaiStream(arrayBuffer));
const myDir = parsedIso.primaryVolDesc.volDescPrimary.rootDir.body.extentAsDir.entries;
console.log(myDir.length)
It is returns only first 50 files, but i have more files in directory. Other directories are good.

Please, any help?

transitionally support js to be loaded as browser module

Propblem:

Error loading KaitaiStream.js or any compiled output js as browser module.

Steps to Reproduce:

  1. Load KaitaiStream.js and some format js (e,g, Ico.js, compiled with kaitai-struct-compiler for Windows) as browser module, like this
import './node_modules/kaitai-struct/KaitaiStream.js'; /* global KaitaiStream */
import './formats/Ico.js'; /* global Ico */

Expected Result:

No loading error, exported variables can be accessed through the global object.

Actual Result:

KaitaiStream.js:9 Uncaught TypeError: Cannot set properties of undefined (setting 'KaitaiStream')
at KaitaiStream.js:9:23
at KaitaiStream.js:11:2
Uncaught TypeError: Cannot read properties of undefined (reading 'KaitaiStream')
at Ico.js:9:29
at Ico.js:11:2

Additional Information:

  • kaitai-struct-compiler for Windows v0.9
  • kaitai-struct javascript runtime 0.9.0
  • Chrome 101

By using official latest UMD template like returnExportsGlobal.js this problem can be solved.

Currently I get over it by modifying js code (the UMD wrapper part), replacing this with typeof self === 'object' ? self : this.

BigInt support

BigInts are now a thing in JS. They're supported in modern browsers and Node 10.4 and higher.

From MDN:

BigInt is similar to Number in some ways, but also differs in a few key matters — it cannot be used with methods in the built-in Math object and cannot be mixed with instances of Number in operations; they must be coerced to the same type.

I think this would mean two runtimes and two compiler targets. Because the KS expression language allows math that gets translated into JS and all the read integers must be the same type, either BigInt or Number, no mixing allowed.
And on the compiler side if there is an expression like offset + 4 we need to know at compile time whether to do 4 (Number) or 4n (BigInt).

Another option with one runtime and target would be to wrap everything that could turn into a number with a function that turns numbers into bigints if required. Seems like it wouldn't be very readable, might work with a very short name though.

bitwise integer for 32bits returns signed value as default instead of unsigned value

Hello,
I recently discovered that the bitwise integers behave differently for 32 bits
Following is my ksy

meta:
  id: testme
  endian: be

seq:
  - id: something
    type: something
types:
  something:
    seq:
      - id: val
        type: b32

if I pass in c0 00 00 02 it evaluates to -1073741822 rather than the unsigned value 3221225474

b16 works as expected. passing in c0 00 evaluates to 49152

processZlib uncompresses incorrect byterange

In this line

var b = new Buffer(buf.buffer);

the buffer to inflate is retrieved from Uint8Array from buf.buffer, but that property returns the whole buffer, ignoring byteOffset and length. Thus when only part of a file is compressed and only that slice is to be inflated, processZlib will instead try to inflate the whole file.
It should be fine to create the buffer with "var b = new Buffer(buf)"

Add package.json

As of right now this runtime is impossible to install using npm. Adding a package.json file will allow npm to install it straight from the Github repo.

Even better would be to also publish the runtime to npm as well.

Different EOF error exceptions

Currently, in JavaScript runtime, reading more bytes than there are in the stream using readBytes(...) results in KaitaiEOFError being thrown.
However, doing the same (i.e. reading past end of buffer) with other primitive reading functions, such as readU4le(), results in RangeError: Offset is outside the bounds of the DataView.

Shall we make it straight and uniform and fix it to adhere to the same standard (for example, RangeError everywhere)?

Open large binary file in kaitai struct for javascript?

The current javascript implementation requires reading a whole file into memory (ArrayBuffer). This is not good if I want to read a large file, for example a very big zip file. Can you add an option to use file descriptor (returned by fs.openSync) instead of ArrayBuffer?

Wrong result with u8

The last byte of u8 is missing.

Example struct:

meta:
  id: test
  endian: le

seq:
  - id: uu64
    type: u8

example data:

"\xf7\x30\x8a\x1e\xce\xf2\xde\xe9\x12\x5f"

result:

  • 0xE9DEF2CE1E8A3000

expected result:

  • 0xE9DEF2CE1E8A30F7

Bit-sized integer decoding invalid results

Hey, recently, I encountered a bug in reading bit-sized integers. In the below example, there are 3 fields:

  • prefix: 4-bit integer,
  • value: 32-bit integer,
  • suffix: 4-bit integer.

Here is the binary representation of a frame (5 bytes):

prefix                 value                suffix
[1000][0111 01110101 00001001 10010001 0001][0000]

Expected values for fields are as follows:

  • prefix: 8,
  • value: 2001770769,
  • suffix: 0.

When I tried to decode the frame using Kaitai (javascript library [email protected]), results were not correct (value field is decoded as 122722577).

Here is the Kaitai template I’m using for an example:

meta:
  id: example
  endian: be
seq:
  - id: prefix
    type: prefix
  - id: value
    type: value
  - id: suffix
    type: suffix
types:
  prefix:
    seq:
      - id: value
        type: b4
  value:
    seq:
      - id: value
        type: b32
  suffix:
    seq:
      - id: value
        type: b4

I also tried to use Kaitai Web IDE, and the results are the same (invalid).

After debugging the code of the library, I think the problem is here (KaitaiStream.prototype.readBitsIntBe method):

  1. When reading value field (the 2nd field in the frame), there are still 4 bits left in this.bits from the previous byte, and it reads 4 next bytes from the buffer to get required 32 bits in total.
  2. Within the for loop, it writes each byte into this.bits.
  3. First 3 iterations works ok, where it increases this.bits size up to 28 bits (starting from 4 bits, it adds 8 bits of every next byte).
  4. Then, when 4th byte is read, its increases this.bits size up to 36 bits.

It seems, like when this.bits is increased over 32 bits, it looses most significant bits (in that case 4 first bits from the left).

There is the corresponding code with comments:

KaitaiStream.prototype.readBitsIntBe = function(n) {
  // JS only supports bit operations on 32 bits
  if (n > 32) {
    throw new Error(`readBitsIntBe: the maximum supported bit length is 32 (tried to read ${n} bits)`);
  }
  var bitsNeeded = n - this.bitsLeft;
  if (bitsNeeded > 0) {
    // 1 bit  => 1 byte
    // 8 bits => 1 byte
    // 9 bits => 2 bytes
    var bytesNeeded = Math.ceil(bitsNeeded / 8);
    var buf = this.readBytes(bytesNeeded);			// [1 point]
    for (var i = 0; i < bytesNeeded; i++) {
      this.bits <<= 8;								// [2-4 points]
      this.bits |= buf[i];							// [2-4 points]
      this.bitsLeft += 8;
    }
  }

	... // the rest is not relevant
};

At 4th iteration of the loop, it should put 4 first bits of the last byte, so instead of this.bits <<= 8, it should do this.bits <<= 4, as left capacity is 4 bits, so we won’t exceed 32 bits.

Here is this.bits value, that is held during the for loop iteration, when we decode value field:

  • 00000000 00000000 00000000 00000111 [initial - before entering the loop - left part from previous field]
  • 00000000 00000000 00000111 01110101 [after 1st iteration]
  • 00000000 00000111 01110101 00001001 [after 2nd iteration]
  • 00000111 01110101 00001001 10010001 [after 3rd iteration]
  • 01110101 00001001 10010001 00010000 [after 4th iteration] (but should be 01110111 01010000 10011001 00010001).

Im not a javascript expert, but it seems like this.bits is able to hold only 32 bits max (not 64 bits).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.