Using DuckDB with Deno about deno

[bartlomieju]

It's an internal bug in Deno, but it seems easy enough to fix it to make example from DuckDB's page work. I'll try to have it done tonight.

[bartlomieju]

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

[bartlomieju]

This is because we currently don't support "lifecycle hooks" when installing npm dependencies (eg. postinstall script). We have a PR that implements that that's WIP: #23558.

You can work around this for now if you use "bring your own node modules" - by installing using npm/yarn/pnpm.

[bartlomieju]

Thanks! I don't get a segfault, but I'm getting a panic like so:

import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 


const con = db.connect();

con.all("SELECT 42 AS fortytwo", function (err, res) {
  if (err) {
    console.warn(err);
  }
  console.log(res[0].fortytwo);
})

env DENO_FUTURE=1 deno run -A index.ts

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.43.1+998036b
Args: ["deno", "run", "-A", "index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2348:63:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'll try to debug this one as well.

[MaximePawlakFr]

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

I've just update to the last canary: deno 1.43.1+998036b
I still get the same segmentation fault.

Keep me in the loop if you make any advance.

Thanks a lot for what you tried.

[nshiab]

Thank you!

[MaximePawlakFr]

Hi @nshiab , have you succeeded?

I tried but I got this error:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

My data/index.ts contains:

import duckdb from "duckdb";
console.log(duckdb)
const db = new duckdb.Database(":memory:");
console.log(db)

The steps are:

• setup the project with npm: npm init
• npm install duckdb
• try to run: deno run --allow-all ./data/index.ts

error: Uncaught (in promise) Error: Cannot find module 'MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/binding/duckdb.node'
Require stack:
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb.js
- MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb.js
    at Function.Module._resolveFilename (node:module:622:15)
    at Function.Module._load (node:module:500:27)
    at Module.require (node:module:675:19)
    at require (node:module:789:16)
    at Object.<anonymous> (file://MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file://MY_PATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:731:34)
    at Object.Module._extensions..js (node:module:745:10)
    at Module.load (node:module:656:32)
    at Function.Module._load (node:module:537:12)

• then move to duckdb: cd node_modules/duckdb
• install it: npm install
• I can see the missing duckdb.node under node_modules > duckdb > lib > binding > duckdb.node
• run again: deno run --allow-all ./data/index.ts
• and finally get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

By adding RUST_BACKTRACE=1, I get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: core::result::Result<T,E>::unwrap
   4: _napi_set_element
   5: __ZNK4Napi6Object3SetIA11_cEEbjRKT_
   6: __ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE
   7: __ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE
   8: _napi_register_module_v1
   9: deno_napi::op_napi_open<NP>::v8_fn_ptr
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

And by adding RUST_BACKTRACE=full, I get:

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
stack backtrace:
   0:        0x1018f46b0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hade97c44b56fc870
   1:        0x100bcd6b0 - core::fmt::write::h81cbefbffc581dab
   2:        0x101901918 - std::io::Write::write_fmt::h125c60058ebfe43c
   3:        0x1018f4508 - std::sys_common::backtrace::print::hfa54be0dd0cf5860
   4:        0x10190838c - std::panicking::default_hook::{{closure}}::h4235e0929057f079
   5:        0x1019080d0 - std::panicking::default_hook::hcf67171e7c25be94
   6:        0x100b05c18 - deno::setup_panic_hook::{{closure}}::h71a800dcbf73da30
   7:        0x101908a2c - std::panicking::rust_panic_with_hook::h1767d40d669aa9fe
   8:        0x1018f49e8 - std::panicking::begin_panic_handler::{{closure}}::h83ff281d56dc913c
   9:        0x1018f4954 - std::sys_common::backtrace::__rust_end_short_backtrace::h2f399e8aa761a4f1
  10:        0x1019086b4 - _rust_begin_unwind
  11:        0x100bd78cc - core::panicking::panic_fmt::hc32404f2b732859f
  12:        0x100bd16f0 - core::result::unwrap_failed::h2ea3b6e22f1f6a7c
  13:        0x1007888a0 - core::result::Result<T,E>::unwrap::h2d955a3171e3fa0a
  14:        0x10078f060 - _napi_set_element
  15:        0x11a7719c0 - __ZNK4Napi6Object3SetIA11_cEEbjRKT_
  16:        0x11a7710a4 - __ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE
  17:        0x11a7728a0 - __ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE
  18:        0x11a771f38 - _napi_register_module_v1
  19:        0x101408960 - deno_napi::op_napi_open<NP>::v8_fn_ptr::h315a352275b5cff8

Have you managed to use duckdb in Deno?
What could I do?

Best

[MaximePawlakFr]

--Update--

I've upgraded to deno 1.43.1 (was 1.40.4), and now I get this error:

error: Uncaught (in promise) Error: Error in native callback
    at Object.Module._extensions..node (node:module:704:20)
    at Module.load (node:module:597:32)
    at Function.Module._load (node:module:484:12)
    at Module.require (node:module:609:19)
    at require (node:module:715:16)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/[email protected]/node_modules/duckdb/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:659:34)
    at Object.Module._extensions..js (node:module:673:10)
    at Module.load (node:module:597:32)

Do you have any idea?

[nshiab]

Hi @MaximePawlakFr!

For now, I am using node/npm. But @bartlomieju is working on it #23558.

[MaximePawlakFr]

Thank you Bartek.

I've just updated deno to deno 1.43.1+cbb78e1.

I tried again. I can load duckdb without the previous error.

But I can not run any request to duckdb. I get a segmentation fault.

Here is my code:

import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 
console.log(db);

Deno.serve(async (req) => {

    const request =
      "SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;";
    console.log(request);

    const result = await new Promise((resolve, reject) => {
      db.all(request, function (err, res) {
        console.log(res);
        if (err) {
          console.warn(err);
          reject(err);
        }
        resolve(res);
      });
    });

    const res = JSON.stringify(
      result,
      (key, value) => typeof value === "bigint" ? value.toString() : value, // return everything else unchanged
    );

    return new Response(
      res,
      {
        headers: { "Content-Type": "application/json" },
      },
    );
});

I run deno run --allow-all index.ts

I see in my terminal:

Database {}
Listening on http://localhost:8000/
SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;
zsh: segmentation fault  deno run --allow-all index.ts

It looks like something went wrong with duckdb. Any idea what?

Regards

[bartlomieju]

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

[nshiab]

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

Unfortunately, I am on a M1 Mac, so I can't install canary versions, from what I gathered.

[bartlomieju]

@nshiab you can, we've been shipping canary version for ARM Macs for a couple months now :)

[nshiab]

@bartlomieju What am I doing wrong? 😢

Thank you so much for your help!

[bartlomieju]

It appears that deno binary you have locally on your system was installed via homebrew. You should remove it using homebrew and then install it again using the script, then add it to you PATH.

[nshiab]

You are right. Stupid mistake on my end. Sorry.

Unfortunately, it doesn't seem to work on my laptop. Should I do something differently?

[bartlomieju]

@nshiab the problem with postinstall scripts is not solved - you still need to use npm install and use DENO_FUTURE=1 env var, I fixed the problem related to actual execution of DuckDB.

[nshiab]

Hmmm. I tried again but I am still doing something wrong I guess. It's probably best if I wait for postinstall scripts to work.

[bartlomieju]

@nshiab you need to specify DENO_FUTURE env var in your terminal when you run the deno command:

# bash 
DENO_FUTURE=1 deno run -A index.ts
# fish
env DENO_DUTURE=1 deno run -A index.ts

[nshiab]

IT WORKS! Amazing! Thanks for your patience @bartlomieju!

But... if I actually run a query, it fails. This is an example from the duckdb repo.