Trouble extracting patch_d2.mpq about diablo2

[miffels]

Curious - installing with Yarn does work for me, as opposed to NPM. Looking good!

[blacha]

Thanks for reporting this, looks like this was caused by a couple of bugs.

mpq.exists(fileName) was always returning true 613888f

and uncompressed data was not being extracted correctly 25edc0c

Let me know if these fix the issues.

[miffels]

Thanks! Couldn't build it locally for some reason so I did some awful surgery I'd rather not comment, but the mpq bit appears to be passing now. Couldn't quite get the bintools to work, but that may be due to a screw-up on my part. Now I just gotta figure out what to do with those buffers... ;) Loading some of text files from data/global/excel and dumping it as utf-8 resulted in a heap of garbled characters, but it may just be that I need to take a closer look at the specs (or your code :))

[blacha]

Couldn't build it locally for some reason

What was the error you get with building locally? Is it related to installing pcap ?

Couldn't quite get the bintools to work,

What sort of problems were you having there?

I have done most of my testing on Path of Diablo/Project Diablo 2 MPQ's so those are v1.13 based I think.

Ill see if I can dig up a 1.14 mpq and see what happens

[blacha]

I think I've found the problem...

My earlier fix was just plain wrong :) the v1.14 example patch_d2.mpq I just found is compressed with isImplode being set I dont handle isImplode https://github.com/blacha/diablo2/blob/master/packages/mpq/src/mpq.ts#L132

Once I added isImplode logic the tbl files extracted fine, however they are version 1 not version 0 so Ill need to add some logic for version 1 tbl Ill look to do that tonight if I get some time.

[blacha]

I have fixed the MPQ implode handling f0e50d7

I have also fixed the issue with v1 .tbl turns out the index count and hash table count of the files differed (All the MPQs I have had the same numbers) 77d2f85

Ive also added some test MPQs with tests so hopefully it should just work going forward .

If you could check if it works for you, Ill then version bump and publish a new copy.

[miffels]

Holy cow, you're on fire :)

Sorry for not getting back earlier. I am based in EU so the night got in the way ;)

For reference, this is the install issue I am facing:

$ git status && npm i && npm run build
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        package-lock.json

nothing added to commit but untracked files present (use "git add" to track)
npm WARN @octokit/[email protected] requires a peer of @octokit/core@>=3 but none is installed. You must install peer dependencies yourself.

added 2 packages from 1 contributor, removed 2 packages and audited 948 packages in 4.737s
found 0 vulnerabilities


> @diablo2/[email protected] build ...\workspaces\dev\diablo2
> tsc -b

packages/mpq/src/header.ts:1:32 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp, StrutInfer } from 'binparse';
                                 ~~~~~~~~~~

packages/bintools/src/lang/lang.reader.ts:1:32 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp, StrutInfer } from 'binparse';
                                 ~~~~~~~~~~

packages/bintools/src/monster/monster.stat.reader.ts:1:32 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp, StrutInfer } from 'binparse';
                                 ~~~~~~~~~~

packages/bintools/src/item/item.reader.ts:1:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/bintools/src/item/item.reader.ts:2:38 - error TS2307: Cannot find module 'binparse/build/src/string' or its corresponding type declarations.

2 import { StrutTypeStringFixed } from 'binparse/build/src/string';
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~

packages/bintools/src/item/item.reader.ts:3:54 - error TS2307: Cannot find module 'binparse/build/src/type' or its corresponding type declarations.

3 import { StrutParserContext, StrutParserInput } from 'binparse/build/src/type';
                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~

packages/bintools/src/item/item.reader.ts:15:37 - error TS2554: Expected 0 arguments, but got 1.

15 const ItemCode = new ItemCodeParser(4);
                                       ~

packages/bintools/src/mpq.loader.ts:1:44 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Diablo2Mpq, Diablo2MpqData } from '@diablo2/data';
                                             ~~~~~~~~~~~~~~~

packages/bintools/src/mpq.loader.ts:2:21 - error TS2307: Cannot find module '@diablo2/mpq' or its corresponding type declarations.

2 import { Mpq } from '@diablo2/mpq';
                      ~~~~~~~~~~~~~~

packages/bintools/src/mpq.loader.ts:3:39 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

3 import { StrutInfer, StrutType } from 'binparse';
                                        ~~~~~~~~~~

packages/packets/src/packet.ts:2:73 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { bp, StrutAny, StrutInfer, StrutParserContext, StrutType } from 'binparse';
                                                                          ~~~~~~~~~~

packages/packets/src/factory.ts:2:36 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { StrutParserContext } from 'binparse';
                                     ~~~~~~~~~~

packages/packets/src/packets-pod/data.ts:1:77 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Act, Attribute, Difficulty, PlayerClass, UnitType, WarpType } from '@diablo2/data';
                                                                              ~~~~~~~~~~~~~~~

packages/packets/src/packets-pod/data.ts:2:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/packets/src/packets-pod/data.ts:11:61 - error TS7006: Parameter 'id' implicitly has an 'any' type.

11 export const DataAttribute = bp.lookup('Attribute', bp.u8, (id) => Attribute[id] ?? 'Unknown');
                                                               ~~

packages/packets/src/packets-pod/client.ts:2:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/packets/src/parser.item.ts:1:103 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Diablo2Mpq, ItemActionType, ItemCategory, ItemContainer, ItemDestination, ItemQuality } from '@diablo2/data';
                                                                                                        ~~~~~~~~~~~~~~~

packages/packets/src/parser.item.ts:2:74 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { BitStream, bp, StrutBase, StrutInfer, StrutParserContext } from 'binparse';
                                                                           ~~~~~~~~~~

packages/packets/src/parser.npc.ts:1:28 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Diablo2Mpq } from '@diablo2/data';
                             ~~~~~~~~~~~~~~~

packages/packets/src/parser.npc.ts:2:69 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

2 import { BitStream, bp, StrutBase, StrutParserContext, toHex } from 'binparse';
                                                                      ~~~~~~~~~~

packages/packets/src/parser.npc.ts:47:53 - error TS2339: Property 'name' does not exist on type 'DataTypeNpc'.

47     if (packetId !== 0xac) throw new Error(`[${this.name}] Invalid packet: ${toHex(packetId)}`);
                                                       ~~~~

packages/packets/src/packets-pod/server.ts:3:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

3 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/packets/src/packets-pd2/client.ts:1:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/packets/src/packets-pd2/server.ts:1:20 - error TS2307: Cannot find module 'binparse' or its corresponding type declarations.

1 import { bp } from 'binparse';
                     ~~~~~~~~~~

packages/packets/src/packets.ts:1:32 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Diablo2Version } from '@diablo2/data';
                                 ~~~~~~~~~~~~~~~

packages/packets/src/packets-pod/__tests__/item.test.ts:2:76 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

2 import { ItemActionType, ItemCategory, ItemDestination, ItemQuality } from '@diablo2/data';
                                                                             ~~~~~~~~~~~~~~~

packages/packets/src/packets-pod/__tests__/npc.test.ts:1:28 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Diablo2Mpq } from '@diablo2/data';
                             ~~~~~~~~~~~~~~~

packages/packets/src/packets-pod/__tests__/packet.test.ts:1:21 - error TS2307: Cannot find module '@diablo2/data' or its corresponding type declarations.

1 import { Act } from '@diablo2/data';
                      ~~~~~~~~~~~~~~~


Found 28 errors.

I imagine @diablo2/data is related to some missing MPQ data dump for a few components. binparse may just be missing from the dependencies, perhaps?

[blacha]

Ahh, This project is a monorpepo and uses yarn workspaces and lerna to manage all the packages.

looks like workspaces was only added to npm v7, so you can upgrade to npm v7 or just use yarn

> npm --version
7.6.0
> ⟩ npm i && npm run build
npm WARN deprecated @types/[email protected]: This is a stub types definition. p-limit provides its own type definitions, so you do not need this installed.
npm WARN deprecated [email protected]: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142

added 1095 packages, and audited 1104 packages in 27s

55 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> @diablo2/[email protected] build
> tsc -b

[miffels]

Sorry for not getting back earlier, had to time to take a look this weekend. So I figure there are other issues related to Windows here, but thanks for pointing me top npm v7. I used it somewhere before, but only did so much with monorepos so I might just have forgotten how it worked... :)

In Windows WSL, there are a few extra things to install:

1. Make
2. g++

Then the install fails at

npm ERR! make: Warning: File 'pcap_binding.target.mk' has modification time 0.44 s in the future
npm ERR! ../pcap_binding.cc:2:10: fatal error: pcap/pcap.h: No such file or directory
npm ERR!     2 | #include <pcap/pcap.h>
npm ERR!       |          ^~~~~~~~~~~~~
npm ERR! compilation terminated.

Which appears to be a known issue with Windows. I'll see later if any of those workarounds there help.

[blacha]

Sorry missed the response,

PCAP is a bit of a pain to build, its only required if you want to sniff diablo2 network traffic, so I have removed it from the main buildpipeline.

I have also added a windows build to the CI system so hopefully window users wont have issues going forward.

[blacha]

@miffels If your ok with this, I am going to close the ticket and make a new release. I believe your problem has been fixed.

[miffels]

Hey, sorry for not getting back earlier - I appreciate your support and am sorry to say that I got somewhat sidetracked by Valheim, among other things... :) Don't have access to my home PC atm but tried running the build on a Windows VM. A few setup notes:

> choco list --localonly
Chocolatey v0.10.15
chocolatey 0.10.15
chocolatey-core.extension 1.3.5.1
chocolatey-dotnetfx.extension 1.0.1
chocolatey-visualstudio.extension 1.9.0
chocolatey-windowsupdate.extension 1.0.4
dotnetfx 4.8.0.20190930
git 2.31.0
git.install 2.31.0
KB2919355 1.0.20160915
KB2919442 1.0.20160915
KB2999226 1.0.20181019
KB3033929 1.0.5
KB3035131 1.0.3
make 4.3
nodejs 15.12.0
nodejs.install 15.12.0
python 3.9.2
python3 3.9.2
vcredist140 14.28.29913
vcredist2015 14.0.24215.20170201
visualstudio-installer 2.0.1
visualstudio2019-workload-nativedesktop 1.0.1
visualstudio2019buildtools 16.9.1.0
23 packages installed.

Also had to install VS 2019 Community Edition + Desktop Development with C++ separately via the MS manual installer, for some reason, despite setting npm config set msvs_version 2019 and installing visualstudio2019buildtools and visualstudio2019-workload-nativedesktop via choco. Now I am back at

> pwd

Path
----
C:\Users\...\workspaces\other\diablo2
> npm -v
7.6.3
> npm i
npm WARN deprecated @types/p-limit@2.2.0: This is a stub types definition. p-limit provides its own type definitions, so you do not need this installed.
npm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code 1
npm ERR! path C:\Users\...\workspaces\other\diablo2\node_modules\pcap
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
npm ERR!   pcap_binding.cc
npm ERR!   pcap_session.cc
npm ERR! C:\Users\...\workspaces\other\diablo2\node_modules\pcap\pcap_binding.cc(2,10): fatal error C1083: Cannot open include file: 'pcap/pcap.h': No such file or directory [C:\Users\...\workspaces\other\diablo2\node_modules\pcap\build\pcap_binding.vcxproj]
npm ERR! C:\Users\...\workspaces\other\diablo2\node_modules\pcap\pcap_session.cc(2,10): fatal error C1083: Cannot open include file: 'pcap/pcap.h': No such file or directory [C:\Users\...\workspaces\other\diablo2\node_modules\pcap\build\pcap_binding.vcxproj]
npm ERR!   win_delay_load_hook.cc
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@5.1.1
npm ERR! gyp info using node@15.12.0 | win32 | x64
npm ERR! gyp info find Python using Python version 3.9.2 found at "C:\Python39\python.exe"
npm ERR! gyp info find VS using VS2019 (16.9.31112.23) found at:
npm ERR! gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Python39\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   'C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\node-gyp\\gyp\\gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'msvs',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\pcap\\build\\config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\node-gyp\\addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   'C:\\Users\\...\\AppData\\Local\\node-gyp\\Cache\\15.12.0\\include\\node\\common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=C:\\Users\\...\\AppData\\Local\\node-gyp\\Cache\\15.12.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\...\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\15.12.0\\\\<(target_arch)\\\\node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\pcap',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\pcap\\build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info spawn C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   'build/binding.sln',
npm ERR! gyp info spawn args   '/clp:Verbosity=minimal',
npm ERR! gyp info spawn args   '/nologo',
npm ERR! gyp info spawn args   '/p:Configuration=Release;Platform=x64'
npm ERR! gyp info spawn args ]
npm ERR! gyp ERR! build error
npm ERR! gyp ERR! stack Error: `C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onExit (C:\Users\...\workspaces\other\diablo2\node_modules\node-gyp\lib\build.js:194:23)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:369:20)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
npm ERR! gyp ERR! System Windows_NT 10.0.19042
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\...\\workspaces\\other\\diablo2\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\Users\...\workspaces\other\diablo2\node_modules\pcap
npm ERR! gyp ERR! node -v v15.12.0
npm ERR! gyp ERR! node-gyp -v v5.1.1
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\...\AppData\Local\npm-cache\_logs\2021-03-18T11_19_33_519Z-debug.log

Can you confirm that commit 6309f29 (master HEAD as of now) should not include pcap?

[blacha]

Can you confirm that commit 6309f29 (master HEAD as of now) should not include pcap?

pcap is not mentioned anywhere in the yarn.lock so there should not be any pcap.

I just cloned a fresh copy and ran yarn and no pcap was installed.

> ls node_modules/pcap
ls: cannot access 'node_modules/pcap': No such file or directory