Support android-arm64/Termux about httptoolkit-server

[pimterry]

btw why does the HTTP toolkit need a node-data channel? The read me says WebRtc, but why would the HTTP toolkit need webrtc support

Because HTTP Toolkit also supports WebRTC 😄. HTTP & WebRTC are very often used together, and so it's necessary if you want to be able to fully debug any P2P web applications. Similar reasoning to why we now support WebSockets.

If you want to test it out, try launching an intercepted Chrome window, opening https://webrtc.github.io/samples/src/content/datachannel/channel/ in two different tabs there, clicking 'Start', then sending some messages back and forth. You should see the connections and data channels appear in HTTP Toolkit.

It's still a bit experimental, but it should work for all simple cases and quite a few more complex scenarios, and it's really useful for pages that are communicating via WebRTC!

Looks like this package always creates issues.

The main issue here is just that the build was setup wrong I think, and so the update to include arm64 support here had to be reverted. On x86 or on M1 if you use Rosetta (which I think should work automatically?) everything should work just fine I think. The built app definitely works everywhere AFAIK, it's just local dev that's tricky.

I do want to fix this though, to make sure local dev on M1 devices works correctly. You'll be excited to hear that I've just bought an M1 Mac Mini specifically to work on this, so hopefully I should be able to bring new improvements here soon! 😄

[pimterry]

Oooh interesting. You'll probably also have issues with node-datachannel due to this revert although hopefully we should be able to fix that soon.

I'm not really sure what the error is complaining about above though. It's some issue with running install steps for @sentry/cli, but it's not clear what exactly has failed or why, as far as I can see. You may be able to fix this by just uninstalling that dependency - it's only used to send sourcemaps to Sentry for debugging later when bundling the server for deployment with webpack, it's probably not relevant for any usage outside the official package.

[Maxr1998]

Thanks for those tips!
Removing Sentry indeed helped me to get past that issue.
As expected, node-datachannel also caused issues, unfortunately just upgrading it to 0.3.6 didn't help. First, I got some warnings while building libdatachannel which caused the build to fail due to -Werror, I managed to suppress those by adding

set(CMAKE_C_FLAGS "-Wno-error=unused-but-set-variable -Wno-error=strict-prototypes")

to the CMakeLists.txt in node-datachannel.
Unfortunately, while I was able to build just libdatachannel outside a node environment, node-datachannel failed to build due to other errors:

Error log

/data/data/com.termux/files/home/apps/httptoolkit-server/node_modules/node-datachannel/src/peer-connection-wrapper.cpp:497:47: error: no matching conversion for functional-style cast from 'Napi::Number' to 'std::chrono::milliseconds' (aka 'duration<long long, ratio<1LL, 1000LL>>')
                    init.reliability.rexmit = std::chrono::milliseconds(reliability.Get("rexmit").As<Napi::Number>());
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/data/data/com.termux/files/usr/include/c++/v1/chrono:1021:28: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'Napi::Number' to 'const std::chrono::duration<long long, std::ratio<1, 1000>>' for 1st argument
class _LIBCPP_TEMPLATE_VIS duration
                           ^
/data/data/com.termux/files/usr/include/c++/v1/chrono:1021:28: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'Napi::Number' to 'std::chrono::duration<long long, std::ratio<1, 1000>>' for 1st argument
/data/data/com.termux/files/usr/include/c++/v1/chrono:1073:18: note: candidate template ignored: requirement 'is_convertible<Napi::Number, long long>::value' was not satisfied [with _Rep2 = Napi::Number]
        explicit duration(const _Rep2& __r,
                 ^
/data/data/com.termux/files/usr/include/c++/v1/chrono:1085:9: note: candidate template ignored: could not match 'duration<_Rep2, _Period2>' against 'Napi::Number'
        duration(const duration<_Rep2, _Period2>& __d,
        ^
/data/data/com.termux/files/usr/include/c++/v1/chrono:1066:9: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
        duration() = default;
        ^
1 error generated.

Sadly, I'm now at a point where I'm lacking experience to get around these myself. I am uncertain if that's the last issue either, since I'm still just trying to npm install node-datachannel independent of httptoolkit.

[pimterry]

Hmm, yes, if you can't npm install node-datachannel working at all for any versions then yes, there's going to be a problem here. There's two options I think:

• You can open an issue with node-datachannel, and ask them to provide prebuilt binaries for your os+architecture combo. When these exist, they're used automatically so there's no build step required during npm install
• Alternatively, you can set up a build environment so that node-datachannel can build successfully when installed on your device. I'm not sure what would cause the build to fail in the specific way you're showing, perhaps that's due to the specific node version you're using? HTTP Toolkit currently uses v16.16.0, which seems to work OK, but it's only building on platforms where there are prebuilds available anyway, so we don't actually run the build step ourselves.

I would test the build with some different node versions, and then if that doesn't work open an issue with node-datachannel to either expand prebuilds or help you with your build error.

[shirshak55]

@pimterry btw why does the HTTP toolkit need a node-data channel? The read me says WebRtc, but why would the HTTP toolkit need webrtc support Looks like this package always creates issues.

[Maxr1998]

Update:
With some help from the node-datachannel maintainer that issue is fixed now. The Sentry issue turned out a bit more complex, but with removing some code I can finally get the httptoolkit-server to build.

Unfortunately, npm start fails with:

u0_a584@localhost ~/a/httptoolkit-server (main)> npm start

> [email protected] start /data/data/com.termux/files/home/apps/httptoolkit-server
> node-dev ./bin/run start

(node:6550) [ERR_SYSTEM_ERROR] SystemError Plugin: httptoolkit-server [ERR_SYSTEM_ERROR]: A system error occurred: uv_interface_addresses returned Unknown system error 13 (Unknown system error 13)
module: @oclif/[email protected]                                                                       task: toCached
plugin: httptoolkit-server
root: /data/data/com.termux/files/home/apps/httptoolkit-server
See more details with DEBUG=*
(Use `node --trace-warnings ...` to show where the warning was created)
 ›   Error: command start not found
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] start: `node-dev ./bin/run start`
npm ERR! Exit status 2

That's again a known issue at termux/termux-packages#10868. Maybe you have an idea how to workaround it?

EDIT:
It can be so easy 😆
Starting the server as root makes it work. Obviously not ideal, since I don't really want to run a full node environment as root..

And even then, intercepting the response doesn't work because it doesn't trust any upstream certificates. Might because of the root environment, but I'm not sure. Termux isn't configured as intercepted in the HttpToolkit app at least.

[pimterry]

Hmmm, both of those issues (trusted certificates & os.networkInterfaces) seem like problems with Termux's node package, not something solvable here I'm afraid.

For the Unknown system error 13 issue, the server uses quite a few dependencies and internal code that needs to query the device's network interfaces, so that can't be avoided, and that seems like a straight up Termux bug. It might be possible for Termux to use dummy data for any details that aren't available (one comment there suggests the MAC address is the problem) since that probably doesn't matter in most cases? Or it could report no network interfaces at all, although that might break things depending on how that's used, I'm not sure. Running as root seems like an OK user workaround for now, but I agree that's not great.

Regarding the certificates, the server (via Mockttp) delegates this almost entirely to Node, so this means that Termux's node build isn't correctly populating the certificates there. Mockttp usually uses the values from tls.rootCertificates as the base certificates to trust, so you can test this yourself by launching node manually and running require('tls').rootCertificates.

You should also test this by using the https module directly in node - that should generally match what Mockttp is doing for upstream connections most of the time as far as I'm aware.

[Maxr1998]

Thanks for the reply. I've checked the tls.rootCertificates for both the local user and superuser (while HttpToolkit isn't running) and they seem to be identical. The error that's logged is:

Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (node:_tls_wrap:1539:34)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket._finishInit (node:_tls_wrap:953:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:734:12) {
  code: 'SELF_SIGNED_CERT_IN_CHAIN' 
}

Could it be that the injected HttpToolkit system certificate is causing this issue, even for apps that aren't intercepted? Maybe it would work to explicitly ignore the injected system cert instead of failing here.

[pimterry]

Ah, so to be clear, you're running HTTP Toolkit on an Android device, and then you're intercepting that device with HTTP Toolkit too? That will almost certainly not work by default, as you'll end up intercepting outgoing upstream traffic from HTTP Toolkit back via HTTP Toolkit itself again, and end up in a loop.

Even before that though, you'll find that the loopback connections fail with certificate errors, since HTTP Toolkit doesn't trust itself for upstream connections (and shouldn't - this shouldn't ever normally happen!). That would explain this: it's not an issue with the real upstream certificate at all, it's a trust issue with HTTP Toolkit's own self-signed certificate, which obviously isn't in Node.js's default trusted certificates.

Are you using HTTP Toolkit's Android app to handle interception? If so, you may be able to fix this by changing the settings in the app once you're connected, to disable interception for all apps except the real target you're interested in (i.e. disabling interception of Termux).

If that's not correct, can you explain more details about your setup, and exactly what you're trying to do?

[Maxr1998]

Yes, the idea is to capture the traffic from other apps running on the same device without requiring an external PC. There are other apps for Android that can do this, but those suffer from the user CA restrictions and are simply not as good as HttpToolkit.

I'm using the HttpToolkit server within Termux and use the app.httptoolkit.tech web UI in Vivaldi Browser. Then, I connect to the locally running "wireless" adb which connects to the HttpToolkit Android app and injects the system certificates. I have nearly all apps unchecked in the app's settings, including Termux and Vivaldi, so those should not be (and aren't) captured. However, the injected system certificates still cause upstream requests from the server Termux to fail. This surprises me, since those requests shouldn't be intercepted nor signed with the self-signed certificate. It seems, to me, that node doesn't trust the whole chain due to the self-signed certificate. No idea why, unfortunately.

[Maxr1998]

For the Unknown system error 13 issue, the server uses quite a few dependencies and internal code that needs to query the device's network interfaces, so that can't be avoided, and that seems like a straight up Termux bug. It might be possible for Termux to use dummy data for any details that aren't available (one comment there suggests the MAC address is the problem) since that probably doesn't matter in most cases? Or it could report no network interfaces at all, although that might break things depending on how that's used, I'm not sure. Running as root seems like an OK user workaround for now, but I agree that's not great.

If I understand the linked Termux issue correctly, it's caused by Android's privacy lockdown that prevents apps from accessing interfaces to not be able to query the hardware mac address, which could be used as a user identifier. Thus, it would be up to the app(s) to handle this correctly. I personally don't find that very realistic, and would also suggest a patch to node, but again, not really knowledgeable in that area and don't really know what the best solution would be here.