httptoolkit / httptoolkit-server

The backend of HTTP Toolkit

HTTP Toolkit Server

This repo contains the backend for HTTP Toolkit, a beautiful, cross-platform & open-source HTTP(S) debugging proxy, analyzer & client.

Looking to file bugs, request features or send feedback? File an issue or vote on existing ones at github.com/httptoolkit/httptoolkit.

What is this?

HTTP Toolkit runs everything possible within the web UI, written as a standard single-page web application. There's a couple of necessary things you can't do in a web application though, especially:

• Start a locally running proxy server (here using Mockttp)
• Launch local applications preconfigured for interception

This server exposes an API that is used by the web UI, exposing these actions and some other related information - see src/api/rest-api.ts for the full API details.

This server is runnable standalone as a CLI using oclif, or can be imported into other modules to be run programmatically. The available interceptors are defined in src/interceptors, and some of these also use other services in here, e.g. src/cert-check-server.ts automatically checks if a certificate is trusted by a browser client, and downloads or installs (depending on the client) the certificate if not.

Note that the set of interceptors available in HTTP Toolkit depends on both the interceptors available on your server and the interceptors defined in the UI - new interceptors will need to be added to both.

This server is typically used by users via httptoolkit-desktop, which builds the server and web UI into an electron application, and starts & stops the server in the background whenever that app is run. Each time the desktop app is built, a new electron app is created containing the latest release from this repo.

Once the server has installed it automatically updates in the background periodically, pulling new releases from the github releases of this repo.

Contributing

If you want to add new interceptors, change/fix existing interceptor behaviour (but not their UI) or reconfigure how the underlying proxy server is set up, then you're in the right place 👍.

To get started:

• Clone this repo.
• npm install
• npm start
• A Mockttp standalone server will start on port 45456, and a graphql management server on 45457.
• Either make requests to the servers by hand, use the production UI by opening https://app.httptoolkit.tech in a Chromium-based browser, or start and use a local UI by:
  • Setting up the web UI locally (see httptoolkit/httptoolkit-ui#contributing).
  • Running npm run start:web there to start the UI without its own server.
  • Opening http://local.httptoolkit.tech:8080 in a Chromium-based browser

A few things to be aware of:

• If you're looking to add a new interceptor, those also need to be registered in src/interceptors/index.ts, and will also need to be added to the UI to make that available.
• Tests (both unit & integration) can be run with npm test, or npm run test:unit/npm run test:integration to run just the unit/integration tests.
• Note that the integration tests assume the required applications are installed and some docker images are already pulled. See ci.yml and the build-base image for an example of how to set this up (or just run the tests, look at the errors, and install whatever's missing).
• If running the server in serious use (self-hosting, or long-term ongoing development) you probably want to set a HTK_SERVER_TOKEN env var with a random key, and pass this similarly to the UI as an authToken=<token> URL parameter. This is useful because the API is very powerful (it can launch arbitrary applications on your machine). The API only listens on localhost and blocks CORS requests, so strictly speaking this shouldn't be necessary, but it is useful as a stronger guarantee & defense in depth. This is handled automatically in the production desktop app.

httptoolkit-server's Issues

Upstream proxy support

Could a method to set an upstream proxy server be added? Either through the CLI or with an environment variable?

Interceptor for brave

Is it possible to intercept brave the same way chrome is being intercepted - and launch it in place of chrome

Certificate injection failure in Android 14

I'm using Android 14 and want to inject a custom CA certificate
The device is rooted using magisk, so for a smooth operation that will be performed every restart, I put the following code in service.sh

API=`getprop ro.build.version.sdk`
if [ "$API" -ge 33 ]; then
while [ "$(getprop sys.boot_completed)" != 1 ];
do
   sleep 1
done
if [ -d "/apex/com.android.conscrypt/cacerts" ]; then
     cp /apex/com.android.conscrypt/cacerts/* /data/local/tmp/htk-ca-copy/
else
     cp /system/etc/security/cacerts/* /data/local/tmp/htk-ca-copy/
fi

# Create the in-memory mount on top of the system certs folder
su -mm -c mount -t tmpfs tmpfs /system/etc/security/cacerts

# Copy the existing certs back into the tmpfs mount, so we keep trusting them
mv /data/local/tmp/htk-ca-copy/* /system/etc/security/cacerts/

# Copy our new cert in, so we trust that too
cp /system/etc/cacustom/* /system/etc/security/cacerts/

# Update the perms & selinux context labels, so everything is as readable as before
chown root:root /system/etc/security/cacerts/*
chmod 644 /system/etc/security/cacerts/*
chcon u:object_r:system_file:s0 /system/etc/security/cacerts/*

echo 'System cacerts setup completed'

           # Create a separate temp directory, to hold the current certificates
             # Without this, when we add the mount we can't read the current certs anymore.

             # Deal with the APEX overrides in Android 14+, which need injecting into each namespace:
             if [ -d "/apex/com.android.conscrypt/cacerts" ]; then
              

                 # When the APEX manages cacerts, we need to mount them at that path too. We can't do
                 # this globally as APEX mounts are name-spaced per process, so we need to inject a
                 # bind mount for this directory into every mount namespace.

                 # First we get the Zygote process(es), which launch each app
                 ZYGOTE_PID=$(pidof zygote || true)
                 ZYGOTE64_PID=$(pidof zygote64 || true)
                 Z_PIDS="$ZYGOTE_PID $ZYGOTE64_PID"
                 # N.b. some devices appear to have both, some have >1 of each (!)

                 # Apps inherit the Zygote's mounts at startup, so we inject here to ensure everything is new
                 # started apps will see these certs straight away:
                 for Z_PID in $Z_PIDS; do
                     if [ -n "$Z_PID" ]; then
                         nsenter --mount=/proc/$Z_PID/ns/mnt -- \
                             /bin/mount --bind /system/etc/security/cacerts /apex/com.android.conscrypt/cacerts
                     fi
                 done

              

                 # Then we inject the mount into all already running apps, so they see these certs immediately.

                 # Get the PID of every process whose parent is one of the Zygotes:
                 APP_PIDS=$(
                     echo $Z_PIDS | \
                     xargs -n1 ps -o 'PID' -P | \
                     grep -v PID
                 )

                 # Inject into the mount namespace of each of those apps:
                 for PID in $APP_PIDS; do
                     nsenter --mount=/proc/$PID/ns/mnt -- \
                         /bin/mount --bind /system/etc/security/cacerts /apex/com.android.conscrypt/cacerts &
                 done
                 wait # Launched in parallel - wait for completion here

               
             fi

It really seems that the personalized certificates enter the system but after a few seconds the device crashes (restarts)
This is the log at the moment of the crash

 !@*** FATAL EXCEPTION IN SYSTEM PROCESS: pool-246-thread-1
                                                                                                    java.lang.AssertionError
                                                                                                    	at com.android.okhttp.OkHttpClient.getDefaultSSLSocketFactory(OkHttpClient.java:649)
                                                                                                    	at com.android.okhttp.OkHttpClient.copyWithDefaults(OkHttpClient.java:605)
                                                                                                    	at com.android.okhttp.OkUrlFactory.open(OkUrlFactory.java:63)
                                                                                                    	at com.android.okhttp.OkUrlFactory.open(OkUrlFactory.java:58)
                                                                                                    	at com.android.okhttp.HttpHandler.openConnection(HttpHandler.java:56)
                                                                                                    	at java.net.URL.openConnection(URL.java:1006)
                                                                                                    	at com.android.server.location.gnss.GnssPsdsDownloader.doDownload(GnssPsdsDownloader.java:165)
                                                                                                    	at com.android.server.location.gnss.GnssPsdsDownloader.doDownloadWithTrafficAccounted(GnssPsdsDownloader.java:152)
                                                                                                    	at com.android.server.location.gnss.GnssPsdsDownloader.downloadPsdsData(GnssPsdsDownloader.java:129)
                                                                                                    	at com.android.server.location.gnss.GnssLocationProvider.lambda$handleDownloadPsdsData$6(GnssLocationProvider.java:909)
                                                                                                    	at com.android.server.location.gnss.GnssLocationProvider.$r8$lambda$JL2e1UXaMzOyFO9jVFmVGlpGR4o(GnssLocationProvider.java:0)
                                                                                                    	at com.android.server.location.gnss.GnssLocationProvider$$ExternalSyntheticLambda17.run(R8$$SyntheticClass:0)

HttpToolkit generating and not cleaning outputs

Hello,

I've been using HttpToolkit for couple of months now and my diskspace started to shrink. I found out, there are blobs, that are not cleaned, even though I am pretty sure I cannot see any archive of requests.

I am not sure if this helps, but here is where I found them.

2,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/59
2,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/66
4,7G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/57
6,6G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/6f
1,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/9b
1,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/0b
5,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/60
1,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/5f
1,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/a3
1,2G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/e2
6,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/fc
2,3G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/fb
3,4G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/74
7,8G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/4c
3,5G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/4d
2,6G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/07
2,2G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/38
7,6G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/6e
2,9G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/36
3,9G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/5d
2,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/5e
1,9G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/01
1,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/06
5,4G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/6c
2,7G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/52
5,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/64
1,3G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/a0
1,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/b8
1,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/d2
1,8G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/b7
1,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/c3
1,1G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/c4
2,0G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/2c
5,5G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/70
8,8G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/4f
2,2G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2/71
TOTAL
137G	~/Library/Application Support/httptoolkit/IndexedDB/https_app.httptoolkit.tech_0.indexeddb.blob/2

Thanks :)

Support android-arm64/Termux

It would be cool to be able to run the httptoolkit server directly on device inside Termux.
Right now, npm install fails with the following trace:

1458 verbose stack Error: command failed
1458 verbose stack     at ChildProcess.<anonymous> (/data/data/com.termux/files/usr/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/lib/index.js:63:27)
1458 verbose stack     at ChildProcess.emit (node:events:513:28)
1458 verbose stack     at maybeClose (node:internal/child_process:1091:16)
1458 verbose stack     at ChildProcess._handle.onexit (node:internal/child_process:302:5)
1459 verbose pkgid @sentry/[email protected]
1460 verbose cwd /data/data/com.termux/files/home/apps/httptoolkit-server
1461 verbose Linux 5.10.81-android12-9-24709469-abF936BXXU1AVGA
1462 verbose node v18.7.0
1463 verbose npm  v8.15.0
1464 error code 1
1465 error path /data/data/com.termux/files/home/apps/httptoolkit-server/node_modules/@sentry/cli
1466 error command failed
1467 error command sh /data/data/com.termux/files/usr/tmp/install-9fe2889a.sh
1468 verbose exit 1
1469 timing npm Completed in 6759ms
1470 verbose unfinished npm timer reify 1664099795483
1471 verbose unfinished npm timer reify:build 1664099801094
1472 verbose unfinished npm timer build 1664099801096
1473 verbose unfinished npm timer build:deps 1664099801096
1474 verbose unfinished npm timer build:run:install 1664099801149
1475 verbose unfinished npm timer build:run:install:node_modules/@sentry/cli 1664099801150
1476 verbose code 1
1477 error A complete log of this run can be found in:
1477 error     /data/data/com.termux/files/home/.npm/_logs/2022-09-25T09_56_35_396Z-debug-0.log

2022-09-25T09_56_35_396Z-debug-0.log

Installing httptoolkit-server for use as a library

Since scripting in the UI is not a thing yet, I wanted to install this package to script responses that way (since, from my understanding, I can use it as a library to accomplish this).
Unfortunately it seems like the npm release is deprecated

HTTP Toolkit Server is no longer published to npm. Latest releases are available from https://github.com/httptoolkit/httptoolkit-server/releases

Although I couldn't find any reason for this or discussion about it.
Trying to npm install it from github (since npm supports github as a package source) using
npm i httptoolkit/httptoolkit-server#v1.14.1
installs the package from github, but seems to only end up with the overrides and the bin folder, which doesn't match up with the content of the windows release download for 1.14.1, and seems to be missing core files required in order to use the package.

How do I use httptoolkit-server as a library (without going the c-style way of just cloning it into a folder in my project)?
Why was the npm download deprecated?

Support darwin-arm64

Tracking issue to support m1 version of macos.

Should we wait for GitHub to support the m1 version of actions? Also, it would be nice if we can publish a way to compile and build httptoolkit desktop app without fetching server from github release section?

'Cannot find module prepend-node.js' when using npx in hooked terminal

My steps to cause this error:

• Copy existing terminal powershell command and run it in a terminal
• Run npx tsx in my typescript/nodejs project

The error:

❯ npx tsc
node:internal/modules/cjs/loader:1042
  throw err;
  ^

Error: Cannot find module 'S:\Dev\my-project\..\js\prepend-node.js'
Require stack:
- internal/preload
    at Module._resolveFilename (node:internal/modules/cjs/loader:1039:15)
    at mod._load (C:\Users\Temm\AppData\Local\httptoolkit-server\client\1.12.3\overrides\js\wrap-require.js:52:26)
    at Module.require (node:internal/modules/cjs/loader:1105:19)
    at Module._preloadModules (node:internal/modules/cjs/loader:1395:12)
    at loadPreloadModules (node:internal/process/pre_execution:621:5)
    at setupUserModules (node:internal/process/pre_execution:125:3)
    at prepareExecution (node:internal/process/pre_execution:116:5)
    at prepareMainThreadExecution (node:internal/process/pre_execution:36:3)
    at node:internal/main/run_main_module:10:1 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ 'internal/preload' ]
}

Node.js v18.13.0

Some extra info:

• This happens consistently
• Just running npx tsc to run the typescript compiler causes the error
• My Typescript package version is 4.8.4

Playground won't load

I set playground: true in src/httptoolkit-server.ts, but http://localhost:45457/ just returns Cannot GET /.

Any ideas?

Fresh Terminal, Wrong git-bash

One issue others may find is that HTTP Toolkit fails to open a New Terminal Window. In my case, the wrong git-bash command/file was being located. To identify which, I ran Get-Command git-bash in Powershell and found that an unexpected batch file was being accessed instead of my git-bash.exe.

To resolve the issue, I made sure my Git directory (C:\Program Files\Git in my case) was added to the Environment Variables, and positioned earlier than the directory which held the batch file with a similar name. This resolved the issue, enabling HTTP Toolkit to locate the proper executable.

httptoolkit-server/src/interceptors/terminal/fresh-terminal-interceptor.ts

Lines 41 to 49 in 6c6249a

	const getWindowsTerminalCommand = async (): Promise<SpawnArgs | null> => {
	if (await commandExists('git-bash')) {
	return { command: 'git-bash' };
	} else if (await canAccess(DEFAULT_GIT_BASH_PATH)) {
	return { command: DEFAULT_GIT_BASH_PATH };
	}
	return { command: 'start', args: ['cmd'], options: { shell: true }, skipStartupScripts: true };
	};

The project presently uses command-exists, which only checks where. The lookpath project takes a slightly more rigorous approach, determining whether the path is an executable:

npm install error: sentry-cli

I did agit clone followed by an npm install on a raspberry pi 4 and got an error

Unable to download sentry-cli binary from https://downloads.sentry-cdn.com/sentry-cli/1.61.0/sentry-cli-Linux-arm64

There were also a lot of npm WARN deprecated messages, including an unmaintained [email protected]

Can't install the module using node:21

          i can't install the module using node:21

C:\Users\Administrator\Documents\Project\httptoolkit-server>npm install
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: '[email protected]',
npm WARN EBADENGINE   required: { node: '20.11.1' },
npm WARN EBADENGINE   current: { node: 'v21.4.0', npm: '10.2.5' }
npm WARN EBADENGINE }
npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\Administrator\\Documents\\Project\\httptoolkit-server\\node_modules\\ssh2',
npm WARN cleanup     [Error: EPERM: operation not permitted, rmdir 'C:\Users\Administrator\Documents\Project\httptoolkit-server\node_modules\ssh2\lib\protocol'] {
npm WARN cleanup       errno: -4048,
npm WARN cleanup       code: 'EPERM',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\Administrator\\Documents\\Project\\httptoolkit-server\\node_modules\\ssh2\\lib\\protocol'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\Administrator\\Documents\\Project\\httptoolkit-server\\node_modules\\node-datachannel',
npm WARN cleanup     [Error: EPERM: operation not permitted, rmdir 'C:\Users\Administrator\Documents\Project\httptoolkit-server\node_modules\node-datachannel\node_modules\@istanbuljs\load-nyc-config'] {
npm WARN cleanup       errno: -4048,
npm WARN cleanup       code: 'EPERM',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\Administrator\\Documents\\Project\\httptoolkit-server\\node_modules\\node-datachannel\\node_modules\\@istanbuljs\\load-nyc-config'
npm WARN cleanup     }
npm WARN cleanup   ]
npm WARN cleanup ]
npm ERR! code 2
npm ERR! path C:\Users\Administrator\Documents\Project\httptoolkit-server\node_modules\node-datachannel
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c prebuild-install || (npm install --ignore-scripts --production=false && npm run _prebuild)
npm ERR! added 604 packages, removed 1 package, and audited 662 packages in 1m
npm ERR!
npm ERR! 69 packages are looking for funding
npm ERR!   run `npm fund` for details
npm ERR!
npm ERR! 11 vulnerabilities (7 moderate, 4 high)
npm ERR!
npm ERR! To address issues that do not require attention, run:
npm ERR!   npm audit fix
npm ERR!
npm ERR! To address all issues (including breaking changes), run:
npm ERR!   npm audit fix --force
npm ERR!
npm ERR! Run `npm audit` for details.
npm ERR!
npm ERR! > [email protected] _prebuild
npm ERR! > prebuild --backend cmake-js
npm ERR!
npm ERR! [
npm ERR!   'C:\\Program Files\\nodejs\\node.exe',
npm ERR!   'C:\\Users\\Administrator\\Documents\\Project\\httptoolkit-server\\node_modules\\node-datachannel\\node_modules\\cmake-js\\bin\\cmake-js',
npm ERR!   'rebuild',
npm ERR!   '--runtime-version=21.4.0',
npm ERR!   '--arch=x64',
npm ERR!   '--runtime=node'
npm ERR! ]
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=21.4.0 runtime=node arch=x64 libc= platform=win32)
npm ERR! npm WARN config production Use `--omit=dev` instead.
npm ERR! npm WARN deprecated [email protected]: this library is no longer supported
npm ERR! npm WARN deprecated [email protected]: Use your platform's native atob() and btoa() methods instead
npm ERR! npm WARN deprecated [email protected]: Use your platform's native DOMException instead
npm ERR! npm WARN deprecated [email protected]: Use your platform's native performance.now() and performance.timeOrigin.
npm ERR! npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm ERR! npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! prebuild info begin Prebuild version 11.0.4
npm ERR! prebuild info build Preparing to prebuild [email protected] for node 21.4.0 on win32-x64 using cmake-js
npm ERR! ERR! OMG CMake is not installed. Install CMake.
npm ERR! ERR! OMG CMake is not installed. Install CMake.
npm ERR! prebuild ERR! build Error: Failed to build cmake with exit code 1
npm ERR! prebuild ERR! build     at ChildProcess.<anonymous> (C:\Users\Administrator\Documents\Project\httptoolkit-server\node_modules\node-datachannel\node_modules\prebuild\cmakebuild.js:32:19)
npm ERR! prebuild ERR! build     at ChildProcess.emit (node:events:519:28)
npm ERR! prebuild ERR! build     at ChildProcess._handle.onexit (node:internal/child_process:294:12)

npm ERR! A complete log of this run can be found in: C:\Users\Administrator\AppData\Local\npm-cache\_logs\2024-02-18T19_33_36_865Z-debug-0.log

C:\Users\Administrator\Documents\Project\httptoolkit-server>npm run dev
npm ERR! Missing script: "dev"
npm ERR!
npm ERR! To see a list of scripts, run:
npm ERR!   npm run

npm ERR! A complete log of this run can be found in: C:\Users\Administrator\AppData\Local\npm-cache\_logs\2024-02-18T19_39_39_425Z-debug-0.log

Originally posted by @fdciabdul in #105 (comment)

Cannot read property 'prototype' of undefined

Hey, trying to run some integration tests locally, this is the error I get. I've got a feeling this is to do with Jest as I know they override requiring. Posting here for tracking.

Can I run the`httptoolkit-server` as a remote service?

I want to run the httptoolkit-server and httptoolkit-ui on the server, and then install httptoolkit-android on the phone, the phone and the server are not in the same wifi.
I just want to view some requests on the phone from the server's UI, and don't want to modify anything.Is this possible?
Or the server running the httptoolkit-server and the mobile phone running the httptoolkit-android must be on the same wifi?

Support running the server on linux-arm64

To do this, we need to do the below in approximately this order:

• Get the server usable locally for development, such that it can be checked out, npm installed and started successfully
• Check for any dependencies that don't provide prebuilds in this environment, and see if we can solve that upstream
• See if we can build production releases for linux-arm64 when running directly on a linux-arm64 machine
• See if we can cross-build linux-arm64 from a linux-x64 machine (that's how CI works for every other target - generally by downloading prebuilds for native components, instead of actually cross-compiling)
• Update CI to automatically build for this target
• Publish a release with a linux-arm64 build

Cant install

Hello, i want to install this. pls help.

Windows PowerShell

Copyright (C) Microsoft Corporation. All rights reserved.

Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows

PS C:\httptoolkit\httptoolkit-server> npm i
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see request/request#3142
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
npm ERR! code 2
npm ERR! path C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c prebuild-install || (npm install --ignore-scripts && npm run _prebuild)
npm ERR! added 600 packages, removed 1 package, and audited 658 packages in 43s
npm ERR!
npm ERR! 68 packages are looking for funding
npm ERR! run npm fund for details
npm ERR!
npm ERR! 4 high severity vulnerabilities
npm ERR!
npm ERR! To address all issues (including breaking changes), run:
npm ERR! npm audit fix --force
npm ERR!
npm ERR! Run npm audit for details.
npm ERR!
npm ERR! > [email protected] _prebuild
npm ERR! > prebuild --backend cmake-js
npm ERR!
npm ERR! [
npm ERR! 'C:\Program Files\nodejs\node.exe',
npm ERR! 'C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel\node_modules\cmake-js\bin\cmake-js',
npm ERR! 'rebuild',
npm ERR! '--runtime-version=18.12.1',
npm ERR! '--arch=x64',
npm ERR! '--runtime=node'
npm ERR! ]
npm ERR! Not searching for unused variables given on the command line.
npm ERR! -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22621.
npm ERR! -- The C compiler identification is MSVC 19.29.30147.0
npm ERR! -- The CXX compiler identification is MSVC 19.29.30147.0
npm ERR! -- Detecting C compiler ABI info
npm ERR! -- Detecting C compiler ABI info - done
npm ERR! -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
npm ERR! -- Detecting C compile features
npm ERR! -- Detecting C compile features - done
npm ERR! -- Detecting CXX compiler ABI info
npm ERR! -- Detecting CXX compiler ABI info - done
npm ERR! -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
npm ERR! -- Detecting CXX compile features
npm ERR! -- Detecting CXX compile features - done
npm ERR! -- Configuring incomplete, errors occurred!
npm ERR! See also "C:/httptoolkit/httptoolkit-server/node_modules/node-datachannel/build/CMakeFiles/CMakeOutput.log".
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=18.12.1 runtime=node arch=x64 libc= platform=win32)
npm ERR! npm WARN deprecated [email protected]: this library is no longer supported
npm ERR! npm WARN deprecated [email protected]: Use your platform's native performance.now() and performance.timeOrigin.
npm ERR! npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm ERR! npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
npm ERR! npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! prebuild info begin Prebuild version 11.0.4
npm ERR! prebuild info build Preparing to prebuild [email protected] for node 18.12.1 on win32-x64 using cmake-js
npm ERR! info TOOL Using Visual Studio 16 2019 generator.
npm ERR! info CMD CLEAN
npm ERR! info RUN [
npm ERR! info RUN 'cmake',
npm ERR! info RUN '-E',
npm ERR! info RUN 'remove_directory',
npm ERR! info RUN 'C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel\build'
npm ERR! info RUN ]
npm ERR! info CMD CONFIGURE
npm ERR! info RUN [
npm ERR! info RUN 'cmake',
npm ERR! info RUN 'C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel',
npm ERR! info RUN '--no-warn-unused-cli',
npm ERR! info RUN '-G',
npm ERR! info RUN 'Visual Studio 16 2019',
npm ERR! info RUN '-A',
npm ERR! info RUN 'x64',
npm ERR! info RUN '-DCMAKE_JS_VERSION=6.3.2',
npm ERR! info RUN '-DCMAKE_BUILD_TYPE=Release',
npm ERR! info RUN '-DCMAKE_RUNTIME_OUTPUT_DIRECTORY=C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel\build',
npm ERR! info RUN '-DCMAKE_JS_INC=C:\Users\bayuf\.cmake-js\node-x64\v18.12.1\include\node;C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel\node_modules\nan',
npm ERR! info RUN '-DCMAKE_JS_SRC=C:/httptoolkit/httptoolkit-server/node_modules/node-datachannel/node_modules/cmake-js/lib/cpp/win_delay_load_hook.cc',
npm ERR! info RUN '-DNODE_RUNTIME=node',
npm ERR! info RUN '-DNODE_RUNTIMEVERSION=18.12.1',
npm ERR! info RUN '-DNODE_ARCH=x64',
npm ERR! info RUN '-DCMAKE_JS_LIB=C:\Users\bayuf\.cmake-js\node-x64\v18.12.1\win-x64\node.lib',
npm ERR! info RUN '-DCMAKE_SHARED_LINKER_FLAGS=/DELAYLOAD:NODE.EXE'
npm ERR! info RUN ]
npm ERR! CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
npm ERR! Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
npm ERR! system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
npm ERR! OPENSSL_INCLUDE_DIR)
npm ERR! Call Stack (most recent call first):
npm ERR! C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
npm ERR! C:/Program Files/CMake/share/cmake-3.25/Modules/FindOpenSSL.cmake:613 (find_package_handle_standard_args)
npm ERR! CMakeLists.txt:26 (find_package)
npm ERR!
npm ERR!
npm ERR! ERR! OMG Process terminated: 1
npm ERR! prebuild ERR! build Error: Failed to build cmake with exit code 1
npm ERR! prebuild ERR! build at ChildProcess. (C:\httptoolkit\httptoolkit-server\node_modules\node-datachannel\node_modules\prebuild\cmakebuild.js:32:19)
npm ERR! prebuild ERR! build at ChildProcess.emit (node:events:513:28)
npm ERR! prebuild ERR! build at ChildProcess._handle.onexit (node:internal/child_process:291:12)

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\bayuf\AppData\Local\npm-cache_logs\2022-12-15T07_49_45_639Z-debug-0.log

and this same with the UI

How to configure mock requests without any browser interaction.

Hey HTTP toolkit team, currently I am trying to set up some automation where I am interested to mock some requests and send that to my local server for some processing.

I can easily do this when I am on GUI but not able to figure it out the same when not using any browser but depends entirely on the terminal.

I also tried login my HTTP toolkit pro account but even this is not possible since your login server is checking the cookie and there's no way, I can provide any configurations to the HTTP toolkit server for things related to these.

Is there any way, i can configure login and mock requests using some JSON files?

fails to intercept oauth2

Hello 👋

I'm trying to intercept some simple ruby code, like this:

# example.rb
require 'oauth2'

puts "env rubylib is:"
puts ENV['RUBYLIB']

client = OAuth2::Client.new(client_id, client_secret, site)

Using a fresh terminal, I run it and it looks like net/https isn't overridden:

env rubylib is:
/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems
Traceback (most recent call last):
	18: from ./example.rb:19:in `<main>'
	17: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/strategy/client_credentials.rb:20:in `get_token'
	16: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/client.rb:147:in `get_token'
	15: from /Users/user/.gem/ruby/2.6.0/gems/oauth2-1.4.4/lib/oauth2/client.rb:99:in `request'
	14: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/connection.rb:492:in `run_request'
	13: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/rack_builder.rb:153:in `build_response'
	12: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/request/url_encoded.rb:23:in `call'
	11: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:66:in `call'
	10: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter.rb:60:in `connection'
	 9: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:68:in `block in call'
	 8: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:126:in `perform_request'
	 7: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:135:in `request_with_wrapped_block'
	 6: from /Users/user/.gem/ruby/2.6.0/gems/faraday-1.0.0/lib/faraday/adapter/net_http.rb:149:in `request_via_request_method'
	 5: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:1470:in `request'
	 4: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:919:in `start'
	 3: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
	 2: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/http.rb:996:in `connect'
	 1: from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `ssl_socket_connect'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

(Note the error):
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

From faraday/adapter/net_http.rb, it looks like it's looking for net/https:

begin
  require 'net/https'
rescue LoadError
  warn 'Warning: no such file to load -- net/https. ' \
    'Make sure openssl is installed if you want ssl support'
  require 'net/http'
end

Should the overrides include a net/https file?

❯ ls "/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems"
http.rb   net       stripe.rb uri
❯ ls "/Applications/HTTP Toolkit.app/Contents/Resources/app/httptoolkit-server/overrides/gems/net"
http.rb

Thanks

build for arm64 failed

I want build in arm64.but, when i run "npm run prepack",There are the following errors:

$ node -v
v14.17.1
kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ uname -a
Linux kylin-v10sp1-arm64-2107 5.4.18-35-generic #21-KYLINOS SMP Tue Jul 20 13:31:32 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

kylin@kylin-v10sp1-arm64-2107:~/httptoolkit-server$ npm run prepack

[email protected] prepack /home/kylin/httptoolkit-server
npm run build:src && oclif-dev manifest

[email protected] build:src /home/kylin/httptoolkit-server
rm -rf lib && tsc -b --force

node_modules/portfinder/lib/portfinder.d.ts:39:1 - error TS1128: Declaration or statement expected.

39 export setBasePort(port: number);

node_modules/portfinder/lib/portfinder.d.ts:39:24 - error TS1005: ',' expected.

39 export setBasePort(port: number);
                       ~

node_modules/portfinder/lib/portfinder.d.ts:49:1 - error TS1128: Declaration or statement expected.

49 export setHighestPort(port: number);

node_modules/portfinder/lib/portfinder.d.ts:49:27 - error TS1005: ',' expected.

49 export setHighestPort(port: number);
~

Found 4 errors.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build:src: rm -rf lib && tsc -b --force
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build:src script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/kylin/.npm/_logs/2022-08-12T05_50_15_861Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] prepack: npm run build:src && oclif-dev manifest
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] prepack script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/kylin/.npm/_logs/2022-08-12T05_50_15_891Z-debug.log

Upgrading packages

It seems there are lots of package that needs to be upgraded. Also, we need to be careful, so there is no breakage in any platform.

Add support for gnome console on linux

This is a feature request. Can you add support for gnome console to fresh-terminal ?

I made a pull request :
#103

Terminal Intercepting crashes when using NodeJS worker threads

HTTP Toolkit Terminal Interceptor currently crashes when executing a script that uses Worker threads

Exception

Worker error: Error [InvalidArgumentError]: Proxy opts.uri is mandatory
    at buildProxyOptions (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\node_modules\undici\lib\proxy-agent.js:28:11)
    at new ProxyAgent (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\node_modules\undici\lib\proxy-agent.js:40:20)
    at Object.wrapUndici [as wrap] (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\prepend-node.js:72:9)
    at fixModule (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\wrap-require.js:32:37)
    at mod._load (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\wrap-require.js:67:24)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (S:\Apps\Scoop\apps\httptoolkit\1.13.0\resources\httptoolkit-server\overrides\js\prepend-node.js:126:5)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10) {
  code: 'UND_ERR_INVALID_ARG'
}

The error originates here:

httptoolkit-server/overrides/js/prepend-node.js

Lines 61 to 71 in 89ba2ec

	wrapModule('undici', function wrapUndici (loadedModule) {
	const ProxyAgent = loadedModule.ProxyAgent;
	const setGlobalDispatcher = loadedModule.setGlobalDispatcher;
	// Old Undici release, which can't be intercepted:
	if (!ProxyAgent || !setGlobalDispatcher) return;
	setGlobalDispatcher(
	new ProxyAgent(process.env.HTTP_PROXY)
	);
	});

After some investigation, it turns out that you are setting the envoirement variable http_proxy, but in code attempt to access it as HTTP_PROXY. In a regular NodeJS context, this works fine since the process.env object does some getter magic to make property accessing case-insensitive. This is correct and expected, as envoirement variables on windows are intended to be case-insensitive.
Documented here: On Windows operating systems, environment variables are case-insensitive.

In worker threads, however, this behavior does not apply, and process.env.HTTP_PROXY returns undefined when the variable is called http_proxy, which causes the ProxyAgent to get called without an uri, causing the crash as seen above.

On new node versions this happens even without importing undici within the worker, since HTTP Toolkit automatically imports undici to proxy the global fetch object.

httptoolkit-server/overrides/js/prepend-node.js

Lines 117 to 124 in 89ba2ec

	if (MAJOR_NODEJS_VERSION >= 18 || global.fetch) {
	// Node 18 enables fetch by default (available previously behind a flag). This does not use
	// the existing agent API, so is not intercepted by global-agent. Instead, the only way to
	// set the HTTP proxy is to separately import Undici (used internally by Node) and configure
	// Undici's global proxy agent. We bundle our own Undici dep so we can do this reliably,
	// and here we import it to trigger the Undici setup hook defined above.
	require('undici');
	}

The worker threads documentation doesn't explicitly state wether process.env in workers should be case-sensitive or -insensitive, but it seems reasonable to assume that this detail was simply forgotten.
I'm thus going to create an issue in the NodeJS repo as well, however HTTP Toolkit should still implement a workaround for this (perhaps trying to get both the fully-uppercase and the fully-lowercase variant, or getting the keys of process.env and finding the one that case-insensitively matches the env var name to even find variables that are matched inconsistently, e.g. Http-Proxy)

Here's a repro of the issue:

Reproduction code

// main.js
const { Worker } = require("worker_threads");
const path = require("path");

const worker = new Worker(path.join(__dirname, "worker.js"));

console.log("in parent:")
console.log("http_proxy:",process.env.http_proxy)
console.log("HTTP_PROXY:",process.env.HTTP_PROXY)

worker.on("message", (data) => {
    console.log("in worker:")
    console.log("http_proxy:", data.http_proxy);
    console.log("HTTP_PROXY:", data.HTTP_PROXY);
});

// ---------

// worker.js
const { parentPort } = require('worker_threads');

parentPort.postMessage({
  http_proxy: process.env.http_proxy,
  HTTP_PROXY: process.env.HTTP_PROXY,
});

logs:

in parent:
http_proxy: test
HTTP_PROXY: test
in worker:
http_proxy: test
HTTP_PROXY: undefined

TLDR: process.env isn't case insensitive in NodeJS Worker Threads, which breaks the Terminal Interceptor. Make the terminal interceptor manually search for the "http_toolkit" variable in different casings as a workaround.

Cant use `npm prepack`

In the NPM task prepack, oclif-dev manifest is not accepted and should be remplaced by build:manifest that does exactly the same thing.

[Feature Request] - Option to pass an custom TLS JA3

Hey team, It's being almost an year i have been using an HTTP Toolkit and quite recently @pimterry written a blog on Fighting TLS fingerprinting with Node.js

This worked great till now, But now it seems Cloudflare started whitelisting the TLS cipher for different browsers and starts blocking unknown TLS fingerprint.

This can be reproduced from https://flyairseoul.com/CW/ko/main.do? (You have to use south korean proxy to verify it)
Just visit the website and try searching any flight, When using the HTTP toolkit with korean proxy, The website will start showing the cloudflare challenge, But if same tried with chrome directly, It will not show the cloudflare challenge.

This creates an issue while doing the HTTP debugging.

Solution in my mind: As such NodeJS doesn't support changing the TLS JA3 directly and this creates an issue, since the whole HTTPToolkit server and the proxies are implemented in NodeJS.

I really feel, we have to somehow shift the proxy implementation to low level language like GoLang for more better configuration.

What do you guys think?

self-signed upstream CA-certificate

I need to use a self-signed upstream CA-certificate within httptoolkit and couldn't find a place where to add this within the httptoolkit stack. Is there a place, like cacerts within jre where a self-signed root cert can be added?

ecapture

hey do you know about this:

https://github.com/gojue/ecapture

I know this seems spammy, but I have no relation to that project, just seems cool. I know you have had trouble recently with the new Android and certificates, so might be a good option. here is a video and images too:

https://medium.com/@cfc4ncs/ecapture-capture-https-traffic-without-ca-certificates-on-android-linux-a1f6f6ab782b

possible to drop url and headers to a file ?

First of all thanks for great httptoolkit tool, so much easier to use.

I want to intercept and extract specific header value(i.e Cookie) from specific url from android app.
Any way I edit any server file, so it drops url and headers to file ? So my program (dotnet) can access it.

Any ideas ?
Thank you

install error

I clone this repo, then run npm i,then throw some errors as follow:

npm WARN deprecated [email protected]: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code 1
npm ERR! path /root/proxy/httptoolkit-server/node_modules/registry-js
npm ERR! command failed
npm ERR! command sh -c -- prebuild-install || node-gyp rebuild
npm ERR! make: Entering directory '/root/proxy/httptoolkit-server/node_modules/registry-js/build'
npm ERR!   SOLINK_MODULE(target) Release/obj.target/registry.node
npm ERR! make: Leaving directory '/root/proxy/httptoolkit-server/node_modules/registry-js/build'
npm ERR! prebuild-install WARN install No prebuilt binaries found (target=3 runtime=napi arch=x64 libc= platform=linux)
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | x64
npm ERR! gyp info find Python using Python version 3.9.9 found at "/usr/bin/python3"
npm ERR! gyp info spawn /usr/bin/python3
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/usr/local/lib/nodejs/node-v16.18.0-linux-x64/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/root/proxy/httptoolkit-server/node_modules/registry-js/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/usr/local/lib/nodejs/node-v16.18.0-linux-x64/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/root/.cache/node-gyp/16.18.0/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/root/.cache/node-gyp/16.18.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/usr/local/lib/nodejs/node-v16.18.0-linux-x64/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/root/.cache/node-gyp/16.18.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/root/proxy/httptoolkit-server/node_modules/registry-js',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info spawn make
npm ERR! gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
npm ERR! make: g++: No such file or directory
npm ERR! make: *** [registry.target.mk:22: Release/obj.target/registry.node] Error 127
npm ERR! gyp ERR! build error 
npm ERR! gyp ERR! stack Error: `make` failed with exit code: 2
npm ERR! gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/nodejs/node-v16.18.0-linux-x64/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:201:23)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
npm ERR! gyp ERR! System Linux 5.14.0-202.el9.x86_64
npm ERR! gyp ERR! command "/usr/local/lib/nodejs/node-v16.18.0-linux-x64/bin/node" "/usr/local/lib/nodejs/node-v16.18.0-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd /root/proxy/httptoolkit-server/node_modules/registry-js
npm ERR! gyp ERR! node -v v16.18.0
npm ERR! gyp ERR! node-gyp -v v9.1.0
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-03-08T09_58_05_063Z-debug-0.log

android: cant find trusted cert after install using root adb

Hello.
Your application has always worked fine for me, but today I suddenly found a window with “Manual setup required for android 11”.
8 hours of research into the problem showed the following:

1. The certificate successfully appears in the system /system/etc/security/cacerts/4f74014f.0
2. When I launch the application, I get in the logs: Cert already installed, nothing to do

Debugging the Android application showed that the whereIsCertTrusted function returns null.
This certificate 4f74014f.0 is not in the keyStore.aliases() array, so the application refuses to start.
This certificate is also not in the list of system certificates in the phone settings.

But the best part is, I used a hack and replaced null with "system" to force the application to think that the certificate was installed, and I got success!

All HTTPS traffic appeared in the application (including from YouTube, etc.), which indicates that the certificate is installed and working.

Total:

1. I don’t know why KeyStore.getInstance("AndroidCAStore") does not see this certificate
2. I don’t know why this worked for me a month ago and for the last 3-5 years.

Since my experiment shows that, despite the absence of a certificate in the list obtained by the KeyStore.getInstance("AndroidCAStore") method, it works successfully, I propose to improve this method whereIsCertTrusted in one of two ways:

1. Check for the presence of a certificate in the /system/etc/security/cacerts/ folder, as the desktop application does.
2. Send a test HTTPS request over the network and if it is successful, it means that the certificate is installed and working successfully.

It would also be a good idea to add a Sentry metric and compare what % of devices have a certificate successfully installed and working but do not have it in KeyStore.getInstance("AndroidCAStore") to understand how often this problem occurs.

software:
HTTP Toolkit desktop v1.14.3
android HTTP Toolkit 1.3.10
OnePlus 9 Pro 5G
Android 11