Python Native Messaging host does not process 1MB input about webextensions-examples

[Rob--W]

What makes you believe that only 1 MB is accepted? When I load the example on Linux, I can successfully continue to post to the native messaging app even after sending 1 MB of data.

// port was defined in background.js via connectNative + onMessage listener.

// Send 5 MB data. The message is ignored because the example only responds to "ping".
port.postMessage(new Array(1024 * 1024 * 5));

port.postMessage("ping");
// Result: "pong3" (after waiting a short while for the previous message to have been flushed)

[guest271314]

The addition of separators is not meaningful at all

Yes, it is. The omission means when we do

port.postMessage(port.postMessage(new Array(174763));

the current Python Native Messaging host exits and this

def encodeMessage(messageContent):
        encodedContent = json.dumps(messageContent).encode('utf-8')
        f = open('data.txt', 'w')
        f.write(str(len(encodedContent)))
        f.close()
        encodedLength = struct.pack('@I', len(encodedContent))
        return {'length': encodedLength, 'content': encodedContent}

will write

1048578

to data.txt, greater than the Native Messaging protocol 1024*1024 - when the length should be

873816

JSON.stringify(new Array(174763)).length // 873816

which means we cannot pass and get echoed the maximum Native Messaging protocol message from client to host, 1024*1024 with separators() omitted.

This

port.postMessage(new Array(209715));

will always exit in the Python script you have in your current code, because the default JSON formatting space character that Python adds, see https://docs.python.org/3/library/json.html

If specified, separators should be an (item_separator, key_separator) tuple. The default is (', ', ': ') if indent is None and (',', ': ') otherwise. To get the most compact JSON representation, you should specify (',', ':') to eliminate whitespace.

See Why is the length of json.dumps(json.loads(line)) bigger than line?

import json

a = [[1, 2, 3], {'a':1, 'b':2, 'c':'ä'}]

print(json.dumps(a))
print(json.dumps(a, separators=(',', ':')))
print(json.dumps(a, separators=(',', ':'), ensure_ascii=False))

gives:

[[1, 2, 3], {"a": 1, "b": 2, "c": "\u00e4"}]
[[1,2,3],{"a":1,"b":2,"c":"\u00e4"}]
[[1,2,3],{"a":1,"b":2,"c":"ä"}]

Those space characters being added are a bug in your scriptm making it in its current form incapable of processing the Native Messaging protocol maximum input capability; see https://discuss.python.org/t/how-to-read-1mb-of-input-from-stdin/22534/17.

[guest271314]

I can pass

port.postMessage(new Array(209715));

to C, C++, QuickJS, Deno, Node.js Native Messaging hosts

• https://github.com/guest271314/native-messaging-c
• https://github.com/guest271314/native-messaging-cpp
• https://github.com/guest271314/native-messaging-quickjs
• https://github.com/guest271314/native-messaging-deno
• https://github.com/guest271314/native-messaging-nodejs

and get the message echoed back.

Your current Python Native Messaging host will exit when input to the host is

port.postMessage(port.postMessage(new Array(174763));

So your current Python Native Messaging host is broken in that it cannot achieve processing the maximum capable input to the host from client per Native Messaging protocol https://developer.chrome.com/docs/apps/nativeMessaging/#native-messaging-host-protocol

The maximum size of a single message from the native messaging host is 1 MB

though easily fixable by removing useless space character formatting which is default behaviour of json.dumps().

[Rob--W]

Are you using a modified version of the example?

According to your report, the input is limited to 1MB. That's not the case: 4GB is the documented maximum of a native messaging host (in practice, much lower due to memory and browser's IPC constraints, but definitely more than 1MB).

The output size of the native app is documented to have a max size of 1MB. The ping_pong example never exceeds that; in fact if it returns anything at all, it's 5 bytes (pong2 or pong3).

I see the value in aiming at a minimal representation of the data, but only in some cases, mainly when arrays are involved.

I'll re-open the PR and put a review there.

[guest271314]

Are you using a modified version of the example?

Yes. https://github.com/guest271314/native-messaging-python/blob/main/nm_python.py

#!/usr/bin/env -S python3 -u
# https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging
# https://github.com/mdn/webextensions-examples/pull/157
# Note that running python with the `-u` flag is required on Windows,
# in order to ensure that stdin and stdout are opened in binary, rather
# than text, mode.

import sys
import json
import struct
import traceback

try:
    # Python 3.x version
    # Read a message from stdin and decode it.
    def getMessage():
        rawLength = sys.stdin.buffer.read(4)
        # if len(rawLength) == 0:
        #    sys.exit(0)
        messageLength = struct.unpack('@I', rawLength)[0]
        message = sys.stdin.buffer.read(messageLength).decode('utf-8')
        return json.loads(message)

    # Encode a message for transmission,
    # given its content.
    def encodeMessage(messageContent):
        # https://stackoverflow.com/a/56563264
        # https://docs.python.org/3/library/json.html#basic-usage
        # To get the most compact JSON representation, you should specify 
        # (',', ':') to eliminate whitespace.
        encodedContent = json.dumps(messageContent, separators=(',', ':')).encode('utf-8')
        encodedLength = struct.pack('@I', len(encodedContent))
        return {'length': encodedLength, 'content': encodedContent}

    # Send an encoded message to stdout
    def sendMessage(encodedMessage):
        sys.stdout.buffer.write(encodedMessage['length'])
        sys.stdout.buffer.write(encodedMessage['content'])
        sys.stdout.buffer.flush()
        
    while True:
        receivedMessage = getMessage()
        sendMessage(encodeMessage(receivedMessage))

except Exception as e:
    sys.stdout.buffer.flush()
    sys.stdin.buffer.flush()
    # https://discuss.python.org/t/how-to-read-1mb-of-input-from-stdin/22534/14
    with open("nm_python.log", "w", encoding="utf-8") as f:
        traceback.print_exc(file=f)
    sys.exit(0

According to your report, the input is limited to 1MB. That's not the case: 4GB is the documented maximum of a native messaging host (in practice, much lower due to memory and browser's IPC constraints, but definitely more than 1MB).

The output size of the native app is documented to have a max size of 1MB. The ping_pong example never exceeds that; in fact if it returns anything at all, it's 5 bytes (pong2 or pong3).

I think you have that mixed up. From the browser we can pass 1MB. The host can send back 4GB.

Because separators() is omitted in your current script json.dumps() by default is including useless space character formatting that is being counted as the length of the input message. So this

port.postMessage(port.postMessage(new Array(174763));

is being counted as over 1MB. So when the message is echoed back to the client the incorrect length is being set.

[guest271314]

Very simple to reproduce. Use Python Native Messaging host in your examples repository modified as above - without including separators() passed to json.dumps().

Now from the client pass

port.postMessage(port.postMessage(new Array(174763));

Your host should not exit whether we are talking about input from the client or data written to stdout from the host

JSON.stringify(new Array(174763)).length < 1024*1024

However, in practice your current Python Native Messaging host does exit.

It took a while figuring out where, when and why.

Write the length to files at different parts of the script and read the message written to file by traceback to gather what is happening. I initially thought the issue was Python not reading the entire stdin, as can happen in Node.js and Deno; however, the issue is really just Python adding space characters to the echoed JSON, so the actual message size in JSON is not the same as the message size when json.dumps() inserts space characters; see How to read 1MB of input from stdin?. Again, easily fixable. Was not so easy to locate the source of the bug in that code.

[Rob--W]

Are you using a modified version of the example?

Yes. https://github.com/guest271314/native-messaging-python/blob/main/nm_python.py

Your modified version has a dynamic output (dependent on the input), which consequently raises the importance of a compact JSON representation. While the original ping_pong example did not have such an issue, I did still reopen the PR because it's worth calling out the existence of the message size limits.

According to your report, the input is limited to 1MB. That's not the case: 4GB is the documented maximum of a native messaging host (in practice, much lower due to memory and browser's IPC constraints, but definitely more than 1MB).

The output size of the native app is documented to have a max size of 1MB. The ping_pong example never exceeds that; in fact if it returns anything at all, it's 5 bytes (pong2 or pong3).

I think you have that mixed up. From the browser we can pass 1MB. The host can send back 4GB.

I didn't mix it up. The documentation on MDN is as I stated. Here is the source code of the web browsers in case you want another source besides the documentation:

• Chrome: https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/extensions/api/messaging/native_message_process_host.cc;l=38;drc=b8524150039182faf7988e9478a9eff89728ac03 , used at https://source.chromium.org/chromium/chromium/src/+/main:chrome/browser/extensions/api/messaging/native_message_process_host.cc;l=299;drc=b8524150039182faf7988e9478a9eff89728ac03
• Firefox: https://searchfox.org/mozilla-central/rev/0947c440956c648b853cf79a4737b282396abf09/toolkit/components/extensions/NativeMessaging.jsm#42-50 , used at https://searchfox.org/mozilla-central/rev/0947c440956c648b853cf79a4737b282396abf09/toolkit/components/extensions/NativeMessaging.jsm#177

I think that you got confused because your example echos the input back, so in your case there is no obvious distinction between errors caused by input to the native messaging host vs output received from it.

[guest271314]

The ping_pong.py example in your repository is broken nonetheless.

This is what I am referring to

// Maximum message size in bytes for messages received from Native Messaging
// hosts. Message size is limited mainly to prevent Chrome from crashing when
// native application misbehaves (e.g. starts writing garbage to the pipe).
const size_t kMaximumNativeMessageSize = 1024 * 1024;

thus the maximum length message from a Native Messaging host is 1024*1024, which in its current form you Python script will not do. Thus this PR, so MDN is not posting broken code the masses will use.

[guest271314]

Surely you are not expecting the user to just stay at posting back "pong3". As soon as any input length is equal to or greater than length 873816 your current Python script will count that as 1048578 and your script will exit.

[guest271314]

Put another way, you folks obviously never tested your script to verify it can process the Native Messaging client input and host output.

More than one developer in the wild has linked to you Python Native Messaging host example. I don't think they tested the capabilities either. I have tested the capabilities of your example code, and it is broken outside of sending back the hardcoded message you have in your example. That is not an optimal representation of Native Messaging capabilities.

[guest271314]

Addressed here https://github.com/mdn/webextensions-examples/blob/main/native-messaging/app/ping_pong.py.

Note, on some Linux OS'es python alone is not an executable. Python 2.X is deprecated. /usr/bin/env -S python3 -u should call python3 executable with unbuffered flag

-u : force the stdout and stderr streams to be unbuffered;
this option has no effect on stdin; also PYTHONUNBUFFERED=x