$TRANS_PROGRAM has grown too large for OS X about translate-shell

[soimort]

Thanks for bringing this issue up - system specific limit is something I wasn't aware of before.

On Linux kernel, the length limitation of a single argument is indeed MAX_ARG_STRLEN (131072) and this has never increased. You can test this easily with the following program:

#include <stdio.h>
#include <unistd.h>

#define LEN 131071

int main()
{
    char s[LEN + 1];
    int i;
    for (i = 0; i < LEN; i++)
        s[i] = 'x';
    s[LEN] = 0;
    return execl("/usr/bin/expr", "expr", "length", s, (char *)NULL);
}

Change LEN to 131072, the execl() call shall fail.

Just like you doubted, your estimation of $TRANS_PROGRAM isn't very correct. The actual lengths are 122233 and 122634 respectively, both did not exceed the MAX_ARG_STRLEN limit on Linux.

$ git checkout 467a56
$ make &>/dev/null && . build/trans -V &> /dev/null && expr length "$TRANS_PROGRAM"
122634
$ git checkout HEAD~1
$ make &>/dev/null && . build/trans -V &> /dev/null && expr length "$TRANS_PROGRAM"
122233

While Linux kernel does check for validity of erch argument length (https://github.com/torvalds/linux/blob/master/fs/exec.c#L477), the OS X kernel, XNU, as well as other BSD derivatives, does not perform such check for each single argument, so there is no real equivalent to MAX_ARG_STRLEN. XNU however, has a limitation on the maximum length of arguments, which is

#define ARG_MAX        (256 * 1024) /* max bytes for an exec function */

as defined in <syslimits.h>.

It may seem a bit weird that the argument of length 122233 is fine but the one of length 122634 crashes, since both are far less than 262144, even if you count in all other arguments 'gawk', '-f'. However, every OS has its quirks. I did a simple grepping into XNU's source, it seems these spaces are reserved not only for argv[], but also for envv[] (http://www.opensource.apple.com/source/xnu/xnu-2782.1.97/bsd/kern/kern_exec.c , see L#3137-3330). I guess that's why we can't have a lengthier argument in its execve() call.
(Not familiar with BSD implementation, so far it's really just a guess. Any experts?)

I cannot tell at exactly what length $TRANS_PROGRAM could break on OS X; it is not solely about a single argument length, unlike Linux, as I explained above. Two straightforward solutions I can think of at the moment:

1. Abandon the lengthy argument wrapping; use shell built-in print to pipe the whole program to gawk. (I bet shell can take infinite arguments, as long as it's not a system call)
2. Reduce the size of the core program and keep it to 100KB or below.

[zmwangx]

Thanks for the clarification; I didn't expect env vars to be counted against ARG_MAX. Given that I'd go for shell builtins (yeah, as far as I know they are not limited by ARG_MAX). Speaking of limiting the program size, it's still possible to exceed ARG_MAX if a user in addition exports some long env vars (albeit unlikely).

[soimort]

Due fix in develop. Closing.