sh.man("bash") hangs about sh

[amoffat]

Try

sh.man("bash", _tty_out=False)

_tty_out is True by default. I've found that most commands behave as users would expect if _tty_out is True, but this is one of those cases where it does not.

[rbbratta]

sh.man("bash", _tty_out=False) works, thanks.

I would expect any command that uses $PAGER with a _tty_out=True would halt as well.

man_bash = sh.man("bash", _env={'PAGER': '/bin/cat'}) also works.

You could remove $PAGER from the env when running with _tty_out, since they will probably never mix well.

I think _tty_out should not be True by default, since the regular case is to return stdout from the function call. I would expect _tty_out to be on when using pexpect or something where I am explicitly emulating a terminal. For a subprocess.Popen replacement I expect everything to be pipes.

Also _tty_out=True is not correctly setting the terminal size when I tested.

>>> man_bash = sh.man("bash", _out=sys.stdout)
BASH(1)                                                                BASH(1)

NAME
       bash - GNU Bourne-Again SHell

SYNOPSIS
       bash [options] [file]

COPYRIGHT
       Bash is Copyright (C) 1989-2009 by the Free Software Foundation, Inc.

DESCRIPTION
       Bash  is  an  sh-compatible  command language interpreter that executes
Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 820, in output_thread
    done = stream.read()
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 1172, in read
    self.write_chunk(chunk)
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 1148, in write_chunk
    self.handler.write(chunk.encode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 84: ordinal not in range(128)

Versus using the regular pager

>>> man_bash = sh.man("bash", _out=sys.stdout, _tty_out=False)
BASH(1)                                                                                                                                                                                                                                                          BASH(1)



NAME
       bash - GNU Bourne-Again SHell

SYNOPSIS
       bash [options] [file]

COPYRIGHT
       Bash is Copyright (C) 1989-2009 by the Free Software Foundation, Inc.

DESCRIPTION
       Bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file.  Bash also incorporates useful features from the Korn and C shells (ksh and csh).

       Bash is intended to be a conformant implementation of the Shell and Utilities portion of the IEEE POSIX specification (IEEE Standard 1003.1).  Bash can be configured to be POSIX-conformant by default.

OPTIONS
       In addition to the single-character shell options documented in the description of the set builtin command, bash interprets the following options when it is invoked:

       -c string If the -c option is present, then commands are read from string.  If there are arguments after the string, they are assigned to the positional parameters, starting with $0.
       -i        If the -i option is present, the shell is interactive.
       -l        Make bash act as if it had been invoked as a login shell (see INVOCATION below).
       -r        If the -r option is present, the shell becomes restricted (see RESTRICTED SHELL below).
       -s        If the -s option is present, or if no arguments remain after option processing, then commands are read from the standard input.  This option allows the positional parameters to be set when invoking an interactive shell.
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 820, in output_thread
    done = stream.read()
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 1172, in read
    self.write_chunk(chunk)
  File "/usr/local/lib/python2.6/dist-packages/sh.py", line 1148, in write_chunk
    self.handler.write(chunk.encode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 263: ordinal not in range(128)

[amoffat]

Re: those UnicodeDecodeErrors, I pushed out some unicode fixes last night... v1.01, so you'll want to upgrade.

I'm not seeing the terminal size problem you're describing though. If _tty_out is False, there is no terminal size set, so lines will never be wrapped. If tty_out is True, the tty win size is set to 24 rows, 80 columns.

About making _tty_out False by default, let me think on that some more. I remember being surprised at how many programs changed their behavior in unexpected ways when they realized their output was not connected to a tty.

[amoffat]

Ah, I remember the issue. I can't control output buffering if the process isn't attached to a TTY. If it's a pipe, I'm at the mercy of the process, whenever it feels like flushing out the data. If I'm a TTY, I get it immediately. This is necessary to make sh's piping operations work correctly. If you need more details, let me know. If you want to mess around with the problem, take a look at test.py's test_piped_generator and test_multiple_pipes. Try to get those to work with _tty_out=False, and you'll see that some programs don't flush their pipes until they exit.

[rbbratta]

If _tty_out is False and _out is actually a tty, e.g. _out=sys.stdout, then man will detect the tty, wrap and run the pager, etc.

The line buffering issue is also explained here: http://www.pixelbeat.org/programming/stdio_buffering/
I thought I saw a link once where someone used Popen's preexec_fn to change the buffering, maybe by calling setvbuf(), but I can't find the example.

The default _tty_out may be fine, I would just make sure to be clear upfront that you are emulating a tty.

I'll try to take a look at the pipe examples.

[rbbratta]

_tty_out works, so we can close this issue.