mobile-shell / mosh

Mobile Shell

Mosh: the mobile shell

Mosh is a remote terminal application that supports intermittent connectivity, allows roaming, and provides speculative local echo and line editing of user keystrokes.

It aims to support the typical interactive uses of SSH, plus:

• Mosh keeps the session alive if the client goes to sleep and wakes up later, or temporarily loses its Internet connection.

• Mosh allows the client and server to "roam" and change IP addresses, while keeping the connection alive. Unlike SSH, Mosh can be used while switching between Wi-Fi networks or from Wi-Fi to cellular data to wired Ethernet.

• The Mosh client runs a predictive model of the server's behavior in the background and tries to guess intelligently how each keystroke will affect the screen state. When it is confident in its predictions, it will show them to the user while waiting for confirmation from the server. Most typing and uses of the left- and right-arrow keys can be echoed immediately.

  As a result, Mosh is usable on high-latency links, e.g. on a cellular data connection or spotty Wi-Fi. In distinction from previous attempts at local echo modes in other protocols, Mosh works properly with full-screen applications such as emacs, vi, alpine, and irssi, and automatically recovers from occasional prediction errors within an RTT. On high-latency links, Mosh underlines its predictions while they are outstanding and removes the underline when they are confirmed by the server.

Mosh does not support X forwarding or the non-interactive uses of SSH, including port forwarding.

Other features

• Mosh adjusts its frame rate so as not to fill up network queues on slow links, so "Control-C" always works within an RTT to halt a runaway process.

• Mosh warns the user when it has not heard from the server in a while.

• Mosh supports lossy links that lose a significant fraction of their packets.

• Mosh handles some Unicode edge cases better than SSH and existing terminal emulators by themselves, but requires a UTF-8 environment to run.

• Mosh leverages SSH to set up the connection and authenticate users. Mosh does not contain any privileged (root) code.

Getting Mosh

The Mosh web site has information about packages for many operating systems, as well as instructions for building from source.

Note that mosh-client receives an AES session key as an environment variable. If you are porting Mosh to a new operating system, please make sure that a running process's environment variables are not readable by other users. We have confirmed that this is the case on GNU/Linux, OS X, and FreeBSD.

Usage

The mosh-client binary must exist on the user's machine, and the mosh-server binary on the remote host.

The user runs:

$ mosh [user@]host

If the mosh-client or mosh-server binaries live outside the user's $PATH, mosh accepts the arguments --client=PATH and --server=PATH to select alternate locations. More options are documented in the mosh(1) manual page.

There are more examples and a FAQ on the Mosh web site.

How it works

The mosh program will SSH to user@host to establish the connection. SSH may prompt the user for a password or use public-key authentication to log in.

From this point, mosh runs the mosh-server process (as the user) on the server machine. The server process listens on a high UDP port and sends its port number and an AES-128 secret key back to the client over SSH. The SSH connection is then shut down and the terminal session begins over UDP.

If the client changes IP addresses, the server will begin sending to the client on the new IP address within a few seconds.

To function, Mosh requires UDP datagrams to be passed between client and server. By default, mosh uses a port number between 60000 and 61000, but the user can select a particular port with the -p option. Please note that the -p option has no effect on the port used by SSH.

Advice to distributors

A note on compiler flags: Mosh is security-sensitive code. When making automated builds for a binary package, we recommend passing the option --enable-compile-warnings=error to ./configure. On GNU/Linux with g++ or clang++, the package should compile cleanly with -Werror. Please report a bug if it doesn't.

Where available, Mosh builds with a variety of binary hardening flags such as -fstack-protector-all, -D_FORTIFY_SOURCE=2, etc. These provide proactive security against the possibility of a memory corruption bug in Mosh or one of the libraries it uses. For a full list of flags, search for HARDEN in configure.ac. The configure script detects which flags are supported by your compiler, and enables them automatically. To disable this detection, pass --disable-hardening to ./configure. Please report a bug if you have trouble with the default settings; we would like as many users as possible to be running a configuration as secure as possible.

Mosh ships with a default optimization setting of -O2. Some distributors have asked about changing this to -Os (which causes a compiler to prefer space optimizations to time optimizations). We have benchmarked with the included src/examples/benchmark program to test this. The results are that -O2 is 40% faster than -Os with g++ 4.6 on GNU/Linux, and 16% faster than -Os with clang++ 3.1 on Mac OS X. In both cases, -Os did produce a smaller binary (by up to 40%, saving almost 200 kilobytes on disk). While Mosh is not especially CPU intensive and mostly sits idle when the user is not typing, we think the results suggest that -O2 (the default) is preferable.

Our Debian and Fedora packaging presents Mosh as a single package. Mosh has a Perl dependency that is only required for client use. For some platforms, it may make sense to have separate mosh-server and mosh-client packages to allow mosh-server usage without Perl.

Notes for developers

To start contributing to Mosh, install the following dependencies:

Debian, Windows Subsystem for Linux:

$ sudo apt install -y build-essential protobuf-compiler \
    libprotobuf-dev pkg-config libutempter-dev zlib1g-dev libncurses5-dev \
    libssl-dev bash-completion tmux less

MacOS:

$ brew install protobuf automake

Once you have forked the repository, run the following to build and test Mosh:

$ ./autogen.sh
$ ./configure
$ make
$ make check

Mosh supports producing code coverage reports by tests, but this feature is disabled by default. To enable it, make sure lcov is installed on your system. Then, configure and run tests:

$ ./configure --enable-code-coverage
$ make check-code-coverage

This will run all tests and produce a coverage report in HTML form that can be opened with your favorite browser. Ideally, newly added code should strive for 90% (or better) incremental test coverage.

More info

• Mosh Web site:

  https://mosh.org

• [email protected] mailing list:

  https://mailman.mit.edu/mailman/listinfo/mosh-devel

• [email protected] mailing list:

  https://mailman.mit.edu/mailman/listinfo/mosh-users

• #mosh channel on Libera Chat

  https://web.libera.chat/#mosh

mosh's Issues

Mac OS X sleeps (!) while running mosh

When logged in remotely with mosh to a Macintosh, OS X seems to go to sleep (and we start getting the "Last contact" countup). It doesn't even respond to pings.

SSHing to the machine seems to wake it up.

It looks like this is a feature -- "pmset -g" shows a sleep time of "10" when not logged in with SSH, and a sleep of "0" (disabled) when logged in with SSH. (I think "ttyskeepawake" is what makes SSH inhibit sleeping.)

Is it possible to have mosh-server also set the sleep time to be 0 when there is a user logged in? Unfortunately pmset requires root... Starting a screen did update utmp but didn't disable "sleep".

Regression tests

Prevent session key from being swapped to disk?

If the memory page holding Mosh's session key is swapped to disk, a copy of that key could persist indefinitely. Here's an attack scenario where this matters.

• I'm using Mosh at the airport over public WiFi. At some point mosh-client is swapped out to disk.
• Thugs from a rival yakuza clan capture my encrypted Mosh traffic.
• I shut down my laptop before going through security.
• The thugs steal my laptop from the security checkpoint.
• They recover my Mosh session key from swap and decrypt the captured session.

Using POSIX mlock, we can prevent this data from being swapped out. But there are some reasons why we might not bother.

• I usually suspend to disk rather than halting the machine. Mosh's features encourage this behavior. mlock does nothing to protect suspended data.
• I have encrypted swap, which protects both suspend-to-disk and online swapping. Full-disk encryption is a common configuration, supported by installers for Debian, Ubuntu, Fedora, etc. Users who care about this kind of attack will likely use it.
• If you're worried about targeted physical theft, you have many other concerns, including cold boot attacks and plain old beatings. A merely opportunistic laptop thief is unlikely to scan the swap partition for Mosh datastructures, let alone have access to a recording of the corresponding encrypted session.

Having established that the benefits of mlock are real but limited, what are the drawbacks?

• Most importantly, it increases complexity in a security-critical region of code.
• While mlock is available on Linux and OS X, it is not required on every POSIX platform, as it's part of the Realtime Extension.
• The amount of locked memory is limited by RLIMIT_MEMLOCK. This wil be (mis)configured to 0 on some systems, breaking Mosh.

From a cursory search, neither OpenSSH nor OpenSSL prevents swapping of cryptographic materials (which is surprising, because unlike Mosh they also handle persistent keys). By contrast, GnuPG includes a custom secure allocator with knowledge of many Unixes.

Perhaps the right solution is to call mlock if it is available, and treat failure as a non-fatal condition. But I'm still worried about increased code complexity, versus the limited enhancement to security. Maybe I should just write the code and see how complex it is.

Thoughts?

Lengthen blue-bar "Last heard..." timeout or increase heartbeat frequency

Currently when idle, the blue bar appears briefly after we've lost just one datagram. (Since timeout is 5 seconds and we send heartbeat every 3 seconds.)

Investigate information leakage and using chaff and timing fuzz

Should we add timing fuzz and bogus packets to foil passive statistical eavesdropping? (We do add chaff to the datagrams we're already sending, but is this enough?)

Make binary pkg or DMG for Mac

Some users do not want to have to compile mosh to be able to install on OS X.

Fix timing bug

We are too timid on packet loss and sometimes wait up to 3 seconds to resend.

Make mosh-server exit if no connection started in 60 seconds

The mosh-server should quit gracefully if there has been no connection established after 60 seconds. This will lead to fewer mosh-servers lingering around.

Misdisplayed prediction with Enter at bottom of screen

If I press Enter after

(Last line of output from the previous command)
andersk@dr-wily:~$ echo hello_

at the bottom of the screen, mosh temporarily shows the following prediction:

(Last line of output from llo
andersk@dr-wily:~$ echo hello_

Allow suspend/resume of client

Right now the client dies on SIGCONT.

We should have a SIGCONT handler that (1) sets repaint_requested and (2) replays the Terminal::Emulator::close() and Terminal::Emulator::open() strings to reset the outer terminal state (in particular the application-mode cursor keys).

This will also be useful for the Android port if we want mosh to keep a connection over the client's being killed and restored.

TCP mode?

Allow degenerate use of TCP and HTTP CONNECT modes for people who can't use UDP.

Support X forwarding

In theory, we could easily support X forwarding (with roaming!) by just conveying a verbatim octet stream in both directions over the underlying network layer -- the same kind of statesync object that we use for user input.

We wouldn't get predictive local UI, but we would get the other benefits of Mosh (support for sleep/wakeup and IP roaming).

Improve scrolling on Terminal.app?

Mac Terminal.app often repaints the screen immediately on scroll, even if there is more input pending. This makes mosh's scrolling optimization sometimes produce intermediate flashes.

We should explore using the scroll-region feature of the outer terminal to do a more graceful scrolling optimization on a partial scroll. Will need to verify that this fixes the flashing problem on Mac.

Terminal echo can interfere with MOSH CONNECT string

If I press any keys while mosh is connecting, they get echoed back before the MOSH CONNECT string, confusing the wrapper script.

$ mosh linerva
Hello, world!Hello, world!MOSH CONNECT 47266 TGj0sGDQij7qF6ZXFadwOA
[mosh-server detached, pid=9169.]
Connection to linerva.mit.edu closed.
/usr/bin/mosh: Did not find mosh server startup message.

Scrollback and alternate screen (was: Use alternate screen on smcup/rmcup )

When the remote application uses smcup to enter the alternate screen, the client should put the real terminal in alternate screen mode too, so that for example the mouse wheel lets you scroll in less and emacs and barnowl.

Clearer error message for missing MOSH IP

Some sites already use SSH ProxyCommand or disable it for some reason. The mosh script should give a distinct error message in that case.

Fix startup bug with forking

The mosh-server child (host) will die if the server detaches before it forks off a shell. Solution is probably to detach before forking the client, but this requires rejiggering.

Submit Fedora package to Fedora

autoconf check for curses.h

Glitch test for underlining

Right now the trigger for underlining predictions is based on SRTT estimate only. We should add a "glitch" test (similar to the one for whether we show predictions in the first place) so long-outstanding predictions eventually get flagged to the user.

Reconnection and garbage collection

Allow reconnection to existing sessions that were somehow severed. Give command to clean up disconnected sessions.

-a argument to always show predictions

mosh client should use terminfo (also for tmux compatibility)

Currently the mosh client hardcodes escape sequences for a vt220-ish outer terminal supporting BCE and xterm’s window title. It should use terminfo for portability to different outer terminals.

This could be done by making the client a full-blown ncurses application, or by reading the terminfo entries directly with tigetstr/tiparm/etc. (which still requires ncurses).

Fix bug with rxvt-unicode (which only supports 88-color mode)

Write man pages

mosh needs a man page

Compress datagrams with zlib

Android port

It would be wonderful to have persistent remote-terminal sessions from an Android device. Theoretically the Mosh code should be compilable on Android and should be runnable from an existing GUI terminal emulator without needing to jailbreak.

Properly support terminals without BCE

In terminaldisplay.cc

Support 256 colors

Because 256 colors ought to be enough for anybody.

Make Debian stable package

For until the package makes it to Debian stable...

Pass command-line options to ssh

mosh could provide a way to pass options to the ssh invocation used for connection setup. For example, the user might want to SSH to a port other than 22.

One can already customize the SSH connection using Host directives in ~/.ssh/config. Perhaps this is good enough, in which case we could suggest it in documentation.

Start login shell

Write utmp/wtmp entries on Mac OS X

Make "late ack" a normal part of server->client stream and always send

The recent change to include a "late ack" showing which client->server keystroke would definitely have been echoed has improved the rate of spurious mispredictions, but we are only sending these opportunistically when they can piggyback. That means the client also needs a timeout trigger for when there are no more replies to piggyback on, and this timeout trigger has been firing spuriously for me, leading to partial eraser of stuff I had already typed. Also the "pending ack" is not acknowledged so if the packet carrying it is dropped, we run into trouble.

We need to convert this from a header field to a protobuf object that is inserted in the stream as pending data, just like a resize or screen delta. This should make the predictions much more robust.

Cons: If we always send the "late ack," we will increase the chattiness because almost every keystroke hit will now result in TWO packets returning from the server.

Take out lambdas

Requirement of g++ >= 4.6 is not realistic if we want to be in Ubuntu LTS or Debian.

Include homebrew recipe and get it committed to homebrew

It builds fine on OS X with just five packages installed:

autoconf automake boost pkg-config protobuf

Should be pretty easy to make the homebrew recipe.

Allow choice of UDP port and use limited range of UDP ports

For people with UDP firewalled, allow explicit request for particular server-side UDP port.

Rewrite startup script in C++

autoconf needs to check for protoc

mosh script interacts badly with ControlMaster

I'm not sure exactly what's going on, but I use ControlMaster to avoid establishing multiple connections to a server. If I have an open ssh connection to a server (and therefore an active ControlMaster socket), starting mosh fails:

evan@warwick:~$ ssh -N [email protected]

evan@warwick:~$ ls .ssh
authorized_keys                    id_dsa      known_hosts
authorized_keys2                   id_dsa.pub  known_hosts.old
config                             id_rsa
[email protected]:22  id_rsa.pub
evan@warwick:~$ mosh [email protected]
/usr/bin/mosh: Did not find mosh server startup message.

The relevant portion of .ssh/config is:

Host *
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlMaster auto
    ControlPersist 1

Shut off adaptive mode more gracefully

Right now adaptive prediction showing (default mode) will shut off the adaptive display even when predictions are outstanding and displayed, leading to flicker.

We should keep the predictions on until the prediction engine is not active.

Add version number

Add a --version option to client.

Have server print version after MOSH CONNECT.

Make MacPorts ports file

Make build on Ubuntu 10.04 LTS lucid

Can we relax the build dependency on debhelper >= 8?

Can we get rid of dh_autoreconf by putting an explicit rule in debian/rules?

Give credit to whitespace predictions

Currently these are scored as CorrectNoCredit

ssh error messages appear out of order

Some error messages from the initial ssh connection appear after the password prompt is reprinted. For example, when inputting an incorrect password:

aroach@champignon:~$ mosh champignon
aroach@champignon's password:
aroach@champignon's password: Permission denied, please try again.

LANG is not set for mosh-server

mosh uses ssh to run mosh-server on a remote host. From where does that mosh-server process get its environment variables, in particular LANG and friends?

Debian, Ubuntu, Red Hat, and some other Linux distros send these variables over SSH, with

SendEnv LANG LC_*

in /etc/ssh/ssh_config, and a corresponding AcceptEnv in sshd_config. This works fine and does what we want. However it isn't the default for vanilla OpenSSH, and it needs to be configured at both ends. This means that (for example) a Gentoo machine will not cleanly accept Mosh connections. We say that using Mosh does not require root on either end, so reconfiguring sshd is not an option in general.

Some users set LANG in ~/.bash_profile or similar. We could launch mosh-server through a shell (perhaps a login shell) and trust that LANG will appear somehow. Cons: we have to deal with everyone's wacky shells and configurations.

Alternatively, mosh-server could ignore LANG and instead take the name of a locale as a command-line parameter. The wrapper script would fill this in from the client's environment. This assumes that the desired locale names are the same on both ends of the connection. This is an assumption already made by the SendEnv mechanism, but under the status quo it isn't our problem.

Mosh uses only a few locale features: UTF-8 encoding/decoding and querying basic properties of Unicode characters. We could replace this with a dedicated library such as ICU. This would also help with porting to platforms where wchar_t can't represent all of Unicode, such as Windows or Android.

At that point we can take a more hands-off approach to locale issues. Whatever the user typically does to make their applications output UTF-8, they should keep doing it with Mosh. If they rely on SSH SendEnv then we will keep honoring that. If their applications are all children of shells and they set LANG in a login script, that's fine too. If they only use one app and it's hardcoded to UTF-8, we don't care.

Of course if those applications don't output UTF-8, Mosh will break, as now. In particular, terminal escape sequences can start with U+009B 'CONTROL SEQUENCE INTRODUCER'. If we get this character in an ISO 8859 encoding, it will wreck the terminal, even if all visible text is ASCII. But we can probably hack around this, if absolutely necessary, because no valid UTF-8 sequence starts with 0x9B.

Write Kerberos/AFS script

Sites using Kerberos and AFS need special care to preserve the user's Kerberos tickets and AFS tokens. Typically these are destroyed when SSH logs out -- which is a problem for us, since SSH logs out immediately before we even start the Mosh session proper.

One approach is to write a mosh-kerberos-server script that manually copies the Kerberos ticket file (and AFS tokens?) to a new location before detaching from the terminal and letting SSH reap the old ones.

Fix signoff sequence

Hardcoded CSI ! p (DECSTR) isn't working well enough.

NAT busting when server is behind NAT

We want to be able to connect to a server behind a NAT, even after the client roams.

We could use techniques like http://samy.pl/pwnat/ (per Nelson Elhage), except we'd need to keep doing them to pick up any client IP address changes.