Too much CPU when idle about mosh-chrome

[keithw]

One thing that may be helpful that we've learned is that binary hardening compiler flags (that Mosh uses by default) can result in very slow code on ARM. (The performance impact seems to be modest on x86-64.) If you're seeing this on ARM, you might look into compiling with ./configure --disable-hardening and see if that helps.

Mosh also could do a much smarter job of comparing the new and old terminal "frame" -- right now we loop over the whole terminal on every frame transition, even if only a small part has changed (like one letter). For huge terminals this makes a substantial difference when things are changing.

I should be open that we weren't planning a new upstream point release until June. (We have some buggy mouse-escape-sequence support that we need to fix, at the least, and I am trying to finish some other stuff in the meantime.) If you need something earlier, though, I'm happy to work with you and maybe we can prioritize differently.

[rpwoodbu]

Thanks for the tips, @keithw. I'm seeing this on x86, so I don't think hardening is the issue. And I'm seeing this while frames aren't changing.

My best guess is that I'm doing something wrong in my POSIX emulation code that interacts badly with Mosh. It could be an upstream Mosh bug, but I'm betting it isn't. Hopefully I'll find some time to look into it, but it isn't critical.

It would be nice to build against a proper release, but I'm content to wait for one. If I do find there's a Mosh bug, I may ask you to help me find a good commit to build against that has the fix and is reasonably stable (or I could cherrypick, as I'm currently building off of a private branch anyway).

[rpwoodbu]

I've done a bit of digging, and my intuition was correct: There are indeed rapid repeated calls to pselect() which are not blocking as they should. This is only happening in pselect() calls with a timeout. The timeout isn't honored for a brief moment, causing a flurry of pselect() calls until the timeout is again honored. The weird thing is that the timeout is handled in my pselect() emulation code by pthread_cond_timedwait(), and when it is in this funk (again, just for a brief moment, just a split second, really), it is returning immediately with ETIMEDOUT, even though there's a 2-second timeout on the call. Spurious wakeups can happen, but those should return 0, not ETIMEDOUT.

The 2-second timeout is interesting. I'm not seeing this on the 3-second timeout calls. I have a hunch that pthread_cond_timedwait() gets angry when it has already returned once from a call with a deadline later than one in a subsequent call (it uses absolute times).

I'll continue to investigate, but if anyone has any ideas, please chime in.

[rpwoodbu]

Best I can tell, there are two independent bugs that contribute separately to CPU spikes, one in the NaCl environment, and one in Mosh.

NaCl bug: My hunch was close, but not quite on the mark. If a call to pthread_cond_timedwait() returns before the timeout, all subsequent calls made during the same second of real time (the clock from clock_gettime() is only precise to the second, it would seem) will return ETIMEDOUT. The only action I have found that I can take to abate the onslaught CPU activity is to detect premature timeouts and call the forever-blocking pthread_cond_wait(). This seems to work, but I'll bet it'll get stuck if keystrokes and incoming packets stop. Without this sloppy workaround, the spinning which ensues will take the whole CPU until the next second rolls around on the clock.

Mosh bug: Sometimes it calls pselect() with a zero timeout, usually in a tight loop for a fraction of a second. This burns up some CPU (I observe just a few percent on a Celeron-based Chromebook), but it is nowhere nearly as egregious as the former bug.

I may try to build with a newer version of the NaCl SDK to see if that helps. I should also try building from a newer Mosh to see if maybe the zero timeout bug is fixed.

[rpwoodbu]

"Fixed" in SHA 0421c6a.

I've found that I can usleep(100000) when I detect a premature ETIMEDOUT, and that works around the problem well enough. And for some reason, this has stopped teasing out the rapid-fire zero-timeout pselect() calls from Mosh.

I'll bring this up to the NaCl folks.

[rpwoodbu]

One more little patch in SHA bb1800c to fix a subtle race I introduced.