Reload signal can cause freeze? about containerpilot

[tgross]

The symptoms I described might be wrong here... the problem might actually be within the re-registration process rather than the signal catching itself.

[tgross]

Ok, got it. We have a deadlock between reloadConfig() and inMaintenanceMode() (in signals.go).

When reloadConfig is hit, it obtains the r/w lock on the signalLock global. This lock is designed to prevent us from trying to handle more than one signal at a time. The inMaintenanceMode query is allowed to get a read lock. Multiple readers are allowed at the same time that one writer -- one signal handler, that is -- has a lock. The inMaintenanceMode query runs in the poll function (in main.go), which is running a select for ticker events or a send on the quit channel. (This will be important in a second.)

The sequence of events that leads to a deadlock is as follows:

• SIGHUP is received
• reloadConfig is called
  • the running goroutine obtains a r/w lock
  • we perform IO at loadConfig, so we yield execution
• a goroutine running poll is scheduled
  • it hits a select on ticker.C and calls inMaintenanceMode
  • inMaintenanceMode asks for a read lock and can't obtain one
  • the polling gorouting blocks, so it yields execution
• our main goroutine is scheduled
  • it hits stopPolling and sends on the quit channel

At this point, no polling goroutine can make forward progress because we are waiting on the read lock. And the main goroutine can't make forward progress because it's waiting to send on a synchronous channel to those blocked goroutines.

---

@justenwalker I think the solution to this is to split the signalLock from the lock around the paused global. We can have toggleMaintenanceMode, terminate, and reloadConfig fight over the signalLock and then have inMaintenanceMode and toggleMaintenanceMode fight over a separate maintModeLock. Because toggleMaintenanceMode doesn't send to the quit channels, we don't have the same possibility of a deadlock there.

I'll test this fix out with my application and see if it does the job.

[tgross]

Opened #74

[tgross]

Merged. Awaiting the 1.0 release.

[tgross]

Released in RC: https://github.com/joyent/containerbuddy/releases/tag/0.1.1