This commit adds a way to differentiate a local failure from a "early
exit" triggered by a failure on another node. Not separating this can
cause a race condition when two nodes fail at roughly the same time.

Fixes #5521