Skip to content

STONITH Concepts

High Availability Clusters rely on mechanisms to prevent split-brain scenarios—situations where network partitions cause multiple nodes to independently assume ownership of shared resources. STONITH (Shoot The Other Node In The Head) is a critical fencing mechanism designed to isolate faulty or unreachable nodes, ensuring data consistency and preventing resource conflicts. By forcibly disconnecting problematic nodes from the cluster, STONITH safeguards the integrity of shared storage and services.

Split-Brain Prevention

Split-brain occurs when a cluster's communication network splits into isolated partitions. Without fencing, nodes in separate partitions may independently claim resources (e.g., a shared disk), leading to data corruption or service outages. STONITH prevents this by:
1. Isolating nodes that fail to respond to heartbeat signals.
2. Forcing a single point of control to determine which node(s) are safe to operate.
3. Preventing resource duplication by ensuring only one node can access critical resources at a time.

Fencing Mechanisms

STONITH leverages hardware or software-based fencing agents to isolate nodes. Common mechanisms include:
- IPMI ( Intelligent Platform Management Interface): Uses out-of-band management to power cycle or reboot nodes.
- Power Switches: Physically disconnects nodes via serial or network-connected power outlets (e.g., APC UPS).
- SSH: Executes commands on a node to terminate processes or reboot.
- FCP (Fibre Channel over Ethernet): Disconnects nodes from shared storage.

Each mechanism requires specific configuration to ensure it can reach and act on the target node. For example, an IPMI-based fence device might be configured with:

pcs fence device add ipmi1 ipmi_host=192.168.1.100 ipmi_user=admin ipmi_password=secret

Fencing Process

The fencing process is triggered when a node fails to respond to cluster heartbeats. Pacemaker evaluates the node's status and applies fencing policies defined in the cluster configuration. Key steps include:
1. Detection: A node is marked unreachable after multiple failed heartbeat checks.
2. Decision: The cluster determines if fencing is necessary (e.g., based on quorum rules).
3. Execution: The appropriate fencing agent isolates the node (e.g., power cycling it or disconnecting its storage access).

Fencing policies must explicitly define which nodes can be fenced and the order of operations. For example:

pcs constraint location node1 -inf: fence=off
pcs constraint location node2 -inf: fence=on

Configuration Examples

To verify fencing status and test configurations:

pcs status | grep 'Fencing'
# Output: Fencing: fence=on (node2), fence=off (node1)

To test a fencing action (use with caution):

pcs cluster enable --all
pcs cluster stop node2
pcs cluster start node2

Key takeaways

  • STONITH is essential for preventing split-brain by isolating faulty nodes.
  • Fencing mechanisms vary by hardware/software and require explicit configuration.
  • Fencing policies must define node isolation rules and priorities.
  • Regular testing and monitoring ensure fencing actions work reliably during failures.
  • Always validate fencing configurations in a safe environment before production use.