Fencing Devices

Pacemaker relies on STONITH (Shoot The Other Node In The Head) devices to ensure data consistency by isolating failed nodes in a cluster. Proper configuration of STONITH devices is critical to prevent split-brain scenarios and ensure reliable failover. This section guides you through discovering, configuring, and validating STONITH devices in a Pacemaker cluster.

Discovering and Configuring STONITH Devices¶

Before configuring a STONITH device, you must first identify supported agents using the pcs stonith discover command. This command lists available STONITH agents, not network devices. Use agent-specific discovery tools for network devices.

Example: Listing Available STONITH Agents¶

pcs stonith discover

Output might include:

ipmi
apc

Example: Discovering IPMI-based STONITH Devices¶

Use fence_ipmi -d for IPMI device discovery:

fence_ipmi -d

Output might include:

IPMI device found at: 192.168.1.100

Example: Discovering APC UPS Devices¶

Use fence_apcups -d for APC device discovery:

fence_apcups -d

Output might include:

APC device found at: 192.168.1.200

Once discovered, configure the device using pcs stonith add. Parameters depend on the STONITH agent.

Example: Configuring an IPMI Device¶

pcs stonith add ipmi-01 ipmi \
  --ipaddr=192.168.1.100 \
  --username=admin \
  --password=securepassword \
  --chassis-type=blade

Example: Configuring an APC UPS¶

pcs stonith add apc-01 apc \
  --ipaddr=192.168.1.200 \
  --username=admin \
  --password=securepassword

Note: For IP-based fencing (e.g., fence_ipmi), ensure the device IP is reachable from all cluster nodes and that firewall rules allow communication on the required port (typically 6666 for IPMI).

Defining Fencing Policies¶

Pacemaker uses the stonith resource to define fencing policies. These policies determine how and when nodes are fenced. Configure stonith resources using pcs commands, not manual XML editing.

Example: Defining a STONITH Resource¶

pcs stonith add ipmi-01 ipmi \
  --ipaddr=192.168.1.100 \
  --username=admin \
  --password=securepassword \
  --chassis-type=blade

Key Fencing Policy Parameters¶

stonith-action: Specifies the fencing method (e.g., reboot, off, poweroff).
reboot-node: Enables node reboot after fencing (useful for IPMI).
fail-count: Number of consecutive failures before fencing is triggered.
timeout: Timeout in seconds for fencing operations.

Example configuration with parameters:

pcs stonith update ipmi-01 \
  --stonith-action=reboot \
  --fail-count=3 \
  --timeout=30

Cluster-Wide Fencing Level¶

Set the fencing-level via stonith resource properties:

pcs stonith update ipmi-01 \
  --fencing-level=strict

- strict: Fences nodes only if they are unreachable. - normal: Fences nodes if they are suspected of failing. - none: Disables fencing (not recommended).

Testing STONITH Configuration¶

After configuring STONITH devices, validate the setup to ensure it works as expected.

Example: Verifying Fencing Configuration¶

pcs cluster verify

This command checks for syntax errors in the cluster configuration.

Example: Simulating a Node Failure¶

pcs node disable <node-name>

This simulates a node failure and triggers the fencing policy. Monitor the output to confirm the STONITH device isolates the node.

Example: Checking Fencing Logs¶

journalctl -u pacemaker -n 50

Review logs for errors during fencing operations, such as authentication failures or network issues.

Key takeaways¶

Use pcs stonith discover to identify supported STONITH agents. Use agent-specific tools (e.g., fence_ipmi -d) for network device discovery.
Configure devices with pcs stonith add, specifying parameters like IP, credentials, and chassis type.
Define fencing policies via the stonith resource, adjusting stonith-action, fail-count, and timeout as needed.
Test fencing with pcs cluster verify and simulated node failures to ensure reliability.
Regularly monitor logs with journalctl to troubleshoot fencing-related issues.