Skip to content

Cluster Health

High availability clusters rely on continuous monitoring to detect failures, ensure resource availability, and maintain quorum. Tools like crm_mon, pcs status, and corosync-quorum provide critical insights into cluster health, node status, and resource assignments. Regular checks using these utilities help administrators proactively address issues before they impact service availability.

Real-Time Cluster Status with crm_mon

The crm_mon command provides a live view of the cluster's state, including node status, resource assignments, and fence device configurations. It is the primary tool for diagnosing cluster health during runtime.

Basic Usage

crm_mon -1
The -1 flag displays one line per node, showing their current status (e.g., Online, Offline, Stopped). For a detailed tree view: 
crm_mon

Interpreting Output

  • Node Status: Online indicates the node is operational; Offline suggests a failure or maintenance mode.
  • Resource Status: Resources like virtual IPs or services are marked as Started or Stopped.
  • Fence Devices: Indicates if fencing mechanisms are active (e.g., Fence or No Fence).

Example Output: 

Stack: corosync
Current DC: node1 (version 2.0.5-125.el7) - partition with quorum
Last updated: Thu Jul 18 14:30:00 2024
Last change: Thu Jul 18 14:25:00 2024 by root via cibadmin on node1

Cluster Configuration and Status with pcs status

The pcs status command provides a high-level overview of the cluster's configuration, including node status, resource placement, and lease information. It is ideal for verifying the cluster's configuration after changes.

Key Output Sections

  1. Cluster Status: Indicates if the cluster is active and has quorum.
  2. Node Status: Lists nodes and their current state (e.g., Online, Offline).
  3. Resources: Shows which resources are running and their assigned nodes.
  4. Leases: Displays lease information for resources, which is critical for failover.

Example Command

pcs status

Example Output: 

Cluster name: mycluster
Stack: corosync
Current DC: node1 (version 2.0.5-125.el7) - partition with quorum
Last updated: Thu Jul 18 14:30:00 2024
Last change: Thu Jul 18 14:25:00 2024 by root via cibadmin on node1

Nodes:
 Online: node1 node2

Resources:
 Resource: vip-clone (ocf::heartbeat:IPaddr2): Started node1
 Resource: web-server (ocf::heartbeat:apache): Started node1

 leases:
 Resource: vip-clone: lease is active

Quorum and Corosync Status with corosync-quorum

Quorum ensures the cluster remains functional even if some nodes fail. The corosync-quorum tool checks the quorum status and helps diagnose split-brain scenarios.

Checking Quorum Status

corosync-quorumstatus

Key Indicators

  • Quorum Status: Quorum: 2 means two nodes are in the majority (required for cluster operation).
  • Split-Brain Detection: If nodes are partitioned, the output may show Quorum: 0 or Quorum: 1, indicating a potential split-brain scenario.

Example Output: 

Quorum: 2
Nodes: 2

Interpreting Cluster Health

  • Normal Operation: All nodes are Online, resources are Started, and quorum is maintained.
  • Warning Signs:
  • Nodes marked Offline or Stopped.
  • Resources in Stopped state.
  • Quorum status below the required threshold.
  • Troubleshooting Steps: Use crm_mon to identify failing resources, pcs status to verify configuration, and corosync-quorum to check for network partitions.

Key takeaways

  • Use crm_mon for real-time cluster state and resource tracking.
  • Run pcs status to validate configuration and resource placement.
  • Monitor quorum with corosync-quorum to prevent split-brain scenarios.
  • Regularly check node status, resource availability, and quorum health to maintain cluster reliability.