IBM AIX 7.1 – Common Problem 136 – Diagnosis and Fix

📖 ~1 min read

Table of contents

1. Symptom & Impact
2. Environment & Reproduction
3. Root Cause Analysis
4. Quick Triage
5. Step-by-Step Diagnosis
6. Solution – Primary Fix
7. Solution – Alternative Approaches
8. Verification & Acceptance Criteria
9. Rollback Plan
10. Prevention & Hardening
11. Related Errors & Cross-Refs
12. References & Further Reading

Symptom & Impact

`clmgr query cluster` shows nodes in `JOINING` state and applications are active on both nodes.

Environment & Reproduction

AIX 7.2 PowerHA SystemMirror 7.2.7 two-node cluster after a heartbeat network outage.

Root Cause Analysis

Heartbeat network and SAN disk heartbeat both failed, causing dual ownership.

Quick Triage

Run `clmgr query node`, `lscluster -m`, and `clRGinfo -v` on both nodes.

Step-by-Step Diagnosis

Inspect `/var/hacmp/log/clutils.log` and `errpt` for CAA `RPC_FAIL` events.

Solution – Primary Fix

Bring the secondary down: `clmgr offline node ` and resync with `clmgr sync cluster`.

Solution – Alternative Approaches

Force the resource group home: `clRGmove -g -n -m`.

Verification & Acceptance Criteria

`clmgr query cluster` reports `STABLE` and the RG is online on exactly one node.

Rollback Plan

Restore from the cluster snapshot with `clsnapshot -a restore -n `.

Prevention & Hardening

Run two independent heartbeat paths (network + SAN) and test split-brain quarterly.

Related Errors & Cross-Refs

Pairs with `CONFIGRM` resource manager and `RMCD_INFO_*` errpt entries.

Related tutorial: View the step-by-step tutorial for aix-7.1.

View all aix-7.1 tutorials on the Tutorials Hub →

Browse all common problems & solutions on the Tutorials Hub.

References & Further Reading

IBM Docs: PowerHA SystemMirror 7.2.x admin guide, `clmgr`, CAA tuning.