IBM PowerHA

PowerHA/SystemMirror

PowerHA implementation steps

PowerHA Limits 

Component	Maximum number/other limits
Nodes	32
Resource groups	64
Networks	48
Network interfaces, devices, and labels	256
Cluster resources	While 128 is the maximum that clinfo can handle, there can be more in the cluster
Parent-Child dependencies	Max of 3 levels
Site	2
Interfaces	7 interfaces per node per network
Application monitors per site	128
Persistent IP alias	One per node per network
XD_data networks	4 per cluster
GLVM Modes	Synchronous, Asynchronous, non concurrent
GLVM Devices	All PVs supported by AIX, no need to be same local and remote

Log Files

File	Description
/var/hacmp/adm/cluster.log	Generated by cluster scripts and daemons
/var/hacmp/log/hacmp.out	Generated by event scripts and utilities
/var/hacmp/adm/history/cluster.mmddyyyy	Cluster history files generated daily
/var/hacmp/clcomd/clcomd.log	Generated by clcomd daemon
/var/hacmp/clcomd/clcomddiag.log	Generated by clcomd daemon, debug information
/var/hacmp/clverify/clverify.log	Generated by Cluster Verification utility
/var/hacmp/log/autoverify.log	Generated by Auto Verify and Synchronize
/var/hacmp/log/clavan.log	Generated by Application Availability Analysis tool
/var/hacmp/log/clinfo.log	Generated by client node running clinfo
/var/hacmp/log/cl_testtool.log	Generated by the Cluster Test Tool
/var/hacmp/log/clconfigassist.log	Generated by Two-Node Cluster Configuration Assistant
/var/hacmp/log/clstrmgr.debug	Generated by the clstrmgr daemon
/var/hacmp/log/clstrmgr.debug.long	Detail information from the clstrmgr daemon
/var/hacmp/log/clutils.log	Generated by cluster utilities and file propagation
/var/hacmp/log/cspoc.log	Generated by CSPOC commands
/var/hacmp/log/cspoc.log.remote	Detail information from CSPOC commands
/var/hacmp/log/migration.log	Generated by cluster migration
/var/hacmp/log/sa.log	Generated by Application Discovery
"odmget HACMPlogs"	Displays a complete list of HACMP Log Files
/var/ha/log/topsvcs.default	Topology Services starup configuration log.
/var/ha/log/topsvcs.dd.hhmmss.lang	Topology Services start/stop log
/var/ha/log/topsvcs.dd.hhmmss	Topology Services activity log
/var/ha/log/nim.topsvcs.IF.clustername	NIM hearthbeat activity log for each interface.
/var/ha/log/nmDiag.topsvcs.IF.cluster	NIM diagnostic log for each interface.
/var/ha/log/grpsvcs.default.nodenum_instnum	Group Services startup log
/var/ha/log/grpsvcs_nodenum_instnum	Group Services activity log.

 
Useful HACMP Commands
 

Command	Purpose
clstat	Displays topology and resource groups status. clinfoES and snmpd must be running.
cldump	Displays topology & resource group status and configuration. snmpd must be running.
cldisp	Like cldump, but application oriented. snmpd must be running.
cltopinfo (cllsif)	Displays topology configuration.
clRGinfo (clfindres)	Displays resource group status.
clsshowres	Displays resource groups configuration.
clshowsrv	Calls lsrsc to display status of: HACMP subsystems (clshowrsrv -a) HACMP and RSCT subsystems (clshowsrv -v)
clcycle	Rotates selected log files.
clgetactivenodes	Displays active nodes. Must specify which nodo to ask (-n node)
clsnap	Save HACMP log files and configuration information.
cl_ls_shared_vgs	List shared vgs.
cl_lsfs	List shared fs.
cllsgrp	List the resource groups.
cllsres	Show short resource group information.
clRGmove	Brings a RG Offline/Online or move it
lssrc -ls clstrmgrES	Displays Cluster Services

Notes:

Installation changes 
The following AIX configuration changes are made: 

1. Files modified:
/etc/hosts
/etc/inittab
/etc/rc.net
/etc/services
/etc/snmpd.conf
/etc/snmpd.peers
/etc/syslog.conf
/etc/trcfmt
/var/spool/cron/crontab/root 

2. The hacmp group is added. 

3. Also, using cluster configuration and verification, the file /etc/hosts can be changed by adding or modifying entries. 

4. The following network options are set to “1” by RSCT topsvcs startup:
- nonlocsrcroute
- ipsrcrouterecv
- ipsrcroutesend
- ipsrcrouteforward
- ip6forwarding

5. The verification utility ensures that the value of each network option is consistent across all cluster nodes for the following settings:
- tcp_pmtu_discover
- udp_pmtu_discover
- ipignoreredirects
- routerevalidate

The cluster communications daemon
With the introduction of clcomdES, there is no need for an /.rhosts file to be configured. The cluster communications daemon is started by inittab, with the entry being created by the installation of PowerHA. The daemon is controlled by the system resource controller, so startsrc, stopsrc and refresh work. In particular, refresh is used to re-read /usr/es/sbin/cluster/etc/rhosts and moving the log files. The cluster communication daemon uses port 6191.

Resource group components:
Service IP Label
Volume Group
Filesystem
Application Server
NFS mounts
NFS exports



Resource group Startup options:
Online on home node only.
Online on first available node.
Online on all available nodes.
Online using distribution policy.


Resource group Fallover options:
Fall over to next priority node in list:
Fallover using dynamic node priority:
Bring offline (on error only)

Resource group Fallback options:
Fall back to higher priority node in list
Never fall back

Resource group attributes
Settling time
Delayed fallback timers
Distribution policy
Dynamic node priorities
Resource group processing order
Priority override location
Resource group dependencies - parent / child
Resource group dependencies - location

Resource Groups operations
Bring a resource group offline
Bring a resource group online
Move a resource group to another node/site
Suspend/resume application monitoring

Sources of HACMP information
HACMP manuals come with the product –READ THEM!
Sales Manual: www.ibm.com/common/ssi
/usr/es/sbin/cluster/release_notes
IBM courses:
– HACMP Administration I: Planning and Implementation (AU54/Q1554)
– HACMP Administration II: Administration and Problem Determination (AU61/Q1561)
– HACMP V5 Internals (AU60/Q1560)
IBM Web Site:
– http://www-03.ibm.com/systems/p/ha/
Non-IBM sources (not endorsed by IBM but probably worth a look):
– http://www.matilda.com/hacmp/
– http://groups.yahoo.com/group/hacmp/