IBM General Parallel File System (GPFS)

Introduction

The General Parallel File System (GPFS) is a high-performance clustered file system developed by IBM. It can be deployed in shared-disk or shared-nothing distributed parallel modes. It is used by many of the world's largest commercial companies, as well as some of the supercomputers on the Top 500 List. For example, GPFS was the filesystem of the ASC Purple Supercomputer which was composed of more than 12,000 processors and has 2 petabytes of total disk storage spanning more than 11,000 disks.

In common with typical cluster filesystems, GPFS provides concurrent high-speed file access to applications executing on multiple nodes of clusters. It can be used with AIX 5L clusters, Linux clusters, on Microsoft Windows Server, or a heterogeneous cluster of AIX, Linux and Windows nodes. In addition to providing filesystem storage capabilities, GPFS provides tools for management and administration of the GPFS cluster and allows for shared access to file systems from remote GPFS clusters.+

GPFS has been available on IBM's AIX since 1998, on Linux since 2001, and on Windows Server since 2008, and it is offered as part of the IBM System Cluster 1350. The most recent release of GPFS 3.5 offers Active File Management to enable asynchronous access and control of local and remote files, thus allowing for global file collaboration.

Below you'll find a complete list of all the GPFS commands, the same are located under /usr/lpp/mmfs/bin, enjoy!

Working with the Cluster

Command	Description
mmcrcluster	Creates GPFS Cluster Usage: mmcrfs Device {″DiskDesc[;DiskDesc...]″ | -F DescFile} [-A {yes | no | automount}] [-D {nfs4 | posix}] [-B BlockSize] [-E {yes | no}] [-j {cluster | scatter}] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-L LogFileSize] [-m DefaultMetadataReplicas] [-M MaxMetadataReplicas] [-n NumNodes] [-N NumInodes[:NumInodesToPreallocate]] [-Q {yes | no}] [-r DefaultDataReplicas] [-R MaxDataReplicas] [-S {yes | no}] [-t DriveLetter] [-T MountPoint] [-v {yes | no}] [-z {yes | no}] [--version Version]
mmlscluster	Displays GPFS cluster configuration information Usage: mmlscluster [--cnfs]
mmlsconfig	Displays the configuration data for a GPFS cluster Usage: mmlsconfig <attribute>
mmgetstate	Displays the state of the GPFS daemon on one or more nodes Usage: mmgetstate [-L] [-s] [-v] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmaddnode	Adds nodes to a GPFS cluster Usage: mmaddnode -N {NodeDesc[,NodeDesc...] | NodeFile}
mmchlicense	Designates appropriate GPFS licenses Usage: mmchlicense {server | client} [--accept] -N {Node[,Node...] | NodeFile | NodeClass}
mmlslicense	Displays information about the GPFS node licensing designation. Usage: mmlslicense [-L]
mmchnode	Changes node attributes Usage: mmchnode change-options -N{Node[,Node...] | NodeFile | NodeClass} Or, Usage: mmchnode {-S Filename | --spec-file=Filename}
mmdelnode	Deletes nodes from a GPFS cluster Usage: mmdelnode {-a | -N Node[,Node...] | NodeFile | NodeClass}
mmchcluster or mmchconfig	Changes the GPFS cluster configuration data
mmlsmgr	Displays the file system manager node Usage: mmlsmgr [Device [Device...]] Or,Usage: mmlsmgr -C ClusterName Or,Usage: mmlsmgr -c
mmchmgr	Changes the file system manager node Usage: mmchmgr {Device | -c} [Node]
mmstartup	Starts GPFS cluster Usage: mmstartup [-a | -N {Node[,Node...] | NodeFile | NodeClass}] [-E EnvVar=value ...]
mmshutdown	Stops GPFS cluster Usage: mmshutdown [-t UnmountTimeout ] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmrefresh	Places the most recent GPFS cluster configuration data files on the specified nodes
mmsdrrestore	Restores the latest GPFS system files on the specified nodes.
mmsdrbackup	Performs a backup of the GPFS configuration data.
mmbackupconfig	Backing up file system configuration information Usage: mmbackupconfig Device -o OutputFile
mmrestoreconfig	Restores file system configuration information Usage: mmrestoreconfig Device -i InputFile [-I {test | yes | continue }] [-Q {yes | no}] [-W NewDeviceName] [-z {yes | no}] Or,Usage: mmrestoreconfig Device -i InputFile -F configFile Or, Usage: mmrestoreconfig Device -i InputFile -I continue

Working with File Systems

Command	Description
mmcrfs	Creates a file system Usage: mmcrfs Device {″DiskDesc[;DiskDesc...]″ | -F DescFile} [-A {yes | no | automount}] [-D {nfs4 | posix}] [-B BlockSize] [-E {yes | no}] [-j {cluster | scatter}] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-L LogFileSize] [-m DefaultMetadataReplicas] [-M MaxMetadataReplicas] [-n NumNodes] [-N NumInodes[:NumInodesToPreallocate]] [-Q {yes | no}] [-r DefaultDataReplicas] [-R MaxDataReplicas] [-S {yes | no}] [-t DriveLetter] [-T MountPoint] [-v {yes | no}] [-z {yes | no}] [--version Version]
mmmount	Mounts a file system Usage: mmmount {Device | DefaultMountPoint | DefaultDriveLetter | all | all_local | all_remote} [-o MountOptions] [-a | -N {Node[,Node...] | NodeFile | NodeClass}] Or,Usage: mmmount Device {MountPoint | DriveLetter } [-o MountOptions] [-a | -N {Node[,Node...] | NodeFile | NodeClass}]
mmumount	Umounts a file system Usage: mmumount {Device | MountPoint | DriveLetter | all | all_local | all_remote} [-f ] [-a | -N {Node[,Node...]| NodeFile | NodeClass}] Or,  Usage: mmumount Device -f -C {all_remote | ClusterName} [-N Node[,Node...]]
mmdelfs	Deletes a file system Usage: mmdelfs Device [-p]
mmdf	Queries available file space on a GPFS file system Usage: mmdf Device [-d | -F | -m] [-P PoolName]
mmlsmount	Determines which nodes have a file system mounted Usage: mmlsmount {Device | all | all_local | all_remote} [-L ] [-C {all | all_remote | ClusterName[,ClusterName...] } ]
mmfsck	Checks and repairs a file system Usage: mmfsck Device [-n | -y] [-c | -o] [-t Directory] [ -v | -V] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmlsfs	Listing file system attributes Usage: mmlsfs {Device | all | all_local | all_remote} [-A] [-a] [-B] [-D] [-d] [-E] [-F] [-f] [-I] [-i] [-j] [-k] [-K][-L] [-M] [-m] [-n] [-o] [-P] [-Q] [-R] [-r] [-S] [-t] [-T] [-u] [-V] [-z]
mmchfs	Modifies file system attributes Usage: mmchfs Device [-A {yes | no | automount}] [-E {yes | no}] [-D {nfs4 | posix}] [-F MaxNumInodes[:NumInodesToPreallocate]] [-k {posix | nfs4 | all}] [-K {no | whenpossible | always}] [-m DefaultMetadataReplicas] [-n NumberOfNodes] [-o MountOptions] [-Q {yes | no}] [-r DefaultDataReplicas] [-S {yes | no} ] [-T Mountpoint] [-t DriveLetter] [-V {full | compat}] [-z {yes | no}] Or,  Usage: mmchfs Device [-W NewDeviceName]
mmlsattr	Querying file replication attributes Usage: mmlsattr [-l] [-L] FileName [FileName...]
mmchattr	Changing file replication attributes Usage: mmchattr [-m MetadataReplicas] [-M MaxMetadataReplicas] [-r DataReplicas] [-R MaxDataReplicas] [-P DataPoolName] [-D {yes | no} ] [-I {yes | defer}] [-i {yes | no}] [-l] Filename [Filename...]
mmrestripefs	Restriping a GPFS file system Usage: mmrestripefile {-b | -m | -p | -r} {-F FilenameFile | Filename [Filename...]}
mmdefragfs	Querying and reducing file system fragmentation Usage: mmdefragfs Device [-i] [-u BlkUtilPct] [-P PoolName] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmbackup	Backing up a file system Usage: mmbackup {Device | Directory} [-f] [-g GlobalWorkDirectory] [-N {Node[,Node...] | NodeFile | NodeClass}] [-s LocalWorkDirectory] [-S SnapshotName] [-t {full | incremental}]

Working with Snapshots

Command	Description
mmcrsnapshot	Creates a snapshot of an entire GPFS file system at a single point in time Usage: mmcrsnapshot Device Directory
mmdelsnapshot	Deletes a GPFS snapshot Usage: mmdelsnapshot Device Directory [-N {Node [,Node...] | NodeFile | NodeClass}]
mmrestorefs	Restores a file system from a GPFS snapshot Usage: mmrestorefs Device Directory [-c]
mmlssnapshot	Displays GPFS snapshot information for the specified file system Usage: mmlssnapshot Device [-d] [-Q]
mmsnapdir	Creates and deletes invisible directories that connect to the snapshots of a GPFS file system, and changes the name of the snapshots subdirectory Usage: mmsnapdir Device {[-r | -a] [-s SnapDirName]} Or, Usage: mmsnapdir Device [-q]

Working with Filesets

Command	Description
mmcrfileset	Creates a GPFS fileset Usage: mmcrfileset Device FilesetName [-t Comment]
mmchfileset	Changes the attributes of a GPFS fileset Usage: mmchfileset Device {FilesetName | -J JunctionPath} {[-j NewFilesetName] | [-t NewComment]}
mmlsfileset	Displays status and attributes of GPFS filesets Usage: mmlsfileset Device {[[Fileset [,Fileset ...]] [-J Junction [,Junction ...]] | -F FileName} [-L] [-d] [-i]
mmdelfileset	Deletes a GPFS fileset Usage: mmdelfileset Device FilesetName [-f]
mmrestripefile	Rebalances or restores the replication factor of the specified files Usage: mmrestripefs Device {-b | -m | -p | -r | -R} [-N {Node[,Node...] | NodeFile | NodeClass}] [-P PoolName]
mmlinkfileset	Creates a junction that references the root directory of a GPFS fileset Usage: mmlinkfileset Device FilesetName [-J JunctionPath]
mmunlinkfileset	Removes the junction to a GPFS fileset Usage: mmunlinkfileset Device {FilesetName | -J JunctionPath} [-f]

Working with Disks

Command	Description
mmcrnsd	Creates network shared disk servers Usage: mmcrnsd -F DescFile [-v {yes |no}]
mmlsnsd	Displaying disks in a GPFS cluster Usage: mmlsnsd [-a | -F | -f Device | -d ″DiskName[;DiskName...]″ ] [-L | -m | -M | -X] [-v]
mmdelnsd	Deletes NSDs from the GPFS cluster Usage: mmdelnsd {″DiskName[;DiskName...]″ | -F DiskFile}
mmadddisk	Adding disks to a file system Usage: mmadddisk Device {″DiskDesc[;DiskDesc...]″ | -F DescFile} [-a] [-r] [-v {yes | no}] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmdeldisk	Deleting disks to a file system Usage: mmdeldisk Device {″DiskName[;DiskName...]″ | -F DescFile} [-a] [-c] [-m | -r | -b] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmrpldisk	Replacing disks in a GPFS file system Usage: mmrpldisk Device DiskName {DiskDesc | -F DescFile} [-v {yes | no}] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmlsdisk	Displaying GPFS disk states Usage: mmlsdisk Device [-d ″DiskName[;DiskName...]″] [-e] [-L] Or,Usage: mmlsdisk Device [-d ″DiskName[;DiskName...]″] {-m | -M}
mmchdisk	Changing GPFS disk states and parameters Usage: mmchdisk Device {suspend | resume | stop | start | change} -d ″DiskDesc [;DiskDesc...]″ | -F {DescFile} [-N {Node [,Node...] | NodeFile | NodeClass}] Or,  Usage: mmchdisk Device {resume | start} -a [[-N {Node[,Node...] | NodeFile | NodeClass}]
mmchnsd	Changing your NSD configuration Usage: mmchnsd {″DiskDesc[;DiskDesc...]″ | -F DescFile}
mmnsddiscover	Rediscovers paths to the specified network shared disks Usage: mmnsddiscover [-a | -d ″Disk[;Disk...]″ | -F DiskFile] [-C ClusterName] [-N {Node[,Node...] | NodeFile | NodeClass}]
mmcrvsd	Creates virtual shared disks for use by GPFS Usage: mmcrvsd [-f FanoutNumber] [-y] [-c] -F DescFile

Working with Quotas

Command	Description
mmcrfs or mmchfs -Q yes	Enabling and disabling GPFS quota management
mmedquota	Explicitly establishing and changing quotas Usage: mmedquota {-u [-p ProtoUser] User...| -g [-p ProtoGroup] Group... | -j [-p ProtoFileset ] Device:Fileset... | -d {-u User...| -g Group... | -j Device:Fileset...} | -t {-u | -g | -j}}
mmcheckquota	Checking quotas Usage: mmcheckquota [-v] [-N {Node[,Node...] | NodeFile | NodeClass}] {Device [Device...] | -a} Or,  Usage: mmcheckquota {-u UserQuotaFilename | -g GroupQuotaFileName | -j FilesetQuotaFilename} Device
mmdefquotaon	Enabling default quota Usage: mmdefquotaon [-u] [-g] [-j] [-v] [-d] {Device [Device... ] | -a}
mmdefedquota	Define default quota Usage: mmdefedquota {-u | -g | -j} Device
mmdefquotaoff	Disable default quota Usage: mmdefquotaoff [-u] [-g] [-j] [-v] {Device [Device...] | -a}
mmlsquota	Listing quotas Usage: mmlsquota [-u User | -g Group | -j Fileset] [-v | -q] [-e] [-C ClusterName] [Device [Device...]] Or,  Usage: mmlsquota -d {-u | -g | -j} [-C ClusterName] [Device [Device...]]
mmquotaon	Activating quota limit checking Usage: mmquotaon [-u] [-g] [-j] [-v] {Device [Device...] | -a}
mmquotaoff	Deactivating quota limit checking Usage: mmquotaoff [-u] [-g] [-j] [-v] {Device [Device ...] | -a}
mmrepquota	Creating file system quota reports Usage: mmrepquota [-e] [-g] [-q] [-u] [-n] [-v] [-j] {Device [Device...] | -a}

Working with ACLs

Command	Description
mmputacl	Setting traditional GPFS access control lists Usage: mmputacl [-d] [-i InFilename] Filename
mmgetacl	Displaying traditional GPFS access control lists Usage: mmgetacl [-d] [-o OutFilename] [-k {nfs4 |posix | native}] Filename
mmeditacl	Changing traditional GPFS access control lists Usage: mmeditacl [-d] [-k {nfs4 | posix | native}] Filename
mmdelacl	Deleting traditional GPFS access control lists Usage: mmdelacl [-d] Filename

Working with Remote Clusters

Command	Description
mmauth	Manages secure access to GPFS file systems Usage: mmauth genkey {new | commit | propagate [-N {Node[,Node...] | NodeFile | NodeClass}]} Or,  Usage: mmauth add RemoteClusterName -k KeyFile -l CipherList Or,  Usage: mmauth update RemoteClusterName -C NewClusterName -k KeyFile [-l CipherList] Or,  Usage: mmauth delete {RemoteClusterName | all } Or, Usage: mmauth grant {RemoteClusterName | all } -f { Device | all } [-a {rw | ro} ] [-r {uid:gid | no}] Or,  Usage: mmauth deny {RemoteClusterName | all } -f { Device | all } Or,Usage: mmauth show [RemoteClusterName | all]
mmexportfs	Exports a file system into the cluster Usage: mmexportfs {Device | all} -o ExportFilesysData
mmimportfs	Imports a file system into the cluster Usage: mmimportfs {Device | all} -i ImportfsFile [-S ChangeSpecFile]
mmremotecluster	Manages information about remote clusters Usage: mmremotecluster add RemoteClusterName [-n ContactNodes] [-k KeyFile] Or,  Usage: mmremotecluster update RemoteClusterName [-C NewClusterName] [-n ContactNodes] [-k KeyFile] Or,Usage: mmremotecluster delete {RemoteClusterName | all} Or, Usage: mmremotecluster show [RemoteClusterName | all]
mmremotefs	Manages the information needed for mounting remote GPFS file systems Usage: mmremotefs add Device -f RemoteDevice -C RemoteClusterName [-T MountPoint] [-t DriveLetter] [-A {yes | no | automount}] [-o MountOptions] Or,  Usage: mmremotefs delete {Device | all | -C RemoteClusterName} Or,  Usage: mmremotefs show [Device | all | -C RemoteClusterName] Or, Usage: mmremotefs update Device [-f RemoteDevice] [-C RemoteClusterName] [-T MountPoint] [-t DriveLetter] [-A {yes | no | automount}] [-o MountOptions]

Miscellaneous

Command	Description
mmaddcallback	Registers a user-defined command into the GPFS system Usage: mmaddcallback CallbackIdentifier --command CommandPathname --event Event[,Event...] [--priority Value][--async | --sync [--timeout Seconds] [--onerror Action]] [-N {Node[,Node...] | NodeFile | NodeClass}] [--parms ParmeterString ...]
mmdelcallback	Deletes one or more user-defined callbacks from the GPFS system Usage: mmdelcallback CallbackIdentifier[,CallbackIdentifier...]
mmlscallback	Lists callbacks that are currently registered in the GPFS system Usage: mmlscallback [CallbackIdentifier[,CallbackIdentifier...]]
mmapplypolicy	Deletes files or migrates file data between storage pools within a single file system in accordance with policy rules. Usage: mmapplypolicy {Device | Directory} [-A IscanBuckets] [-a IscanThreads] [-B MaxFiles] [-D yyyy-mm-dd [@hh:mm[:ss]]] [-e] [-f FileListPrefix] [-g GlobalWorkDirectory] [-I {yes | defer | test | prepare}] [-i InputFileList] [-L n] [-M  name=value...] [-m ThreadLevel] [-N {all | mount | Node[,Node...] | NodeFile | NodeClass}] [-n DirThreadLevel] [-P PolicyFile] [-q] [-r FileListPathname...] [-S SnapshotName] [-s LocalWorkDirectory]
mmchpolicy	Establishes policy rules for a GPFS file system Usage: mmchpolicy Device PolicyFileName [-t DescriptiveName ] [-I {yes | test} ]
mmlspolicy	Displays policy information Usage: mmlspolicy Device [-L]
mmpmon	Monitors GPFS performance on a per-node basis Usage: mmpmon [-i CommandFile] [-d IntegerDelayValue] [-p] [-r IntegerRepeatValue] [-s] [-t IntegerTimeoutValue]
mmtracectl	Sets up and enables GPFS tracing Usage: mmtracectl {--start | --stop | --off | --set} [--trace={io | all | def | ″Class Level [Class Level ...]″] [--trace-recycle={off | local | global | globalOnShutdown}] [--aix- trace-buffer-size=BufferSize] [--tracedev-buffer-size=BufferSize] [--tracedev-compression-level=Level] [--tracedev-overwrite-buffersize= BufferSize] [--tracedev-write-mode={blocking | overwrite}] [--trace-file-size=FileSize] [--trace-dispatch={yes | no}] [-N {Node [,Node...] | NodeFile | NodeClass}]
gpfs.snap	Tools to gather preliminary data when a GPFS problem is encountered Usage: gpfs.snap [-c "CommandString"] [-d OutputDirectory] [-z] [-a | -N {Node[,Node...] | NodeFile | NodeClass}] [--check-space | --no-check-space | --check-space-only] [--exclude-aix-disk-attr] [--exclude-aix-lvm] [--exclude-net] [--exclude-merge-logs] [--mmdf]
mmfsadm	Extracts data from GPFS without using locking, so that it can collect the data in the event of locking errors Usage: TBD
mmfileid	Determine which files are located on areas of a disk that are damaged or considered to be suspect Usage: mmfileid Device {-d DiskDesc|-F DescFile} [-o OutputFile] [-f NumThreads] [-t Directory] [-N {Node[,Node...]  | NodeFile | NodeClass}]
mmfsctl	Issues a file system control request Usage: mmfsctl Device {suspend | resume} Or,Usage: mmfsctl Device {exclude | include} {-d ″DiskName[;DiskName...]″ | -F DiskFile | -G FailureGroup} Or, Usage: mmfsctl Device syncFSconfig {-n RemoteNodesFile | -C remoteClusterName} [-S SpecFile]
dsh	Concurrently runs commands on multiple nodes and hardware devices. Usage: TBD

Specifying nodes as input to GPFS commands

Many GPFS commands accept a node or multiple nodes as part of their input, using the -N flag. Nodes
can be specified to GPFS commands in a variety of ways:

Node A representation of an individual node, which can be any of these:

• Short GPFS administration node interface name.
• Long GPFS administration node interface name.
• Short GPFS daemon node interface name.
• Long GPFS daemon node interface name.
• IP address corresponding to the GPFS daemon node interface.
• GPFS node number.

Node - Node
A node range, indicated by specifying two node numbers separated by a hyphen (-), with the
first node number being less than or equal to the second node number. For example, node range
3-8 specifies the nodes with node numbers 3, 4, 5, 6, 7, and 8.

NodeClass
A set of nodes that share a particular characteristic. These node classes are known to GPFS:

• all All of the nodes in the GPFS cluster.
• clientnodes: All nodes that do not participate in file system administration activities.
• managernodes: All nodes in the pool of nodes from which file system managers and token managers are selected.
• mount: For commands involving a file system, all of the local nodes on which the file system is mounted
• nonquorumnodes: All of the non-quorum nodes in the GPFS cluster.
• nsdnodes: All of the NSD server nodes in the GPFS cluster.
• quorumnodes: All of the quorum nodes in the GPFS cluster.

NodeFile
A file that contains a list of nodes. A node file can contain individual nodes or node ranges.

Logs, traces and dumps

GPFS writes both operational messages and error data to the MMFS log file. The MMFS log can be found in the /var/adm/ras directory on each node.

The MMFS log file is named mmfs.log.date.nodeName, where date is the time stamp when the instance of
GPFS started on the node and nodeName is the name of the node. The latest mmfs log file can be found by using the symbolic file name /var/adm/ras/mmfs.log.latest.

The MMFS log from the prior startup of GPFS can be found by using the symbolic file name /var/adm/ras/mmfs.log.previous. All other files have a timestamp and node name appended to the file name.

The error log (errpt) contains information about several classes of events or errors. These classes are:

• “MMFS_ABNORMAL_SHUTDOWN”
• “MMFS_DISKFAIL”
• “MMFS_ENVIRON”
• “MMFS_FSSTRUCT”
• “MMFS_GENERIC”
• “MMFS_LONGDISKIO”
• “MMFS_QUOTA”
• “MMFS_SYSTEM_UNMOUNT”
• “MMFS_SYSTEM_WARNING”

Configuration Files

/var/mmfs/gen/mmfsNodeData	Contains GPFS cluster configuration data pertaining to the node.
/var/mmfs/gen/mmsdrfs	Contains a local copy of the mmsdrfs file found on the primary and secondary GPFS cluster configuration server nodes.
/var/mmfs/gen/mmfs.cfg	Contains GPFS daemon startup parameters.

GPFS port usage

The nodes in a GPFS cluster communicate with each other using the TCP/IP protocol. The port number used by the main GPFS daemon (mmfsd) is controlled with the tscTcpPort configuration parameter. The default port number is 1191.

Sources: 

General Parallel File System - Administration and Programming Reference - Version 3 Release 3(SA23-2221-03)
General Parallel File System -  Problem Determination Guide - Version 3 Release 3 (GA76-0415-03)
General Parallel File System - Advanced Administration Guide - Version 3 Release 3 (SC23-5182-03)