Purpose (top - intro - usage - overview - examples)

Describe the ARSC push command. The push command originated to propagate ConfigFiles to nodes and has additional functionality to execute commands on remote nodes, copy files, and back-copy files. The command uses the machines.* file for identifying target nodes for command or copy. Although push is self-documenting with the '-?' option, understanding the command structure (technical overview) and specific usage examples in this document may be necessary to fully exploit capabilities.

Document Index / Related Documentation

Introduction (top - intro - usage - overview - examples)

It is easier to understand push by reviewing some command examples (each links to command output in examples at bottom):

1. mcgee: push all uname -a
   Push command to all nodes.
   
2. mcgee: push cmd -eq uname -a
   Push command to all nodes (with less noise).
   Same as above but output is cleaner by '-e'choing 'host:' w/output and '-q'uieter without summary reporting.
3. csmflyer: push --eq cmd 'if [ 2 == `uname -r` ]; then uname -a; else oslevel -s; fi'
   Push complex commands.
   
   • The syntax is ksh.
   • By default command errors are trapped terminating the command string.
   • The exit code is checked for reporting nodes with errors.
   • If first node exits with error by default rest are skipped.
   • Note in this example the 'eq' preceded the cmd.
4. csmflyer: push -os 5.3 cmd -q "hostname; uptime; df-h -q /tmp; echo"
   Push commands to all AIX 5.3 nodes.
   Note just '-q' was used (host name not echo'd except from hostname command).

5. csmflyer: push config -eq inetd.conf
   Push ConfigFile to all nodes.
   
6. csmflyer: push one csmflyer inetd.conf
   Push ConfigFile to one (not all) node.
   Multiple nodes can be comma separate or space separated if enclosed in quotes.
7. csmflyer: push -m f2n1,f1n1 config -eq bin/passwd
   Push ConfigFile to multiple nodes.
   Effectively same as above.
   Note, the ConfigFile is matched right-to-left, the 'bin/' was required to distinguish from /etc/passwd.

8. csmberg: push -u io file -q mmq
   Safe copy a file to all IO nodes.
   With a single argument it copies to same location on remote nodes. Like ConfigFiles, this will append ':new' to the file name, mv any existing to the /usr/software/spool/backout directory, then mv the file into place. The safe copy will preserve ownership only if run '-s[udo]|-root'.
9. csmberg: push -u io rcp -q mmq
   Copy a file to all IO nodes.
   Same as above but not "safe" unless '-s' is added, in this case ':new' is not appended.
   Any 'export PUSH_*' variables are also not cleared.
10. csmberg: push -u io -m !b1n6 rcp -rq mmq mmq
    Reverse copy files from IO nodes to server, excluding b1n6..
    The node name will be appended to the local (csmberg) copy.
    This is useful for back-copying files into a ConfigFiles directory.

11. mcgee: push -s -ibm -u admin --eq cmd /usr/local/krb5/bin/klist -ekt /etc/krb5.keytab
    Check keytabs on IBM admin nodes (use ssh-adm root).
    Using '-s[udo]|-root' or '-sysmon' can be used to nodes where ssh-adm has been configured.

There are many conceivable variations to the above. Alternative machines.list files can be built. At this time nodes are done serially which can take time on large clusters. It is important that offline machines are updated in the machines.list file to avoid rsh|ssh|rcp|scp timeout delays. The dsh command on the IBMs, and pdsh being reviewed for other platforms, do parallel execution and may respect node offline state automatically, review man pages for those commands.

A command push does the following:

$R_Command /bin/ksh <<!EOF |
if [ -n "$KSH_VERIFY" ]; then set -$KSH_VERIFY; fi
if [ -z "$OPT_t$OPT_T" ]; then  trap 'echo "!Err$ONTO:\$?" >&2; exit 1' ERR; fi
if [ -z "$OPT_p" ]; then        PATH=$PUSH_PATH; fi
$COMMAND
sts=\$?
if [ 0 = \$sts ]; then  echo "$OK"; exit 0; else exit 1; fi
!EOF
while read push_rsh_line; do
...

Note:

• Syntax is specifically ksh.
• Use 'export KSH_VERIFY=x' if debugging is required.
• By default an error trap is established.
• By default PATH is passed for remote commands.
• A zero exit from last command is used to echo a string detected by the subsequent 'while read' to check for normal completion. This is necessary because rsh|ssh can only report if the command was attempted and not actual status.

A ConfigFiles push conceptually does the following:

sudo scp -p file host:file:new
sudo ssh host "chown owner:group file:new;
               mv file /usr/software/spool/backout/file.date-time;
               mv file:new file"

Note:

• The "safe" copy is used to not clobber existing file.
• The ownership is set which 'rcp|scp -p' do not do by default.
• The old file is mv'd to a backout directory before the new is mv'd into place.
  This leaves a backout copy on the local node and, for binaries on the same filesystem as backout directory, preserves the inode so any open accesses to the file are preserved.
• In some cases additional commands may need to be push'd to activate a file, such as a newaliases for /etc/aliases. Note, crontabs are reloaded with a ConfigFiles push.

Usage (top - intro - usage - overview - examples)

The push command is self-documenting by using -?.
Platforms may have different type/* files.
Usage for all (properly constructed) type/* files is also displayed.

admin: push -?

Usage:  push [-s|-root|-sysmon] [-Features] type [-Options] arguments
   -s|-root  Re-invoke w/sudo (must be first option)
   -sysmon   Re-invoke as sysmon (sudo -u sysmon)

Features:
  -sgi    Override for Irix ConfigFiles
  -sun    Override for SunFiles
  -ibm    Override for IBM ConfigFiles
  -ibmnet Override for IBM Network Hosts
  -hmc    Override for HMC
  -lnx    Override for LinuxFiles

    Note, the following are superceded any type specific arguments:

  -m[achines]|-hosts|-w  Specify hosts,   e.g.: -hosts burbot,gold
  -t[ype]                Specify type,    e.g.: -type O2
  -g[roup]|-usage        Specify group,   e.g.: -usage staff
  -v[ersion]|-os         Specify version, e.g.: -os 6.5.17
  -a[vail]|-status       Specify status,  e.g.: -status offline
  -o[ther]|-frame|-rack  Specify other(sn) field check
      To generate an exclude list start with !, e.g.,
      -m !b1n5,b1n6 will exclude those nodes.
  --type_Options
#........................................................................
type_Options:
  Appropriate options are pre-set by the type wrapper in some cases
    but may be specified to over-ride certain behaviours.

  May be specified as either
      push type -options
  or: push --options type # default type settings take precedence

general:
  -q  quiet     do not echo hostname or display Summary
  -Q  Quiet     echo hostname:result (and no Summary)
  -S  Summary   display Summary
  -V  Verbose   display 'Push using' messages and Summary
  -v  verbose   show skipped hosts
  -l  local     local host exclusion from command
  -f  first     do NOT stop if first push fails
  -d  debug     show command but don't execute it
  -?  help
for rsh:
  -t  trap      do NOT trap errors, implies -f
  -T  trap      do NOT trap, but require normal exit
  -p  path      do NOT use PUSH_PATH
  -e  echo      echo host with command output
for rcp:
  -t  target    target==source[s]
  -p  path      do NOT imply /usr/local/adm/pkg/push/bin if no leading '/'
  -P  poff      do NOT use -p option for [r]cp
  -s  safe      do NOT clobber original with copy, see PUSH_BACKOUTS
  -c  config    use config file list, see PUSH_CONFIG,   implies -ts
  -C  config    use config file list, override NOPUSH files
  -r  reverse   reverse rcp (pull file vs. push file)
                  invalid with -s|-c|-t
                  appends . to target
  -x  xfer      imply rcp if function not defined

The default is rsh except if -s, -c, -r, or -x which default as rcp.

#Reference:  https://int.arsc.edu:8080/changeprob/configmgmt/push.html
#........................................................................
# Available push 'type' on mcgee:

Usage: push [-Features] all [-options] command|file(s)
Usage: push [-Features] all.but host1[,host2...] command
Usage: push [-Features] cmd [-options] command
Usage: push [-Features] config [-options] target(s)
Usage: push [-Features] config.but [-options] host1[,host2...] target(s)
Usage: push [-Features] file [-options] source [target]
Usage: push [-Features] one [-options] host1[,host2...] target(s)
Usage: push [-Features] rcp -options whatever
Exports:  PUSH_machine, PUSH_type, PUSH_group, PUSH_ver, ...
Usage: push [-Features] rsh -options whatever
Exports:  PUSH_machine, PUSH_type, PUSH_group, PUSH_ver, ...
Usage: push [-Features] staff [-options] command

Technical Overview (top - intro - usage - overview - examples)

• Command structure, /usr/local/adm/pkg/push:

  bin/
    push       - general command
    get_ConfigFiles.ksh - used by push.rcp.f to identify ConfigFiles
    get_Machines.ksh    - used by push.setup to identify target machines
  type/*       - wrappers (invoked by 'push wrapper-name'
  sbin/
    push.setup - setup & common functions for push rsh|rcp scripts
    push.rsh.f - push_rsh function and parsing, sourced by push.setup
    push.rcp.f - push_rcp function and parsing, sourced by push.setup
  

  The push command has two basic functions:
  1. Execute command on remote nodes.
  2. Copy files to or from remote nodes.
  To accomplish this, sub-functions in the sbin/ are invoked. The ConfigFiles push types are specialized copy and command functions.

  Between the general command and sbin/ functions PUSH_* environment variables may be passed. Many of the type/ files unset these variables and rely on -Features and -Options for targeting remote nodes.

• type/* files:

  The type/* files are very short scripts primarily for customized node selection or options. For each platform there may be different type/* files based on platform needs, the following are generally implemented:

  cmd	execute commands
  config	push ConfigFiles
  file	copy a file (with -s for "safe")
  one	push ConfigFiles to select nodes
  rsh	execute commands (preserves PUSH_* variables)
  rcp	copy a file (preserves PUSH_* variables)

  Using the -Features is an alternative to many type/* files, for example:
  • 'push -u staff cmd' is the same as 'push staff'
  • 'push -m b1n1,b1n2 config' is the same as 'push one b1n1,b1n2'
  The type/* files are invoked via 'push type' and invoke sbin/ functions after pre-setting PUSH_* variables. The following general capabilities are provided:
  • A (./push.setup) script utilized by other push scripts.
  • Determine rsh vs. rcp usage as appropriate for request.
  • Establishes options (Arg $1 if prefaced by '-').
  • Displays filtering options (PUSH_* environment variables).
  • Performs argument checking for rcp scripts.
  • Performs error checking (trapping) by default for rsh scripts.
  • Performs 'safe' (inode preservation) copies for rcp of config files.
  • Sets trap for quit or interupt for cleaner exit.
  • Invokes push_check to walk PUSH_LIST and execute.
  • Invokes push_status to report on results of push.
  Note:
  • The generic rsh and rcp assume 'export PUSH_*' variables have been properly pre-set by the user.
  • Most type/* files issue 'unset' commands for all PUSH_* variables before exporting the ones required for the wrapper.
  • The non-generic versions are generally "safer".

• PUSH_* variables:

  Typically these variables are cleared and set via type/* wrappres or via -Features.
  These variables do NOT generally need to set unless you have a specific need for the generic 'push rsh' or 'push rcp'.

  export  PUSH_machine=host     # include if machine found in list
  export  PUSH_type=type        # include if hardware type in list
  export  PUSH_avail=state      # include if available state in list
  export  PUSH_group=group      # include if usage group in list
  export  PUSH_ver=os_version   # include if OS version in list
  export  PUSH_sno=other        # include if other field in list
  
    PUSH_avail defaults to whatever the platform defines as "online" in
      the machines.list file, everything else defaults as "anything".
  
    Checks are cumulative (must match all include criteria).
  
    Lists can be specified with comma or space delimeter, e.g.:
          PUSH_machine="leinie weinhard molson"
          PUSH_machine="leinie weinhard molson"
  
    If initial character is '!' the list is exclude, i.e.:
          PUSH_group="!staffonly"
  
  export  PUSH_LIST=/usr/local/adm/etc/machines.all # host list default
  
  export  PUSH_OPTIONS=""                           # pre-defined options
  
    No default except as specified by type/* files.
    Used by 'push config' to pre-set "-cs".
  
  export  PUSH_PATH=$PATH         # Execution path for push_rsh function
  
    Utilized for push_rsh, defaults as $PATH on most platforms except for
      admin servers which are a different architecture than client nodes.
  
  export  PUSH_BACKOUTS=/usr/software/spool/backout # -s of rcp's
  
    Utilized by -s under push_rcp.
    Use 'PUSH_BACKOUTS=" "' to keep in same directory, this
      is the default for non-root usage.
    The file is copied as "*:New" then mv'd if copy is sucessful.
    Original file is saved (mv'd) as PUSH_BACKOUTS/TARGET:DATE.TIME.
    PUSH_BACKOUTS must be in the same file system to preserve inode.
    If root, chmod/chown/chgrp commands are issued to ensure
      target is the same as source.
  
  export  PUSH_DATE=":`date +%Y%m%d.%H%M%S`"      # backout date stamp
  
    Utilized by -s under push_rcp.
    Default trailer for PUSH_BACKOUTS is shown above.
  
  export  PUSH_CONFIG=/var/local/ConfigFiles/List.txt
  
    Utilized by -c under push_rcp.
    Default config file is as shown above.
    Under '-c' option of push_rsh use source|target based on config.
    Crontabs are copied to /usr/software/spool and updated with
      a 'crontab filename' command.
  
  export  PUSH_HOME=/var/local
  
    Utilized by -c under push_rcp.
    Default config file source location is as shown above.
  
  export  PUSH_BIN=/usr/local/adm/sbin
  
    Directory to find pre-defined functions.
    Default is as defined above.
  
  export  PUSH_FUNCTION=*         # Execution function
  
    Unless -s|-c|-x PUSH_FUNCTION will default to push_rsh.
    With   -s|-c|-x PUSH_FUNCTION will default to push_rcp.
    An other pre-defined functions are described below.
    Custom PUSH_FUNCTION's can be defined by the invoking script.
    PUSH_FUNCTION is responsible for returning non-0 for failures.
  

• Internal functions:

  push.setup	push_check	invokes get_Machines.ksh to generate list of target hosts invokes PUSH_FUNCTION checks status from PUSH_FUNCTION builds lists of skipped, executed, and failed hosts
  push.setup	push_status	displays counts and skipped|error|done hosts
  push.rsh.f	push_rsh	a generic push_rsh
  push.rcp.f	push_rcp	a generic push_rcp
  push.rcp.f	push_file_check	check if file exists, readable, and normal completes full path unless '-p' specified
  push.rcp.f	push_rcp_safe	'safe' push, copies to *:new, mv's old, mv's new if root ensures mode|ownership target==source
  get_ConfigFiles     .ksh	get_ConfigFiles	invoked within push.rcp.f identifies correct ConfigFiles extension for host

Examples (top - intro - usage - overview - examples)

1. Push command to all nodes. (intro, usage)

   mcgee: push all uname -a
   #
   #Host      pike:uname -a
   IRIX pike 6.5 10070056 IP32
   #
   #Host    pumice:uname -a
   IRIX64 pumice 6.5 10070055 IP30
   ...
   # Found 23 hosts: 13 skipped, 0 errors, 10 done, 0 N/A.
   #
   # Skipped:
   #       biglake chevak loon falcon steam chase bethel kotsina magma plume 
   # Done:
   #       pike pumice lava lynx igie marten gate iedata mcgrew mcgee
   

2. Push command to all nodes (with less noise). (intro, usage)

   mcgee: push cmd -eq uname -a
       pike:IRIX pike 6.5 10070056 IP32
     pumice:IRIX64 pumice 6.5 10070055 IP30
       lava:IRIX64 lava 6.5 10070055 IP30
       lynx:IRIX64 lynx 6.5 10070055 IP30
       igie:IRIX64 igie 6.5 10070055 IP35
     marten:IRIX64 marten 6.5 10070055 IP30
   Kerberos Principal Name [kcarlson]: ^d
   #Err: gate! Command completed with non-zero status
     iedata:IRIX iedata 6.5 10070056 IP32
     mcgrew:IRIX64 mcgrew 6.5 10070055 IP30
      mcgee:IRIX64 mcgee 6.5 10070055 IP30
   # Errors:
   #       gate
   

3. Push complex commands. (intro, usage)

   csmflyer: push --eq cmd 'if [ 2 == `uname -r` ]; then uname -a; else oslevel -s; fi'
     csmflyer:AIX csmflyer 2 5 00095C1A4C00
         f2n1:AIX iceflyer 2 5 000A684F4C00
         f1n1:AIX f1n1 2 5 0022F52A4C00
         f1n3:AIX f1n3 2 5 0022F52A4C00
         f3n1:AIX f3n1 2 5 00012DDA4C00
         f4n1:5300-05-04
         f4n2:5300-05-04
         f4n3:5300-05-04
         f4n4:5300-05-04
   

4. Push commands to all AIX 5.3 nodes. (intro, usage)

   csmflyer: push -os 5.3 cmd -q  "hostname; uptime; df-h -q /tmp; echo"
   f4n1
   07:19AM   up 3 days, 12 mins,  0 users,  load average: 0.04, 0.02, 0.02
   /dev/hd3                AIX  2.0G    2M  2.0G     1%  /tmp
   
   f4n2
   07:19AM   up 3 days,  20:36,  0 users,  load average: 0.00, 0.00, 0.01
   /dev/hd3                AIX  2.0G    2M  2.0G     1%  /tmp
   
   f4n3
   07:19AM   up 3 days,  20:37,  0 users,  load average: 0.00, 0.01, 0.00
   /dev/hd3                AIX  2.0G    2M  2.0G     1%  /tmp
   
   f4n4
   07:19AM   up 3 days,  20:36,  0 users,  load average: 0.46, 0.17, 0.16
   /dev/hd3                AIX  2.0G    2M  2.0G     1%  /tmp
   

5. Push ConfigFile to all nodes. (intro, usage)

   csmflyer: push config -eq inetd.conf
   
   push_rcp: 1 file(s)
     csmflyer:/bin/cp -p /var/local/ConfigFiles/etc/inetd.conf/csmflyer.Normal /etc/inetd.conf:new
         f2n1:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f2n1:/etc/inetd.conf:new
         f1n1:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f1n1:/etc/inetd.conf:new
         f1n3:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f1n3:/etc/inetd.conf:new
         f3n1:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f3n1:/etc/inetd.conf:new
         f4n1:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f4n1:/etc/inetd.conf:new
         f4n2:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f4n2:/etc/inetd.conf:new
         f4n3:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f4n3:/etc/inetd.conf:new
         f4n4:/usr/bin/rcp -p /var/local/ConfigFiles/etc/inetd.conf/Network f4n4:/etc/inetd.conf:new
   

6. Push ConfigFile to one (not all) node. (intro, usage)

   csmflyer: push one csmflyer inetd.conf
   
   push_rcp: 1 file(s)
   #
   #Host  csmflyer:
          csmflyer:/bin/cp -p /var/local/ConfigFiles/etc/inetd.conf/csmflyer.Normal /etc/inetd.conf:new
   #
   # Found 10 hosts: 9 skipped, 0 errors, 1 done, 0 N/A.
   #
   # Skipped:
   #       f2n1 iceflyer f1n1 f1n3 f3n1 f4n1 f4n2 f4n3 f4n4
   # Done:
   #       csmflyer
   

7. Push ConfigFile to multiple nodes. (intro, usage)

   csmflyer: push -m f2n1,f1n1 config -eq bin/passwd
   
   push_rcp: 1 file(s)
     f2n1:/usr/bin/rcp -p /var/local/ConfigFiles/usr/bin/passwd/passwd.template f2n1:/usr/bin/passwd:new
     f1n1:/usr/bin/rcp -p /var/local/ConfigFiles/usr/bin/passwd/passwd.template f1n1:/usr/bin/passwd:new
   

8. Safe copy a file to all IO nodes. (intro, usage)

   csmberg: cd /tmp/kcarlson; uals mmq*
   - 0640  kcarlson staff    7540 070106.0655 mmq
   csmberg: push -u io file -q mmq
   - 0640  kcarlson staff    7540 070106.0655 mmq
   
   push_rcp: 1 file(s)
     b1n3:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n3:/tmp/kcarlson/mmq:new
     b1n4:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n4:/tmp/kcarlson/mmq:new
     b1n5:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n5:/tmp/kcarlson/mmq:new
     b1n6:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n6:/tmp/kcarlson/mmq:new
   

9. Copy a file to all IO nodes. (intro, usage)

   csmberg: push -u io rcp  -q mmq
   - 0640  kcarlson staff    7540 070106.0655 mmq
   
   push_rcp: 1 file(s)
     b1n3:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n3:/tmp/kcarlson/mmq
     b1n4:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n4:/tmp/kcarlson/mmq
     b1n5:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n5:/tmp/kcarlson/mmq
     b1n6:/usr/bin/rcp -p /tmp/kcarlson/mmq b1n6:/tmp/kcarlson/mmq
   

10. Reverse copy files from nodes to server. (intro, usage)

    csmberg: push -u io -m !b1n6 rcp -rq mmq mmq
    
    push_rcp: from:/tmp/kcarlson/mmq  to:/tmp/kcarlson/mmq
      b1n3:/usr/bin/rcp -p b1n3:/tmp/kcarlson/mmq /tmp/kcarlson/mmq.b1n3
      b1n4:/usr/bin/rcp -p b1n4:/tmp/kcarlson/mmq /tmp/kcarlson/mmq.b1n4
      b1n5:/usr/bin/rcp -p b1n5:/tmp/kcarlson/mmq /tmp/kcarlson/mmq.b1n5
    csmberg: uals mmq*
    - 0640  kcarlson staff    7540 070106.0655 mmq
    - 0640  kcarlson staff    7540 070106.0655 mmq.b1n3
    - 0640  kcarlson staff    7540 070106.0655 mmq.b1n4
    - 0640  kcarlson staff    7540 070106.0655 mmq.b1n5
    Total:        4 files     41k size,  for: .
    

11. Check keytabs on IBM admin nodes (use ssh-adm root). (intro, usage)

    mcgee: push -s -ibm -u admin --eq cmd /usr/local/krb5/bin/klist -ekt /etc/krb5.keytab
      flyman:Keytab name: FILE:/etc/krb5.keytab
      flyman:KVNO Timestamp         Principal
      flyman:---- ----------------- --------------------------------------------------------
      flyman:14 01/02/07 13:55:07 host/flyman@ARSC.EDU (Triple DES cbc mode HMAC/sha1)
      flyman:14 01/02/07 13:55:07 host/flyman@ARSC.EDU (AES-256 CTS mode 96-bit SHA-1 HMAC)
     bergman:Keytab name: FILE:/etc/krb5.keytab
     bergman:KVNO Timestamp         Principal
     bergman:---- ----------------- --------------------------------------------------------
     bergman:22 01/02/07 13:55:14 host/bergman@ARSC.EDU (Triple DES cbc mode HMAC/sha1)
     bergman:22 01/02/07 13:55:14 host/bergman@ARSC.EDU (AES-256 CTS mode 96-bit SHA-1 HMAC)