Internal Production: Last Updated 2009-04-03

ARSC ConfigFiles and Push Processes

Purpose (top - introduction - files - procedure - example - tools)

This document is intended to supplement the General Configuration and Change Management Plan for ARSC systems by providing details on how configuration files and local modifications are maintained across all ARSC platforms.

Document Index / Related Documentation

Introduction (top - introduction - files - procedure - example - tools)

The general principle is files are not arbitrarily modified and are monitored for correctness including type, content, ownership, and mode (access permission). In addition, change history and backout copies are retained for all configuration changes and for recovery and security these are maintained on a administrative host not exposed to general users. These principles apply to all aspects of systems management, but software installed and managed by vendor tools (rpm, pkginst, lslpp, ...), or under /usr/local using 'Installing Third Party Software' are managed by those procedures. Likewise account creation is also handled by separate procedures even though the related files are monitored under ConfigFiles.

The ConfigFiles are typically under /etc or /usr, but may fall under other directories. Typically /usr/local files do not fall under ConfigFiles, but there are a few exceptions for critical files. Any file in system space which must be added, changed, or replaced (e.g., a binary) outside of a vendor provided tool is a candidate for ConfigFiles to ensure it is not regressed or arbitrarily lost or changed. Any files which have significant security ramifications, whether locally modified or not, are also monitored by ConfigFiles. The suid|sgid files which must be monitored for security are handled via the permchk process, a few exceptions are in ConfigFiles for local modification or version tracking.

ARSC uses similar procedures across all platforms:

Platform
Type		 host(s) admin
from		 /var/local/
directories		 List and machines
 Uses
push?		 Other
platform
tools		 Email
Linuxpuppychow, ag*, cms,
workstations (48+),
...		 dispatch LinuxFiles LinuxFiles/List.txt
LinuxFiles/Long.txt
etc/machines.linux		 yes   linuxman
Sun
Solaris		40+ systems dispatch SunFiles SunFiles/List.txt
SunFiles/Long.txt
etc/machines.sun		 yes mush sunman
Sun
Linux		midnight
(420+ nodes)		 mn1sm
 LinuxFiles LinuxFiles/List.txt
LinuxFiles/Long.txt
etc/machines.linux		 yes pdsh midman
Cray
CLE		pingo (25)
ognip (7)		 piboot
ogboot		 CrayFiles CrayFiles/List.txt
CrayFiles/Long.txt
etc/machines.cray		 yes xtopview
xtspec
pdsh		 crayman
All ConfigFiles changes are evaluated for impact. Major changes are installed on test systems and may be implemented during scheduled downtime. Major changes and significant changes are recorded via RT entries. Changes are typically made by System Administrator staff on the associated Support and Usability (S&U) Project, but with project overlaps or in oncall situations may be made by other ARSC IT II staff.

Management Files (top - introduction - files - procedure - example - tools)

Procedure (top - introduction - files - procedure - example - tools)

Any time ConfigFiles are updated the following will generally be followed:

  1. Pre-planning as appropriate:
    1. Review within S&U project or related projects
    2. Open RT entry
    3. Define change test and validation procedures
    4. Define backout plan
    5. Establish implementation schedule
    Minor changes under common procedures may be done with minimal planning and no associated RT entry.

  2. admin: cd /var/local/*Files/directory
    1. Check if any file specified README.ARSC exists and follow those instructions.
    2. Ensure the current version has been copied into ./backout/file.yyyymmdd.hhmm
      This provides a central tracking and a recovery method.
      The ckbko and mkbko commands can be used for verifying or generating backout copies.

  3. admin: /usr/local/adm/bin/edconfig filename.ext
    1. Make necessary changes.
    2. If permitted by the file, update the file history at the top or bottom with the date (YYYY-mm-dd), your initials, and a brief summary of what was changed and why, reference RT number if appropriate.
    Note, the edconfig command will automatically make the backout/*.yyyymmdd.hhmm copy.

  4. Copy file to production, using one of:

    1. If the file is not to be immediately pushed, create a NOPUSH file to prevent others from accidentally pushing out the change:
      admin: sudo touch ./NOPUSH
      Be sure to email appropriate folks (*man email alias) to notify them of the work-in-progess and that chk_sanity will likely report this file.

    2. If the platform supports push, push the file out to the target systems: 
      admin: /usr/local/adm/bin/push config filename
or
admin: /usr/local/adm/bin/push one system1[,system2,...] filename
or
admin: /usr/local/adm/bin/push -m system1[,system2,...] config filename

    3. If the file or platform does not support push, copy the file to production location.
      Note, for binaries be sure to first mv (same filesystem) the target to preserve the inode if the binary is likely in active use.

  5. Validate or test the change as appropriate.

  6. Appropriately log and notify that file(s) have been pushed:
    1. For platforms which use logit, make an entry.
      Note, both edconfig and push invoke sudo which logs what was done (but not why).
    2. Update any RT entries.
      Not all ConfigFiles changes require RT entries, significant changes or changes with potential user impact will. Note, the RT email facilities can be used in conjunction with the following notifications.
    3. For significant changes email the appropriate *man alias describing the change.
    4. Copy consult and/or the S&U project alias if the change may have user impact.
    5. If the change includes or is implemented with a system restart, the downtime procedure must be followed to log the outage. The downtime email is a substitute for *man|consult email.

  7. Next businesss day review the sysmon chk_sanity message to ensure the modifications were properly logged and installed.

Example (top - introduction - files - procedure - example - tools)

The following fabricated ConfigFile demonstrates symlink translations, including: 

csmflyer: uals -y Mog; pwd
d 0750 root     ibmman backout
l 0777 kcarlson ibmman test.csmflyer -> Directory  
l 0777 kcarlson ibmman test.f1n1 -> xxx
l 0777 kcarlson ibmman test.f1n3 -> ./xxx
- 0640 kcarlson ibmman test.f2n1
l 0777 kcarlson ibmman test.f4n1 -> nofile
l 0777 kcarlson ibmman test.f4n2 -> ./NOFILE
l 0777 kcarlson ibmman test.f4n3 -> yyy
l 0777 kcarlson ibmman test.f4n4 -> ./yyy
- 0640 kcarlson ibmman xxx
/var/local/ConfigFiles/var/local/test

csmflyer: dsh -av -n csmflyer \
 uals -dMog /var/local/test 2>&1 | sort
csmflyer: d 0700  root     root   /var/local/test
f1n1: - 0640 kcarlson ibmman /var/local/test
f1n3: l 0777 root     root   /var/local/test -> ./xxx
f2n1: - 0640 kcarlson ibmman /var/local/test
f3n1: uals: /var/local/test: No such file or directory
f4n1: uals: /var/local/test: No such file or directory
f4n2: uals: /var/local/test: No such file or directory
f4n3: l 0777 root     root   /var/local/test -> yyy
f4n4: l 0777 root     root   /var/local/test -> ./yyy

csmflyer: push config -q test

push_rcp: 1 file(s)
 csmflyer:#symbolic link: /var/local/ConfigFiles/var/local/test/test.csmflyer
     f2n1:/usr/bin/rcp -p /var/local/ConfigFiles/var/local/test/test.f2n1 f2n1:/var/local/test:new
     f1n1:/usr/bin/rcp -p /var/local/ConfigFiles/var/local/test/xxx f1n1:/var/local/test:new
     f1n3:#symbolic link: /var/local/ConfigFiles/var/local/test/test.f1n3
     f3n1:#no appropriate /var/local/ConfigFiles/var/local/test/test.*
     f4n1:#linked NOFILE: /var/local/ConfigFiles/var/local/test/test.f4n1
     f4n2:#linked NOFILE: /var/local/ConfigFiles/var/local/test/test.f4n2
     f4n3:#symbolic link: /var/local/ConfigFiles/var/local/test/test.f4n3
     f4n4:#symbolic link: /var/local/ConfigFiles/var/local/test/test.f4n4

Associated Tools: (top - introduction - files - procedure - example - tools)

The following tools reside in /usr/local/adm/bin and are used to validate or manage system configuration: