Administrator's Guide

A DRM User Scenario

This section provides a real life example of using DRM to prepare for disaster recovery, recover the server, and recover the clients. The detailed steps are followed by an outline that gives you a brief day-by-day picture of the DRM tasks.

Disaster Recovery Preparation

The ADSM server named ADSMSERV contains the backups for the FileRight Company accounts receivable application. FileRight uses ADSM server database mirroring and copy storage pools whose volumes are kept onsite. For disaster recovery preparation, FileRight uses ADSM server database backup and copy storage pool volumes which are immediately moved offsite after creation. FileRight also uses the AIX mksysb function to create a bootable image of the base operating system and ADSM code.

1. The administrator records the following recovery information in the RECOVERY.INSTRUCTIONS stanza source files:
   • Software license numbers
   • Sources of replacement hardware
   • The FileRight Company's specific recovery steps

2. The administrator schedules an automatic nightly backup to occur in the following order:

   1. Primary Storage Pools

   2. Database

3. At 8 a.m. each morning, the administrator creates a list of the previous night's database and storage pool backup volumes to be sent offsite. The administrator issues the following command:

   query drmedia * wherestate=mountable
   

4. The administrator checks the volumes out of the library by issuing the following command:

   move drmedia * wherestate=mountable
   

5. The administrator sends the volumes offsite and records that the volumes were given to the courier by issuing the following command:

   move drmedia * wherestate=notmountable
   

6. The administrator updates the volume information in the recovery plan by issuing the following command:

   prepare
   

   The administrator copies the recovery plan file to a diskette to be given to the courier.

7. Copy storage pool volume CSP01 had been previously reclaimed and its volume status changed to PENDING. The volume is at the offsite vault. Last night the PENDING window passed for CSP01 and its status changed to EMPTY. The volume no longer contains valid backup data and should be brought back onsite for reuse or disposal.

   The administrator creates a list of tapes that contain data that is no longer valid and that should be returned to the site, by issuing the following command:

   query drmedia * wherestate=vaultretrieve
   

   Volume CSP01 is in the list.

8. The administrator gives the courier the following: database and storage pool backup tapes, the recovery plan file diskette, and the list of volumes to be returned from the vault.

9. The courier gives to the administrator any tapes that were on the previous day's return from the vault list. To update the state of these tapes and to check them into the library, the administrator issues the following command:

   move drmedia * wherestate=courierretrieve cmdf=/drm/checkin.libvol
     cmd="checkin libvol libauto &vol  status=scratch"
   

   The volume records for the tapes that were in the COURIERRETRIEVE state are deleted from the database. The MOVE DRMEDIA command also generates the CHECKIN LIBVOL command for each tape processed in the file /drm/checkin.libvol:

   checkin libvol libauto tape01 status=scratch
   checkin libvol libauto tape02 status=scratch
       .
       .
       .
   

   Note:

   An administrator can run the ADSM MACRO command by specifying /drm/checkin.libvol > dsmadmc -id=xxxxx -pa=yyyyyy MACRO /drm/checkin.libvol

10. The courier brings the database and storage pool backup tapes, the recovery plan diskette, and the list of volumes to return from the vault.

11. The ADSM administrator calls the vault and verifies that the backup tapes arrived and are secure and the tapes to be returned to the site have been given to the courier.

12. To set the location of these volumes sent to the VAULT, the administrator issues the following command:

    move drmedia * wherestate=courier
    

13. To set the location of these volumes given by the vault to the courier, the administrator issues the following command:

    move drmedia * wherestate=vaultretrieve
    

Recovering the Server Scenario

The following scenario shows how an administrator uses the recovery plan file to recover the ADSM server. The ADSM server in this example does not use virtual volumes. A disaster has destroyed the ADSM server and a complete recovery of the server is required. The administrator has the latest recovery plan file.

1. The administrator reviews the recovery steps described in the RECOVERY.INSTRUCTIONS.GENERAL stanza of the plan.

2. The administrator requests the server backup tapes from the offsite vault.

3. The administrator views the RECOVERY.INSTRUCTIONS.OFFSITE stanza for the name and telephone number of the courier the company uses to move tapes between the data center and the offsite vault.

4. The administrator uses a locally written procedure to break out the recovery plan file stanzas into multiple files. For additional information, see page Breaking Out a Disaster Recovery Plan File.) These files can be optionally viewed, updated, printed, or executed as ADSM macros or scripts.

5. The administrator prints out the RECOVERY.VOLUMES.REQUIRED file. The printout is handed to the courier who goes to the offsite vault to obtain the backup volumes.

6. In the meantime, the administrator must find a suitable replacement machine. Stanza RECOVERY.DEVICES.REQUIRED specifies the required tape drive type that will be needed to read the backup tapes. Stanza SERVER.REQUIREMENTS summarizes the required amount of disk space.

7. The administrator restores an AIX image to the replacement machine using a mksysb tape. This tape is created whenever software updates or configuration changes are made to the AIX system. It includes the ADSM server software. This tape and its location were specified in the RECOVERY.INSTRUCTIONS.INSTALL stanza.

   Restoring from the mksysb tapes include:

   • Recreating the root volume group.
   • Recreating the file system that the database, recovery log, storage pool and disk volumes are located.

8. The administrator ensures the environment is the same as when the disaster recovery plan file was created. The environment includes:

   • The directory structure of the ADSM server executable and disk formatting utility

   • The directory structure for ADSM server configuration files (disk log, volume history file, device configuration file, and server options file)

   • The directory structure for database, log, and storage pool volume

   • The directory structure as well as the individual files created when the disaster recovery plan file was split into multiple files

9. The administrator reviews the ADSM macros contained in the recovery plan. At the time of the disaster, the courier had not picked up the database and storage pool incremental backup volumes created the previous night. However, they were not destroyed by the water. The administrator removes the entry for the storage pool backup volume from the COPYSTGPOOL.VOLUMES.DESTROYED file.

10. The courier returns with the required volumes. Somehow, the vault could not find one of the copy storage pool volumes. There is not enough time to wait for the vault location to find the lost volume. The administrator removes the entry for that volume from the COPYSTGPOOL.VOLUMES.AVAILABLE file.

11. All of the server's primary volumes were destroyed. The administrator decides there are no changes required to the PRIMARY.VOLUMES script and ADSM macro files.

12. To restore the server database to a point where clients can be recovered, the administrator invokes the RECOVERY.SCRIPT.DISASTER.RECOVERY.MODE script file by entering the script file name at the command prompt.

    Note:

    Alternatively the administrator could have used the steps in the recovery script as a guide, and manually executed each step.

    The following are the steps executed in this recovery script:

    1. Copy the ADSM server options file from the DSMSERV.OPT file to its original location.

    2. Copy the volume history file required by ADSM DSMSERV RESTORE DB processing from the VOLUME.HISTORY.FILE file to its original location.

       Note:

       Use this copy of the volume history file unless you have a more recent copy (after the disaster occurred).

    3. Copy the device configuration file required by ADSM DSMSERV RESTORE DB processing from the DEVICE.CONFIGURATION.FILE file to its original location.

    4. Create the ADSM server recovery log and database volumes using DSMFMT.

    5. Issue DSMSERV FORMAT for the recovery log and database files.

    6. Issue the DSMSERV RESTORE DB command.

    7. Start the server.

    8. Register ADSM server licenses.

    9. Mark copy storage pool volumes retrieved from vault as available.

    10. Mark copy storage pool volumes which cannot be obtained as unavailable.

    11. Mark primary storage pool volumes as destroyed.

13. The administrator invokes the RECOVERY.SCRIPT.NORMAL.MODE script file to restore the server primary storage pools.

    Note:

    This action is optional at this time because ADSM can access the copy storage pool volumes directly to restore client data. Using this feature, the administrator can minimize client recovery time because server primary storage pools do not have to be restored first. However, in this scenario, the client machines were not damaged, so the focus of the administrator is to restore full ADSM server operation.

    If client machines are damaged, you may want to delay this action until after all clients are recovered.

    Alternatively, the administrator could have used the steps in the recovery script as a guide, and manually executed each step.

    The steps executed in this recovery script are as follows:

    1. Create replacement primary volumes.

    2. Define the replacement primary volumes to ADSM.

    3. Restore the primary storage pools.

14. The administrator collects the database backup and copy storage pool volumes used in the recovery so that they can be returned to the vault. For these backup volumes to be returned to the vault using the routine MOVE DRMEDIA process, the administrator executes the following ADSM administrative commands:

    update volhist TPBK50 devcl=lib8mm ormstate=mountable
    update volhist TPBK51 devcl=lib8mm ormstate=mountable
    

    The copy storage pool volumes used in the recovery already have the correct ORMSTATE.

15. The administrator then runs the BACKUP DB command to back up the newly restored database.

16. The administrator issues the MOVE DRMEDIA * WHERESTATE=MOUNTABLE command to check the volumes out of the library.

17. To create a list of the volumes to be given to the courier, the administrator issues the QUERY DRMEDIA * WHERESTATE=NOTMOUNTABLE.

18. After the administrator packages the volumes and gives them to the courier, the MOVE DRMEDIA * WHERESTATE=NOTMOUNTABLE command is issued.

19. The administrator issues PREPARE.

Recovering ADSM Clients Scenario

The following scenario demonstrates the recovery of ADSM clients. A week after the ADSM server was recovered, another water pipe burst in the building that houses distributed systems applications. Many machines that were backed up using ADSM clients are destroyed. A disaster is declared.

1. To view a list of client machines that were lost in building 21 and their restore priority, the administrator issues the following command:

   query machine building=021 format=detailed
   

   ADSM displays information similar to the following:

   +--------------------------------------------------------------------------------+
   |          Machine Name: POLARIS                                                 |
   |      Machine Priority: 1                                                       |
   |              Building: 21                                                      |
   |                 Floor: 2                                                       |
   |                  Room: 1                                                       |
   |          ADSM Server?: No                                                      |
   |           Description: Payroll                                                 |
   |             Node Name: POLARIS                                                 |
   |   Recovery Media Name: MKSYSB1                                                 |
   |      Characteristics?: Yes                                                     |
   |Recovery Instructions?: Yes                                                     |
   +--------------------------------------------------------------------------------+

   
   

2. For each machine, the administrator issues the following commands:

   1. To determine the location of the boot media, the administrator issues the QUERY RECOVERYMEDIA command. For example:

      query recoverymedia mksysb1
      

      ADSM displays the following information:

      +--------------------------------------------------------------------------------+
      |Recovery Media Name  Volume Names    Location    Machine Name                   |
      |-------------------- -----------     ----------  ----------------               |
      |MKSYSB1              vol1 vol2       IRONVAULT    POLARIS                       |
      |                      vol3                                                      |
      +--------------------------------------------------------------------------------+

      
      

   2. To determine the machine specific recovery instructions for the POLARIS machine, the administrator issues:

      query machine polaris format=recoveryinstructions
      

      ADSM displays the following:

      +--------------------------------------------------------------------------------+
      |Recovery Instructions for Polaris.                                              |
      |Primary Contact:                                                                |
      |   Jane Smith (wk 520-000-0000 hm 520-001-0001)                                 |
      |Secondary Contact:                                                              |
      |   John Adams (wk 520-000-0001 hm 520-002-0002)                                 |
      +--------------------------------------------------------------------------------+

      
      

   3. To determine the machine hardware requirements for POLARIS, the administrator issues:

      query machine polaris format=characteristics
      

      ADSM displays information similar to the following:

      +--------------------------------------------------------------------------------+
      |devices                                                                         |
      |aio0         Defined                Asynchronous I/O                            |
      |bus0         Available 00-00        Microchannel Bus                            |
      |fd0          Available 00-00-0D-00  Diskette Drive                              |
      |fda0         Available 00-00-0D     Standard I/O Diskette Adapter               |
      |fpa0         Available 00-00        Floating Point Processor                    |
      |gda0         Available 00-04        Color Graphics Display Adapter              |
      |hd1          Defined                Logical volume                              |
      |hd2          Defined                Logical volume                              |
      |hd3          Defined                Logical volume                              |
      |hdisk0       Available 00-01-00-00  400 MB SCSI Disk Drive                      |
      |hdisk1       Available 00-01-00-40  Other SCSI Disk Drive                       |
      |hft0         Available              High Function Terminal Subsystem            |
      |inet0        Available              Internet Network Extension                  |
      |ioplanar0    Available 00-00        I/O Planar                                  |
      |kbd0         Defined   00-00-0K-00  United States keyboard                      |
      |lb0          Available 00-02-00-20  ADSM Library                                |
      |lo0          Available              Loopback Network Interface                  |
      |loglv00      Defined                Logical volume                              |
      |lp0          Available 00-00-0P-00  IBM 4201 Model 3 Proprinter III             |
      |lv03         Defined                Logical volume                              |
      |lv04         Defined                Logical volume                              |
      |lvdd         Available              N/A                                         |
      |mem0         Available 00-0B        8 MB Memory Card                            |
      |mem1         Available 00-0C        16 MB Memory Card                           |
      |mous0        Defined   00-00-0M-00  3 button mouse                              |
      |mt0          Available 00-02-00-40  ADSM Tape Drive                             |
      |ppa0         Available 00-00-0P     Standard I/O Parallel Port Adapter          |
      |pty0         Available              Asynchronous Pseudo-Terminal                |
      |rootvg       Defined                Volume group                                |
      |sa0          Available 00-00-S1     Standard I/O Serial Port 1                  |
      |sa1          Available 00-00-S2     Standard I/O Serial Port 2                  |
      |scsi0        Available 00-01        SCSI I/O Controller                         |
      |scsi1        Available 00-02        SCSI I/O Controller                         |
      |sio0         Available 00-00        Standard I/O Planar                         |
      |siokb0       Available 00-00-0K     Keyboard Adapter                            |
      |sioms0       Available 00-00-0M     Mouse Adapter                               |
      |siotb0       Available 00-00-0T     Tablet Adapter                              |
      |sys0         Available 00-00        System Object                               |
      |sysplanar0   Available 00-00        CPU Planar                                  |
      |sysunit0     Available 00-00        System Unit                                 |
      |tok0         Available 00-03        Token-Ring High-Performance Adapter         |
      |tr0          Available              Token Ring Network Interface                |
      |tty0         Available 00-00-S1-00  Asynchronous Terminal                       |
      |tty1         Available 00-00-S2-00  Asynchronous Terminal                       |
      |usrvice      Defined                Logical volume                              |
      |veggie2      Defined                Volume group                                |
      |logical volumes by volume group                                                 |
      |veggie2:                                                                        |
      |LV NAME             TYPE       LPs   PPs  PVs  LV STATE      MOUNT POINT        |
      |hd2                 jfs        103   103  1    open/syncd    /usr               |
      |hd1                 jfs        1     1    1    open/syncd    /home              |
      |hd3                 jfs        3     3    1    open/syncd    /tmp               |
      |hd9var              jfs        1     1    1    open/syncd    /var               |
      |file systems                                                                    |
      |Filesystem    Total KB    free %used   iused %iused Mounted on                  |
      |/dev/hd4          8192     420   94%     909    44% /                           |
      |/dev/hd9var       4096    2972   27%      87     8% /var                        |
      |/dev/hd2        421888   10964   97%   17435    16% /usr                        |
      |/dev/hd3         12288   11588    5%      49     1% /tmp                        |
      |/dev/hd1          4096    3896    4%      26     2% /home                       |
      +--------------------------------------------------------------------------------+

      
      

   4. With the necessary recovery information now available, the administrator successfully restores each client machine.

Summary of Disaster Recovery Manager Use

This section presents a sample overview of DRM use in normal routine processing and during disaster recovery.

Setup

1. License DRM
2. Ensure the device configuration and volume history information files exist
3. Back up the database (full) and storage pools
4. Define your site-specific server recovery instructions
5. Describe priority ADSM client machines
6. Generate the disaster recovery plan

Daily Operations

Day 1

1. Back up client files
2. Back up the primary storage pools
3. Perform an incremental database backup
4. Send the backup volumes to the vault
5. Generate the disaster recovery plan

Day 2

1. Back up client files
2. Back up the primary storage pools
3. Perform an incremental database backup
4. Send the new backup volumes to the vault
5. Generate the disaster recovery plan

Day 3

1. Automatic storage pool reclamation processing occurs
2. Back up client files
3. Back up the primary storage pools
4. Perform an incremental database backup
5. Send the new backup volumes a list of expired volumes and volumes to be recalimed to the vault
6. The vault acknowledges receipt of the volumes sent on the previous day
7. Generate the disaster recovery plan

Disaster and Recovery

Day 4

1. The server and the client machines are destroyed
2. Restore the server using the latest recovery plan
3. Identify the top priority client node(s) at the disaster site
4. Restore client machine files from the copy storage pools
5. Restore the primary storage pools
6. Move database backup and copy storage pool volumes to the vault

Daily Operations

Day 5

1. Back up client files
2. Back up the database (full) and storage pools
3. Send the backup volumes to the vault
4. Generate the disaster recovery plan

Disaster Recovery Manager Checklist

The following checklist can help you plan the tasks required for DRM implementation.

Table 21. DRM Checklist

Activity	Start Date	End Date	Status	Person Resp.	Backup Person
DRM Planning
Evaluate your disaster recovery requirements What are the business priorities for recovering your ADSM clients? Where is the recovery site? Is the recovery site hot, warm, or cold? Do the clients have connectivity to recovery server? Who are the system and ADSM administrators? Will you need to return to the original site?
Where are the offsite backups stored? How does the vault handle the backup media? How are the backups packaged or processed? Who provides the courier service?
Evaluate the current storage pool backup implementation What primary storage pools are being backed up? When are the backups performed? Backup purpose: offsite or onsite Backup media Naming conventions for replacement volumes for primary storage pools
Evaluate the current database backup implementation When are the backups performed? Backup purpose: offsite or onsite Backup media How many backup series do you want maintained and for how long? Review the copy storage pool REUSEDELAY value and verify that it is the same as the SET DRMDBBACKUPEXPIREDAYS value.
Determine which primary storage pools are to be managed by DRM
Determine which copy storage pools are to be managed by DRM Offsite copy storage pools
Plan for the Recovery Plan File (RPF) What is the RPF pathname prefix? How many RPFs should be kept? How will RPFs be made available at the recovery site? Print and store offsite Tape/diskette copy stored offsite Copy sent/NFS to recovery site
Determine where you want to create the user-specified recovery instructions What is the instructions pathname prefix?
Analyze the sequence of steps related to the PREPARE command backup movement Document the flow of activities and timings Sending of volumes offsite Return on empty volumes PREPARE timing
DRM Installation
Receive the ADSM code
Install the ADSM code
Get licensed for DRM REGISTER LICENSE or Update the server options
Customize the DRM options The administrator with system authority issues: SET DRMDBBACKUPEXPIREDAYS to define the Database backup expiration SET DRMPRIMSTGPOOL to specify the DRM-managed storage pools SET DRMCOPYSTGPOOL to specify the DRM-managed copy storage pools SET DRMPLANVPOSTFIX to specify 1 character to be appended to new storage pools SET DRMPLANPREFIX to specify the RPF prefix SET DRMINSTRPREFIX to specify the user instruction file prefix SET DRMNOTMOUNTABLENAME to specify the default location for media to be sent offsite SET DRMCOURIERNAME to specify the default courier SET DRMVAULTNAME to specify the default vault SET DRMCMDFILENAME to specify the default file name to contain the commands specified with the CMD parameter on MOVE and QUERY DRMEDIA SET DRMCHECKLABEL to specify whether volume labels are verified when checked out by the MOVE DRMEDIA command
Define the site-specific recovery instructions Identify: Target disaster recovery server location Target server software requirements (OS or ADSM) Target server hardware requirements (storage devices) ADSM administrator contact Courier name and telephone number Vault location and contact person Create: Enter the site-specific recovery instructions data into files created in the same path/HLQ as specified by SET DRMINSTRPREFIX
DRM Test
Test the DRM installation and customization Q DRMSTATUS to display the DRM setup Back up the primary storage pools Back up the ADSM database Q DRMEDIA to list the backup volumes MOVE DRMEDIA to move offsite PREPARE to create the RPF
Examine the RPF created
Test the RPF break out awk script planexpl.awk Locally written procedure
DRM Production
Set up the schedules for automated functions
Implement the DRM procedures