24 Orchestrating Recovery Scenarios

24.1 Overview

The Recovery Scenarios feature allows N2W Software users to design an object that will automatically coordinate a sequence of recoveries for several or all backup targets of a single policy during one recovery session.

  • A Recovery Scenario object is created with the saved configurations of successful backups for the particular policy.

  • The user will save the recovery configuration for each selected backup target and add it to the Recovery Scenario object.

  • At runtime, the user selects a successful backup record to use in the recovery.

Following are the options for executing a Recovery Scenario:

  • Test the success of the Recovery Scenario configuration using the Dry Run command.

  • Execute an ad hoc run of the Recovery Scenario using the Run Scenario command.

  • Execute the Recovery Scenario on a schedule. The last successful backup is automatically selected as input. Assign or create a schedule in the Recovery Scenario Details tab.

Backups in the Freezer are not recoverable as part of a Recovery Scenario.

24.2 Conditions

  • During the Recovery Process:

    • All Recovery Scenario targets share the same destination account and destination region, which are set as part of the Recovery Scenario parameters.

    • Recovery Scenarios can have pre- and post- scripts which will run, respectively, prior to recovery execution and subsequent to recovery completion.

      • In case of a pre-script failure, the Recovery Scenario will not execute.

      • In case of a Recovery Scenario failure or pre-script failure, the post-script will not run.

  • Every Recovery Scenario target has a sequential Recovery Order value within the Recovery Scenario which determines the order in which each target is recovered.

    • Execution of a target recovery within the recovery scenario is sequenced using the targets Recovery Order value. The target with the lowest Recovery Order value runs first.

    • All recovery targets sharing the same Recovery Order value will run in an arbitrary sequence.

    • If the recovery of a target fails, the targets next in sequential order will not run, unless Recovery Scenario’s Continue recovering ignoring failures parameter is enabled.

    • Testing: You can verify the Recovery Scenario input parameters, such as key pair, security groups, and VPC, by selecting the Dry Run link. You will be prompted to select a successful backup for the Dry Run just as with an actual Run Scenario.

24.3 Creating a Recovery Scenario

Be sure to execute a successful Dry Run of the Recovery Scenario before assigning a schedule.

To add the details for a recovery scenario:

  1. Select the Recovery Scenarios tab, and then select New.

  2. In the Recovery Scenario Details tab, complete the fields as follows:

    • Name - Enter a unique name.

    • User, Account, Policy - Select from the lists or select New. After the addition, select Refresh. Select the policy for which the Recovery Scenario is defined.

    • Recovery Destination Account and Recovery Destination Region - Select from the lists.

    • Schedule - Optionally select a schedule from the list, or select New to create a new schedule, for running the Recovery Scenario.

    • Recipients - Enter the email addresses of users to receive notification of Recovery Scenario Run Scenario / Dry Run status. If SES is disabled, run status emails are not sent to recipients.

    • Enable Agent Scripts – Select if the Recovery Scenario will be run by a custom script. The default is not to run user scripts. See section 24.7.

      • Select Agent Script Timeout in seconds from the list. When the timeout is reached, N2WS will skip the script and continue with the recovery scenario.

      • Collect Script Output – Whether to collect script output in a log. Default is to collect.

    • Continue recovering ignoring failures – Whether to continue the sequence of recoveries in the scenario if there is a failure. The default is to not continue the script on the failure of a recovery.

  3. Select Save.

To add the recovery targets:

  1. Select the Recovery Targets tab.

  2. In the Add Recovery Targets menu, select a resource type from the target policy to add to the scenario. Reminder: S3 Bucket Sync is not an option since it is not a backup action.

  3. In the Add resource type screen, select one or more Recovery Targets for the resource type, and then select Add selected.

Every Recovery Scenario target has a number identifying the sequential Recovery Order of execution within the Recovery Scenario. The execution of the Recovery Source within the Recovery Scenario is sequenced using the target’s Recovery Order value. The recovery of the target with the lowest Recovery Order value runs first.

4. To change the Recovery Source or Recovery Order for a target, select a value from its list.

For Instance, Volume, and EFS Recovery Targets, it is important to configure the recovery details for each target. Select a recovery target and then select Configure. See section 24.3.1 for configuring an Instance Recovery Target.

When all details are complete, select Save in the Create Recovery Scenario screen.

24.3.1 Configuring an Instance Recovery Target

The Configuration screen opens with additional tabs:

  • Basic Options

  • A tab for the resource type, such as Volumes

  • Advanced Options

The configuration Auto assigned values may be different than the values that are shown as grayed-out. To be sure about a value, you need to assign it.

For each data item in the configuration tabs, assign the appropriate value. In each tab, you can customize a setting by turning off its Auto assigned toggle. Depending on the data item, you can:

  • Select a different value from the Custom drop-down list.

  • Enable or disable a feature.

  • Enter a new value.

When finished with each tab, select Close.

In the Basic Options tab, you can configure basic recovery actions, such as whether to launch from a snapshot or image, which key pair to use, and network placement.

Since not all instance types are available in all AWS regions, recovery of an instance type to a region where the type is unsupported may fail. Where the instance type is not supported yet in an AWS region, we recommend configuring a supported Instance Type in the Basic Options parameters. See section 10.3.1.

In the Volumes tab, you can configure device information, such as capacity, type, and whether to preserve tags and delete on termination. To expand the configuration section for a volume, select the right arrow.

In the Advanced Options tab for an instance, you can customize recovery target features, such as architecture, shutdown behaviour, whether to enable ENA and user data.

For complete details about performing an instance recovery, see section 10.3.

24.4 Testing a Recovery Scenario

The Dry Run option allows you to determine whether the input parameters, such as key pair, security groups, and VPC, are correct for the recovery.

To test a Recovery Scenario:

  1. In the Recovery Scenarios tab, select a Recovery Scenario and then select Dry Run.

  2. In the list of successful backups, select one backup to perform the test with, and then select Dry Run.

  3. Open the Recovery Scenario Monitor.

  4. In the Status column for the Recovery Scenario, you will see a success message for the test:

  5. Selecting Recoveries brings you to the regular Recovery Monitor.

24.5 Managing Recovery Scenarios and Targets

To manage a Recovery Scenario object:

  1. In the Recovery Scenarios tab, select a scenario.

  2. Select Edit, Delete, Run Scenario, or Dry Run, as needed.

To manage targets in the scenario:

  1. In the Recovery Scenarios tab, select a scenario, and then select Edit.

  2. To delete a target, select the Recovery Targets tab, select a target, and then select Remove from List.

  3. Depending on the resource type, the action Configure is available. Configure opens tabs for Basic Options, resource type details, and Advanced Options.

24.6 Running and Monitoring a Recovery Scenario

A Recovery Scenario can also be run on a schedule using the last successful backup. To assign or create a schedule, see section 24.3.

  1. ​In the Recovery Scenarios tab, select a Recovery Scenario and then select Run Scenario. A list of backups, successful and unsuccessful, opens.

  2. Select one successful backup to recover from and then select Recover. The started message opens in the top right corner:

  3. To open the Recovery Scenario Monitor, select the link, or select the Recovery Scenario Monitor tab.

    1. A Status of ‘Recovery succeeded’ with a test tube symbol next to it indicates that the recovery was a Dry Run.

To view a live recovery, in the Recovery Scenario Monitor, select a scenario, and then select Recoveries. The Recovery Monitor opens.

Deleting a run record will trigger the deletion of all its target recovery records.

To delete a run record, select a scenario, and then select Delete Record.

To view details of the recovery in the Run Log, select a Recovery Scenario and then select Log.

24.7 Recovery Scenario User Scripts

When Enable Agent Scripts is set in the Recovery Scenario Details tab, N2WS will run two scripts, one before and one after the recovery run:

  • before_<recovery-scenario-name>

  • after_<recovery-scenario-name>

A file extension is optional and, if added, may be for any interpreter.

This is somewhat similar to the Linux Backup Scripts feature described in the Before Script and After Script topics, sections 6.3.1 and 6.3.2.

These scripts must be located on the N2WS server in the following folders:

  • For root user: /cpmdata/scripts/scenario

  • For non-root user: /cpmdata/scripts/scenario/user_names

24.7.1 Before Script

The before script passes the following parameters, in the following order:

#

Parameter

Notes

1

Scenario Id

2

Account Id

May be null, if the value is NULL.

3

Policy account Id

4

Destination region

May be null, if the value is NULL.

24.7.2 After Script

The after script passes the same parameters as the before with the addition of parameters for the scenario’s recovery targets:

#

Param.

Notes

1-4

...

Same as before_ parameters

5

Target lists

Each target is in colon-separated format:

RecoveryType:OriginalAwsResourceId:

OriginalRegion:RecoveredAwsResourceId

RecoveryType

Single character identifying resource type:

I - Instance

V - Volume

R - RDS Database

A - RDS (Aurora) Cluster

C - Redshift Cluster

D - DynamoDB

E - EFS

OriginalAwsResourceId

AWS ID of the original resource

OriginalRegion

AWS region of the original resource

RecoveredAwsResourceID

AWS ID of the recovered resource. If not recovered, then 'null'.

Following is an example of an after_ script for a Recovery Scenario that was defined with 2 targets: an EC2-instance and an EC2-volume. The after_ script passes 6 parameters, 2 of which are for the targets. In the following example, the instance recovery target was not recovered:

1
null
1
null
I:i-0a87ab83ca3fa62c2:us-east-1:null
V:vol-0197aba1f7090c513:us-east-1:vol-03336f4ed151b5d29