24 Orchestrating Recovery Scenarios

Design a procedure to execute multiple recoveries within one recovery session.

Recovery Scenarios are available for AWS and Azure policies.

All Azure targets (Virtual Machines, Disks, and SQL Servers) are supported.

24.1 Overview

The Recovery Scenarios feature allows N2W Software users to design an object that will automatically coordinate a sequence of recoveries for several or all backup targets of a single policy during one recovery session.

• A Recovery Scenario object is created with the saved configurations of successful backups for the policy.

• The user will save the recovery configuration for each selected backup target and add it to the Recovery Scenario object.

• At runtime, the user selects a successful backup record to use in the recovery.

Following are the options for executing a Recovery Scenario:

• Test the success of the Recovery Scenario configuration using the Dry Run command.

• Execute an ad hoc run of the Recovery Scenario using the Run Scenario command.

• Execute the Recovery Scenario on a schedule. The last successful backup is automatically selected as input. Assign or create a schedule in the Recovery Scenario Details tab.

Backups in the Freezer are not recoverable as part of a Recovery Scenario.

24.2 Conditions

• During the Recovery Process:

  • All Recovery Scenario targets share the same destination account and destination region, which are set as part of the Recovery Scenario parameters.

  • Recovery Scenarios can have pre- and post-scripts which will run, respectively, before recovery execution and after recovery completion.

    • In case of a pre-script failure, the Recovery Scenario will not execute.

    • In case of a Recovery Scenario failure or pre-script failure, the post-script will not run.

• Every Recovery Scenario target has a sequential Recovery Order value within the Recovery Scenario which determines the order in which each target is recovered.

  • Execution of a target recovery within the recovery scenario is sequenced using the target's Recovery Order value. The target with the lowest Recovery Order value runs first.

  • All recovery targets sharing the same Recovery Order value will run in an arbitrary sequence.

  • If the recovery of a target fails, the targets next in sequential order will not run, unless Recovery Scenario’s Continue recovering ignoring failures parameter is enabled.

  • Testing: You can verify the Recovery Scenario input parameters, such as key pair, security groups, and VPC, by selecting the Dry Run link. You will be prompted to select a successful backup for the Dry Run just as with an actual Run Scenario.

24.3 Creating a Recovery Scenario

Be sure to execute a successful Dry Run of the Recovery Scenario before assigning a schedule.

To add the details for a recovery scenario:

1. Select the Recovery Scenarios tab.

2. On the New menu, select AWS Recovery Scenario or Azure Recovery Scenario.
3. In the Recovery Scenario Details tab, complete the fields as follows:

   • Name - Enter a unique name.

   • User, Account, Policy - Select from the lists or selectNew. After the addition, select Refresh. Select the policy for which the Recovery Scenario is defined.
   • Recovery Destination Account and Recovery Destination Region - Select from the lists.

   • AWS Credentials - Select Use account AWS Credentials or Provide Alternate AWS Credentials.

The AWS credentials are per Recovery Scenario and not per target. All targets within the Recovery Scenario will use the selected credentials.

• Schedule - Optionally select a schedule from the list, or select New to create a new schedule, for running the Recovery Scenario.
• Recipients - Enter the email addresses of users to receive notification of Scenario Run Scenario / Dry Run status.

If SES is disabled, emails are not sent to recipients.

• Enable Agent Scripts – Select if the Recovery Scenario will be run by a custom script. The default is not to run user scripts. See section 24.7.

  • Select Agent Script Timeout in seconds from the list. When the timeout is reached, N2WS will skip the script and continue with the recovery scenario.

  • Collect Script Output – Whether to collect script output in a log. Default is to collect.

• Continue recovering ignoring failures – Whether to continue the sequence of recoveries in the scenario if there is a failure. The default is to not continue the script on the failure of recovery.

4. Select Save.

To add the recovery targets:

1. Select the Recovery Targets tab.

2. In the Add Recovery Targets menu, select a resource type from the target policy to add to the scenario. Reminder: S3 Bucket Sync is not an option since it is not a backup action.

3. In each Add resource type screen, select one or more Recovery Targets for the resource type, and then select Add selected.

Every Recovery Scenario target has a number identifying the sequential Recovery Order of execution within the Recovery Scenario. The execution of the Recovery Source within the Recovery Scenario is sequenced using the target’s Recovery Order value. The recovery of the target with the lowest value runs first.

4. To change the Recovery Source or Recovery Order for a target, select a value from its list.

5. For each Instance, Volume, EFS, Azure Virtual Machine, or SQL Server Recovery Target, it is important to configure the recovery details. In the Recovery Targets tab, select the target, and then selectConfigure.

1. For Instance, Volume, and EFS targets, see section 24.3.1. 2. For Azure Virtual Machine targets, see section 24.3.2. 3. For Azure SQL Server targets, see section 24.3.3. 4. For Azure Disk targets, see section 24.3.4.

6. When all details are complete, select Save in the Create Recovery Scenario screen.

24.3.1 Configuring an Instance Recovery Target

The Configuration screen opens with additional tabs:

• Basic Options

• A tab for the resource type, such as Volumes

• Advanced Options

The configuration Auto assigned values may be different than the values that are shown as grayed-out. To be sure about a value, you need to assign it.

For each data item in the configuration tabs, assign the appropriate value. In each tab, you can customize a setting by turning off its Auto assigned toggle. Depending on the data item, you can:

• Select a different value from the Custom drop-down list.

• Enable or disable a feature.

• Enter a new value.

When finished with each tab, select Close.

In the Basic Options tab, you can configure basic recovery actions, such as whether to launch from a snapshot or image, which key pair to use, and network placement.

Since not all instance types are available in all AWS regions, recovery of an instance type to a region where the type is unsupported may fail. Where the instance type is not supported yet in an AWS region, we recommend configuring a supported Instance Type in the Basic Options parameters. See section 10.3.1.

In the Volumes tab, you can configure device information, such as capacity, type, and whether to preserve tags and delete on termination. To expand the configuration section for a volume, select the right arrow.

In the Advanced Options tab for an instance, you can customize recovery target features, such as architecture, shutdown behaviour, whether to enable ENA and user data.

Select Close.

For complete details about performing an instance recovery, see section 10.3.

24.3.2 Configuring an Azure Virtual Machine Recovery Target

See section 24.3 for instructions on how to create a Recovery Scenario and Add Recovery Targets.

After adding the Recovery Targets, select a Virtual Machine target, and then selectConfigure.

When configuring a Virtual Machine target, the Configuration screen opens with tabs for Basic Options and Disks.

The Auto assigned values may be different than the values that are shown as grayed-out in the Configuration screen. To ensure a correct value, assign it.

1. In the Basic Options tab, configure basic recovery actions by selecting a Resource Group, Availability Type, and Networking details. Select Close.

2. In the Disks tab, configure disk information, such as Encryption Set. To expand the configuration section for a disk, select the right arrow . Change Name to the desired name for the recovered disk. Select Close.

24.3.3 Configuring an Azure SQL Server Recovery Target

See section 24.3 for instructions on how to create a Recovery Scenario and Add Recovery Targets.

After adding the Recovery Targets, select an SQL Server target, and then selectConfigure.

When configuring an SQL Server target, the Configuration screen opens with tabs for Basic Options, Network, SQL Server's Databases, and Worker Options. See section 26.9.4.

24.3.4 Configuring an Azure Disk Recovery Target

See section 24.3 for instructions on how to create a Recovery Scenario and Add Recovery Targets.

In the Recovery Targets tab, expand the details for a disk by selecting the right arrow () next to it. Change Name to the desired name for the recovered disk, and update other fields as needed.

24.4 Testing a Recovery Scenario

The Dry Run option allows you to determine whether the input parameters, such as key pair, security groups, and VPC, are correct for the recovery.

To test a Recovery Scenario:

1. In the Recovery Scenarios tab, select a Recovery Scenario and then select Dry Run.
2. In the list of successful backups, select one backup to perform the test with, and then select Dry Run.

3. Open the Recovery Scenario Monitor.

4. In the Status column for the Recovery Scenario, you will see a success message for the test: ​
5. Selecting Recoveries brings you to the regular Recovery Monitor.

24.5 Managing Recovery Scenarios and Targets

To manage a Recovery Scenario object:

1. In the Recovery Scenarios tab, select a scenario.

2. Select Edit, Delete, Run Scenario, or Dry Run, as needed.

To manage targets in the scenario:

1. In the Recovery Scenarios tab, select a scenario, and then select Edit.
2. To delete a target, select the Recovery Targets tab, select a target, and then select Remove from List.
3. Depending on the resource type, the action Configure is available. Configure opens tabs for Basic Options, resource type details, and Advanced Options.

24.6 Running and Monitoring a Recovery Scenario

A Recovery Scenario can also be run on a schedule using the last successful backup. To assign or create a schedule, see section 24.3.

1. ​In the Recovery Scenarios tab, select a Recovery Scenario and then select Run Scenario. A list of backups, successful and unsuccessful, opens.

2. Select one successful backup to recover from and then select Recover. The started message opens in the top right corner:​

3. To open the Recovery Scenario Monitor, select the link, or select the Recovery Scenario Monitor tab.

• A Status of ‘Recovery succeeded’ with a test tube symbol next to it indicates that the recovery was a Dry Run.

4. To view details of the recovery in the Run Log, select a Recovery Scenario and then select Log.

5. To delete a run record, select a scenario, and then select Delete Record.

Deleting a run record will trigger the deletion of all its target recovery records.

6. To view a live recovery, in the Recovery Scenario Monitor, select a scenario, and then select Recoveries. The Recovery Monitor opens.

24.7 Recovery Scenario User Scripts

When Enable Agent Scripts is set in the Recovery Scenario Details tab, N2WS will run two scripts, one before and one after the recovery run:

• before_<recovery-scenario-name>

• after_<recovery-scenario-name>

A file extension is optional and, if added, may be for any interpreter.

This is somewhat like the Linux Backup Scripts feature described in the Before Script and After Script topics, sections 6.3.1 and 6.3.2.

These scripts must be located on the N2WS server in the following folders:

• For root user: /cpmdata/scripts/scenario

• For non-root user: /cpmdata/scripts/scenario/user_names

24.7.1 Before Script

The before script passes the following parameters, in the following order:

#	Parameter	Notes
1	Scenario Id
2	Account Id	May be null, if the value is NULL.
3	Policy account Id
4	Destination region	May be null, if the value is NULL.

24.7.2 After Script

The after script passes the same parameters as the before with the addition of parameters for the scenario’s recovery targets:

#	Param.	Notes
1-4	...	Same as before_ parameters
5	Target lists	Each target is in colon-separated format:
		RecoveryType:OriginalAwsResourceId: OriginalRegion:RecoveredAwsResourceId
		RecoveryType	Single character identifying resource type:
			I - Instance
			V - Volume
			R - RDS Database
			A - RDS (Aurora) Cluster
			C - Redshift Cluster
			D - DynamoDB
			E - EFS
			F - FSX
		OriginalAwsResourceID	AWS ID of the original resource
		OriginalRegion	AWS region of the original resource
		RecoveredAwsResourceID	AWS ID of the recovered resource. If not recovered, then 'null'.

Following is an example of an after_ script for a Recovery Scenario that was defined with 2 targets: an EC2-instance and an EC2-volume. The after_ script passes 6 parameters, 2 of which are for the targets. In the following example, the instance recovery target was not recovered:

1
null
1
null
I:i-0a87ab83ca3fa62c2:us-east-1:null
V:vol-0197aba1f7090c513:us-east-1:vol-03336f4ed151b5d29