21 Managing Snapshots with Lifecycle Policies

In addition to creating and managing EBS snapshots, N2WS can store backups in Simple Storage Service (S3) and S3 Glacier, allowing you to lower backup costs when storing backups for a prolonged amount of time. N2WS allows you to create a lifecycle policy, where older snapshots are automatically moved from high-cost to low-cost storage tiers. A typical lifecycle policy would consist of the following sequence:

  1. Store daily EBS snapshots for 30 days.

  2. Store one out of seven (weekly) snapshots in S3 for 3 months.

  3. Finally, store a monthly snapshot in S3 Glacier for 7 years, as required by regulations.

Storing snapshots in S3 is not supported for periods of less than 1 week.

Configuring a lifecycle management policy in N2WS consists of the following sequence:

  1. Defining how many EBS snapshots to keep.

  2. Enabling and configuring Backup to S3.

  3. Enabling and configuring Archive to S3 Glacier.

For detailed S3 storage class information, refer to https://aws.amazon.com/s3/storage-classes.

21.1 Using S3 with N2WS

Using the N2WS Copy to S3 feature, you can:

  • Define multiple folders, known as repositories, within a single S3 bucket

  • Define the frequency with which N2WS backups are moved to a Repository in S3, similar to DR backup. For example, copy every third generation of an N2WS backup to S3.

  • Define backup retention based on time and/or number of generations per Policy.

  • N2WS stores backups in S3 as block-level incremental backups.

AWS Encryption at the bucket-level must be enabled.

Only one S3 operation is allowed for a policy at a time – Copy, Recovery, Archive, or retention Cleanup.

  • For instance, an S3 Copy or S3 Recovery is not allowed when the S3 backup retention Cleanup is executing. If a new backup is created while a copy of a previous backup is still running, the new backup will not be copied to S3.

  • Likewise, only one Archive or Cleanup operation can run for a policy at a time, so if a new backup is created while another backup is being archived, or while cleanup is running for the policy, no Archive or Cleanup will be performed for the policy following the completion of the copy operation.

  • If the S3 Cleanup process is running at the time of an S3 Copy or Recovery, you can abort the Cleanup process to allow the Copy or Recovery process to continue. See section 21.5.3.

  • S3 buckets used by Copy to S3 should not be used by other applications.

  • Versioning at the bucket level should be disabled.

Before continuing, consider the following:

  • Copy to S3 currently supports only backups of Windows and Linux instances. RDS, DynamoDB, etc. are not supported.

  • N2WS stores backups in S3 as block-level incremental backups.

  • Most N2WS operations related to the S3 repository (e.g. writing objects to S3, clean up, restoring, etc.) are performed by launching N2WS worker instances in AWS. The worker instances are terminated when their tasks are completed.

21.1.1 Limitations

Only the copy of instance backups is supported.

  • Copy to S3 is supported for weekly and monthly backup frequencies only. Daily backup copies to S3 are not supported.

  • Copy to S3 is not supported for other AWS resources that N2WS supports, such as RDS and Aurora.

  • Snapshots consisting of ‘AMI-only’ cannot be copied to an S3 repository.

  • Due to AWS service restrictions in some regions, the root volume of instances purchased from Amazon Marketplace, such as instances with product code, may be excluded from Copy to S3. The data volumes of such instances, if they exist, will be copied.

  • Backup records that were copied to S3 cannot be moved to the Freezer.

  • Users cannot delete specific snapshots from an S3 repository. S3 snapshots are deleted according to the retention policy. In addition, users can delete all S3 snapshots of a specific policy, account, or an entire repository. See sections 21.2.2 and 21.5.4.

  • A separate N2WS server, for example, one with a different “CPM Cloud Protection Manager Data” volume, cannot reconnect to an existing S3 repository.

  • To use the Copy to S3 functionality, the cpmdata policy must be enabled. See N2WS User Guide for details on enabling the cpmdata policy.

  • Only a single S3 operation is possible on a policy at any given time. Additional executions of Copy to S3 backups will not occur if the previous execution is still running. Restore from S3 is always possible unless the backup itself is being cleaned up.

  • AWS accounts have a default limit to the number of instances that can be launched. Copy to S3 launches extra instances as part of its operation and may fail if the AWS quota is reached. See N2WS User Guide for details.

  • Copy and Restore of volumes to/from regions different from where the S3 bucket resides may incur long delays and additional bandwidth charges.

  • Instance names may not contain slashes (/) or backslashes (\) or the copy will fail.

21.1.2 Cost Considerations

N2W Software has the following recommendations to N2WS customers for help lowering transfer fees and storage costs:

  • When an ‘N2WSWorker’ instance is using a public IP (or NAT/IGW within a VPC) to access an S3 bucket within the same region/account, it results in network transfer fees.

  • Using a VPC endpoint instead will enable instances to use their private IP to communicate with resources of other services within the AWS network, such as S3, without the cost of network transfer fees.

  • For further information on how to configure N2WS with a VPC endpoint, see section Appendix A.

21.1.3 Overview of S3 and N2WS

The Copy to S3 feature is similar in many ways to the N2WS Disaster Recovery (DR) feature. When Copy to S3 is enabled for a policy, copying EBS snapshot data to S3 begins at the completion of the EBS backup, similar to the way DR works. Copy to S3 can be used simultaneously with the DR feature.

21.1.4 Workflow for Using S3 with N2WS

  1. Define an S3 Repository.

  2. Define a Policy with a Schedule, as usual.

  3. Configure the policy to include Copy to S3 by selecting the Lifecycle Management (Snapshot/S3/Glacier tab. Turn on the Backup to S3 toggle and complete the parameters.

  4. If you are going to back up and restore S3 instances and volumes across accounts and regions, prepare a Worker Configuration using the Worker Configuration tab. See section 22.

  5. Use the Backup Monitor and Recovery Monitor, with some additional controls, to manage S3 snapshots as usual.

21.2 The S3 Repository

The cpmdata policy must exist before configuring an S3 Repository.

21.2.1 Configuring an S3 Repository

There can be multiple repositories in a single AWS S3 bucket.

  1. In N2WS, select the S3 Repositories tab, and select New.

  2. In the New S3 Repository screen, complete the following fields, and select Save when complete.

    • Name - Type a unique name for the new repository, which will also be used as a folder name in the AWS bucket. Only alphanumeric characters and the underscore are allowed.

    • Description - Optional brief description of the contents of the repository.

    • User – Select the user in the list.

    • Account - Select the account that has access to the S3 bucket.

    • AWS Region - Select the region in which the S3 bucket is located.

    • S3 Bucket Name - Type the name of the S3 bucket that exists in this region.

AWS encryption must have been enabled for the bucket.

21.2.2 Deleting an S3 Repository

You can delete all snapshots copied to a specific S3 repository.

Deleting a repository is not possible when the repository is used by a policy. You must change any policy using the repository to a different repository before the repository can be deleted.

  1. Select the Repositories tab, and then select a repository.

  2. Select Delete.

21.3 The S3 Policy

21.3.1 Configuring a Policy for Backup to S3

Configuring a Policy for Copy to S3 backups includes definitions for the following:

  • Name of the S3 Repository defined in N2WS.

  • Interval of AWS snapshots to copy.

  • Snapshot retention policy. Selecting the Delete instance snapshots from EBS after storing in S3 option minimizes the time that CPM holds any backup data in the EBS snapshots service. CPM achieves that by deleting any EBS snapshot immediately after copying it to S3.

It is possible to retain a backup based on both time and number of generations copied. If both Time Retention (Keep backups in S3 for at least x time) and Generation Retention (Keep backups in S3 for at least x generations) are enabled, both constraints must be met before old snapshots are deleted or moved to Glacier, if enabled.

For example, when the automatic cleanup runs:

  • If Time Retention is enabled for 7 days and Generation Retention is disabled, S3 snapshots older than 7 days are deleted or archived.

  • If Run ASAP is executed 10 times in one day, none of the snapshots would be deleted until they are more than 7 days old.

  • If Generation Retention is enabled for 4 and Time Retention is disabled, the 4 most recent S3 snapshots are saved.

  • If Time Retention is enabled for 7 days and Generation Retention is enabled for 4 generations, a single S3 snapshot would be deleted, or archived, after 7 days if the number of generations had reached 5.

If Delete instance snapshots from EBS after storing in S3 is enabled in Lifecycle Management, snapshots are deleted regardless of whether the Copy to S3 operation succeeded or failed.

  1. In the left panel, select the Policies tab.

  2. Select a Policy, and then select Edit.

  3. Select the Lifecycle Management tab.

  4. Select the number of EBS Backup Snapshot Generations to keep in the list.

  5. Complete the following fields:

    • Backup to S3 – By default, Backup to S3 is disabled. Turn the toggle on to enable.

    • Store EBS snapshots in S3 based on the following settings:

      • Delete instance snapshots from EBS after storing in S3 –If selected, N2WS will automatically set the Backup to S3 every n (EBS) Backup Snapshot Generations to 1 and will delete snapshots from EBS after performing the Copy to S3 operation.

      • Backup to S3 every n (EBS) Backup Snapshot Generations – Select the maximum number of backup snapshot generations to keep. This number is automatically set to 1 if you opted to Delete instance snapshots from EBS after storing in S3.

  6. In the Keep backups in S3 for at least lists, select the duration and/or number of backup generations to keep.

  7. To Archive to Glacier, see section 21.4.

  8. In the Storage settings section, choose the following parameters:

    1. Select the Target Repository in the S3 bucket to move the backup to, or select New to define a new repository. If you define a new repository, select Refresh before selecting.

    2. Choose an S3 Storage Class that meets your needs:

      1. Standard - (Frequent Access) for Frequent access and backups.

      2. Infrequent Access - For data that is accessed less frequently.

      3. Intelligent Tiering - Automatic cost optimization for S3 copy. Intelligent Tiering incorporates the Standard (Frequent Access) and Infrequent Access tiers. It monitors access patterns and moves objects that have not been accessed for 30 consecutive days to the Infrequent Access tier. If the data is subsequently accessed, it is automatically moved back to the Frequent Access tier.

      See information on S3 Storage Class charges below.

  9. If Archive to Glacier is enabled, select the Archive Storage class.

  10. Select Save.

Storage Class charges:

  • S3 Infrequent Access and Intelligent Tiering have minimum storage duration charges.

  • S3 Infrequent Access has a per GB retrieval fee.

For complete information, refer to AWS S3 documentation.

21.3.2 Recovering an S3 Backup

You can recover an S3 backup to the same or different regions and accounts.

If you Recover Volumes Only, you can:

  • ‌Select volumes and Explore folders and files for recovery.

Explore fails on non-supported file systems. See section 13.1.

  • ‌Define Attach Behaviour

  • Define the AWS Credentials for access

  • Configure a Worker in the Worker Configuration tab.

  • Clone a VPC

If you recover an S3 Instance, you can specify the recovery encryption key:

  • If Use Default Volume Encryption Keys is enabled, the recovered volumes will have the default key of each encrypted volume.

  • If Use Default Volume Encryption Keys is disabled, all encrypted volumes will be recovered with the same key that was selected in the Encryption Key list.

‘Marked for deletion’ snapshots can no longer be recovered.

To recover an S3 backup:

  1. In the Backup Monitor tab, select a relevant backup that as a Lifecycle Status of 'Stored in S3', and then select ​ Recover.

  2. In the Restore from drop-down list of the Recover screen, select the name of the S3 Repository to recover from. If you have multiple N2WS accounts defined, you can choose a different target account to recover to.

  3. In the Restore to Region drop-down list, select the Region to restore the S3 copy to. The source Region of the S3 copy is displayed in the Region column.

  4. Continue with the regular recovery procedure for the resource:

    1. To recover an instance, see section 10.3.

    2. To recover a volume, see section 10.4.

    3. To recover folders or files, see section 13.

To follow the progress of the recovery, select Open Recovery Monitor in the ‘Recovery started’ message ​at the top right corner, or select the Recovery Monitor tab.

To abort a recovery in progress, in the Recovery Monitor, select the recovery item and then select ​​ Abort Recover from S3.

There may be occasions when you need to recover from S3 or retrieve S3 data while there is an active S3 archiving operation. In the Policies tab, select the active policy, and then select Stop S3 / Archive Operation.

21.3.3 Forcing a Single Full Copy

By default, Copy to S3 is performed incrementally for data modified since the previous snapshot was stored. However, you can force a copy of the full data for a single iteration to your S3 Repository. While configuring the Backup Targets for a policy with Copy to S3, select Force a single full Copy. See section 4.2.3.

This option is only available for Copy to S3.

21.3.4 Changing the S3 Retention Rules for a Policy

You can set different retention rules in each Policy.

To update the S3 retention rules for a policy:

  1. In the Policies column, select the target policy.

  2. Select the Lifecycle Management tab.

  3. Update the Keep backups in S3 for at least lists for time and generations, as described in section 21.3, and select Save.

21.4 The Glacier Archive

21.4.1 Archiving Snapshots to S3 Glacier

Amazon S3 Glacier and S3 Glacier Deep Archive provide comprehensive security and compliance capabilities that can help meet regulatory requirements, as well as durable and extremely low-cost data archiving and long-term backup.

CPM allows customers to use the Amazon Glacier low-cost cloud storage service for data with longer retrieval times.

The CPM can now backup your data to a cold data cloud service on Amazon Glacier by moving infrequently accessed data to archival storage to save money on storage costs.

S3 is a better fit than AWS' Glacier storage where the customer requires regular or immediate access to data.

  • Use Amazon S3 if you need low latency or frequent access to your data.

  • Use Amazon S3 Glacier if low storage cost is paramount, and you do not require millisecond access to your data.

21.4.2 Pricing

Following are some of the highlights of the Amazon pricing for Glacier:

  • Amazon charges per gigabyte (GB) of data stored per month on Glacier.

  • Objects that are archived to S3 Glacier and S3 Glacier Deep Archive have a minimum of 90 days and 180 days of storage, respectively.

  • Objects deleted before 90 days and 180 days incur a pro-rated charge equal to the storage charge for the remaining days.

For more information about S3 Glacier pricing, refer to sections ‘S3 Intelligent – Tiering’ / ‘S3 Standard-Infrequent Access’ / ‘S3 One Zone - Infrequent Access’ / ’S3 Glacier’ / ’S3 Glacier Deep Archive’ at https://aws.amazon.com/s3/pricing/

21.4.3 Configuring a Policy to Archive to S3 Glacier

To configure archiving S3 backups to Glacier:

  1. From the left panel, in the Policies tab, select a Policy and then select Edit.

  2. Select the Lifecycle Management (Snapshot / S3 / Glacier) tab. See section 21.3.

  3. Follows the instructions for Backup to S3. See section 21.3.1.

  4. Turn on the Archive to Glacier toggle.

  5. Complete the following parameters:

    • Move one expired S3 backup to Glacier every X period – Select the frequency of archiving.

    • Keep in Glacier for X period– Select the duration of the archive in Glacier.

  6. Select the Archive Storage class:

    • Glacier - Designed for archival data that will be rarely, if ever, accessed.

    • Deep Archive - Solution for storing archive data that only will be accessed in rare circumstances.

The duration is measured from the creation of the original EBS snapshot, not the time of archiving.

21.4.4 Recovering Snapshots from Archive

Archived snapshots cannot be recovered directly from Glacier. The data must first be copied to S3 (‘retrieved’) before it can be accessed.

The retrieving process only runs on objects that have never been retrieved. In other words, an archived snapshot can only be retrieved once.

The process of retrieving data from Archive to S3 is automatically and seamlessly managed by N2WS. However, to recover an archived snapshot, the user should specify the following parameters:

  • Retrieval tier

  • Days to keep

Duration and cost of Instance recovery are determined by the retrieval tier selected. Depending on the Retrieval option selected, the restore completes in:

  • Expedited - 1-5 minutes

  • Standard - 3-5 hours

  • Bulk - 5-12 hours

A typical instance backup that N2WS stores in Glacier is composed of many data objects and will probably take much longer than a few minutes.

To restore data from S3 Glacier:

  1. Follow the steps for Recovering an S3 Backup. See section 21.3.2.

  2. In the Backup Monitor, select a successful Glacier copy, and then select Recover.

  3. In the Restore from drop-down list, select ‘Glacier’. N2WS will copy the data from Glacier to S3 and keep it for the expiration set in Keep data in S3 for x days.

  4. In the Restore to Region list, select the target region.

  5. Select a Retrieval option (Bulk, Standard, or Expedited) and then select Retrieve.

21.5 Monitoring Lifecycle Activities

After a policy with Backup to S3 starts, you can:

  • Follow its progress in the Status column the Backup Monitor.

  • Abort the copy of snapshots to S3.

  • Stop S3 and Archive operations.

  • Delete S3 snapshots.

21.5.1 Viewing Status of Backups in S3 or Glacier

You can view the progress and status of S3 and archived backups in the Backup Monitor.

  1. Select the Backup Monitor tab.

  2. In the Lifecycle Status column, the realtime status of an S3 Copy is shown. Possible lifecycle statuses include:

  • Storing to S3 (n%)

  • Stored in S3

  • Not stored in S3 – Operation failed or was aborted by user.

  • Archiving

  • Archived

  • Marked as archived – Some or all the snapshots of the backup were not successfully moved to Archive storage, either due to the user aborting the operation or an internal failure. However, the snapshots in the backup will be retained according to Archive retention policy, regardless of their actual storage.

  • Deleted from S3/Archive – Snapshots were successfully deleted from either S3 or Archive. See section 21.5.4.

  • Marked for deletion – The backup was scheduled for deletion according to the retention policy and will be deleted shortly.

‘Marked for deletion’ snapshots can no longer be recovered.

21.5.2 Aborting a Copy to S3 ‘In Progress’

The Copy to S3 portion of a Policy backup occurs after the non-S3 backups have completed.

Aborting an S3 Copy does not stop the non-S3 backup portion of the policy from completing. Only the Copy to S3 portion is stopped.

To stop an S3 Copy in progress:

  1. In the Backup Monitor, select the policy.

  2. When the Lifecycle Status is ‘Storing to S3 ...’, select Abort Copy to S3 Snapshots.

21.5.3 Stopping an S3 Cleanup in Progress

If an S3 retention Cleanup is ‘In progress’, in the Policies tab, select the S3 policy and then select Stop S3 / Archive Operations to stop the Cleanup. See the Information in section 21 for the reasons you might want to stop the S3 Cleanup.

  • Stopping S3 Cleanup does not stop the non-S3 cleanup portion of the policy from completing. Only the S3 cleanup portion is stopped.

  • Stopping S3 Cleanup of a policy containing several instances will stop the cleanup process for a policy as follows:

    • N2WS will perform the cleanup of the current instance according to its retention policy.

    • N2WS will terminate all S3 Cleanups for the remainder of the instances in the policy.

    • N2WS will set the session status to Aborted.

    • N2WS user will get an ‘S3 Cleanup of your policy aborted by user’ notification by email.

To stop an S3 Cleanup in progress:

  1. Determine when the S3/Archiving is taking place by going to the Backup Monitor

  2. Select the policy and then select Log.

  3. When the log indicates the start of the Cleanup, select Stop S3 /Archive Operations.

21.5.4 Deleting Copy to S3 Snapshots in a Repository

When deleting Policies and Snapshots in the Policies tab or Account and Data in the Accounts tab, S3 copies are also deleted.

To delete only the snapshots copied to a specific S3 repository:

  1. Select the S3 Repositories tab.

  2. Select a repository, and then select ​ Delete.