# Appendix H - Velero Installation Guide This guide explains how to install Velero on an EKS cluster using only the AWS plugin (without CSIsnapshots). * Installation approaches: Helm or Terraform. * Authentication options: IRSA or EKS Pod Identity. ### Prerequisites * **EKS Cluster Requirements** * **EKS cluster** with either IRSA or Pod Identity enabled * **Kubernetes version** v1.16+ * **AWS Resources** * **S3 bucket** for storing backups * **IAM role** with necessary permissions * **AWS CLI** configured with appropriate credentials * **Tools** * **kubectl** configured to access your cluster * **Helm** for Helm approach * **Terraform** for Terraform approach ### Authentication Methods **Option 1: IRSA (IAM Roles for Service Accounts)** * Uses OIDC provider and IAM roles * More secure, recommended for production * Requires OIDC provider configuration **Option 2: EKS Pod Identity (pod identity agent)** * Simpler setup (no OIDC provider; no SSO/Identity Center involved) * Requires the EKS Pod Identity Agent addon * Trust policy principal: `Service: pods.eks.amazonaws.com` * Create a Pod Identity Association mapping `/` → IAM role ### Approach 1: Helm Installation #### **Step 1**: Create S3 Bucket Create an S3 bucket through the AWS Console or CLI: ``` aws s3 mb s3://your-velero-backup-bucket --region us-east-1 aws s3api put-bucket-versioning --bucket your-velero-backup-bucket --versioning-configuration Status=Enabled ``` #### **Step 2**: Create IAM Role and Policy **Required IAM Permissions** Velero needs specific AWS permissions to perform backups and restores. When creating the IAM policy in the AWS Console, include these permissions: **S3 Permissions** (scope to your Velero bucket only): * s3:ListBucket - on arn:aws:s3:::your-velero-backup-bucket * s3:GetObject - on arn:aws:s3:::your-velero-backup-bucket/\* * s3:PutObject - on arn:aws:s3:::your-velero-backup-bucket/\* * s3:DeleteObject - on arn:aws:s3:::your-velero-backup-bucket/\* **EC2 Permissions** (for EBS snapshot management, resource: \*): * ec2:DescribeVolumes * ec2:CreateSnapshot * ec2:DeleteSnapshot * ec2:DescribeSnapshots * ec2:CreateTags * ec2:CreateVolume * ec2:AttachVolume * ec2:DetachVolume * ec2:DescribeAvailabilityZones * ec2:DescribeTags **Understanding Service Accounts and IRSA** **What is IRSA?** IRSA (IAM Roles for Service Accounts) allows Kubernetes pods to securely use AWS services without storing static credentials. It's the recommended approach for EKS workloads. **How it works:** 1. Your EKS cluster has an OIDC provider that acts as an identity issuer 2. You create an IAM role that trusts this OIDC provider 3. Velero's Kubernetes ServiceAccount ( `velero-server` ) gets annotated with the IAM role ARN 4. When Velero pods run, they automatically receive temporary AWS credentials by assuming the IAM role 5. No AWS access keys are stored in your cluster - credentials are temporary and auto-rotate **Creating the IAM Role (AWS Console)** **Step 2.1: Create IAM Policy** 1. Go to AWS Console → IAM → Policies → Create Policy 2. Click JSON tab and paste a policy with the permissions listed above 3. Name it `VeleroBackupPolicy` 4. Create the policy **Step 2.2: Create IAM Role with OIDC Trust** 1. Go to IAM → Roles → Create Role 2. Select "Web Identity" as trusted entity type 3. Choose your EKS cluster's OIDC provider from the dropdown 4. For "Audience", select `sts.amazonaws.com` 5. Add a condition: 1. Condition: `StringEquals` 2. Key: `:sub` 3. Value: `system:serviceaccount:velero:velero-server` 6. Attach the VeleroBackupPolicy you created 7. Name the role (e.g., `VeleroIRSARole` ) 8. Copy the Role ARN - you'll need it for Helm installation {% hint style="info" %} The OIDC provider URL can be found in EKS Console → Your Cluster → Overview → OpenIDConnect provider URL {% endhint %} **Step 3: Install Velero with Helm** Replace \ , \ , and \ with your actual values. \# Add Helm repository ``` helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts helm repo update ``` \# Install Velero with command-line flags ``` helm install velero vmware-tanzu/velero \ --namespace velero \ --create-namespace \ --set configuration.backupStorageLocation[0].name=default \ --set configuration.backupStorageLocation[0].provider=aws \ --set configuration.backupStorageLocation[0].bucket= \ --set configuration.backupStorageLocation[0].config.region= \ --set configuration.volumeSnapshotLocation[0].name=default \ --set configuration.volumeSnapshotLocation[0].provider=aws \ --set configuration.volumeSnapshotLocation[0].config.region= \ --set initContainers[0].name=velero-plugin-for-aws \ --set initContainers[0].image=velero/velero-plugin-for-aws:v1.10.0 \ --set initContainers[0].volumeMounts[0].mountPath=/target \ --set initContainers[0].volumeMounts[0].name=plugins \ --set credentials.useSecret=false \ --set serviceAccount.server.create=true \ --set serviceAccount.server.name=velero-server \ --set serviceAccount.server.annotations."eks.amazonaws.com/role-arn"= \ --set deployNodeAgent=true \ --set uploaderType=kopia ``` **Step 4: Verify Installation** \# Check pods are running `kubectl get pods -n velero` \# Verify ServiceAccount annotation `kubectl get serviceaccount velero-server -n velero -o yaml` \# Check Velero can authenticate to AWS `kubectl exec -n velero deployment/velero -- aws sts get-caller-identity` \# Check backup storage location `kubectl get backupstoragelocation -n velero` **Step 5: Install Velero CL**I Download the Velero CLI to manage backups from your machine: \# Linux ``` wget https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz tar -xvf velero-v1.14.0-linux-amd64.tar.gz sudo mv velero-v1.14.0-linux-amd64/velero /usr/local/bin/ ``` \# macOS `brew install velero` \# Verify `velero version` #### `Step 6: Test Backup` \#Create a test backup `velero backup create test-backup --include-namespaces default` \# Check status `velero backup describe test-backup` \# List backups `velero backup get` ### Approach 2: Terraform Bootstrapping #### 2.1 Choose authentication (IRSA or EKS Pod Identity) **IRSA (existing/standard)** * Ensure an EKS OIDC provider exists. * Create an IAM role with trust for that OIDC provider and `system:serviceaccount:::` . * Annotate the ServiceAccount with the role ARN: * `serviceAccount.server.annotations."eks.amazonaws.com/role-arn" = ` * Use the key exactly `eks.amazonaws.com/role-arn` (case-sensitive) **EKS Pod Identity** * Install addon: `eks-pod-identity-agent` . * Create an IAM role with trust principal `Service: pods.eks.amazonaws.com` . * Create a Pod Identity Association mapping ` /` → role. * Remove any IRSA annotations from the SA. #### 2.2 Minimal Terraform for Velero (AWS plugin by default) ``` resource "helm_release" "velero" { name = "velero" repository = "https://vmware-tanzu.github.io/helm-charts" chart = "velero" version = "10.0.11" namespace = "velero" create_namespace = true values = [yamlencode({ configuration = { # Default: AWS plugin only (no CSI) features = "" backupStorageLocation = [{ name = "default" provider = "aws" bucket = "" config = { region = "" } }] # Enable CSI snapshots (optional): # features = "EnableCSI" # volumeSnapshotLocation = [{ name = "default", provider = "aws", config = { region = "" } }] # extraArgs = ["--default-snapshot-move-data=true"] # Only applies to CSI data-mover flows } # Plugins plugins = ["velero/velero-plugin-for-aws:v1.12.0"] # FS backups (optional but recommended to have available) deployNodeAgent = true uploaderType = "kopia" # IRSA: annotate SA with the role (if using IRSA) # serviceAccount = { server = { create = true, name = "velero-server", annotations = { "eks.amazonaws.com/role-arn" = var.velero_role_arn } } } })] } ``` #### 2.3 CSI prerequisites (only if enabling CSI) * Install the snapshot-controller (e.g., Piraeus `snapshot-controller` chart) and CRDs. * Create an EBS CSI `VolumeSnapshotClass` ( `driver: ebs.csi.aws.com` ). * Label the VSC so Velero detects it: `velero.io/csi-volumesnapshot-class=true` . {% hint style="info" %} Tip: Run snapshot-controller with `replicaCount: 2` . {% endhint %} #### 2.4 Deploy ``` terraform init terraform apply -auto-approve ``` #### 2.5 Verify ``` kubectl -n velero get deploy velero -o jsonpath='{.spec.template.spec.containers[0].args}'; echo kubectl -n velero logs deploy/velero | grep -Ei 'velero-plugin|credentials|error' || true ``` ### Verification #### **Check Installation** \# Check Velero pods `kubectl get pods -n velero` \# Test backup `kubectl exec -n velero deployment/velero -- velero backup create test-backup --include-namespaces default` ### **Required IAM Permissions** **S3**: List, Get, Put, Delete objects in backup bucket **EC2**: Describe volumes, create/delete snapshots, create tags ### **Troubleshooting** **Common Issues** 1. **Authentication not working**: Check IRSA or Pod Identity configuration 2. **S3 access denied**: Verify IAM role permissions 3. **Plugin not found**: Check plugin image #### Debug Commands \# Check Velero logs `kubectl logs -n velero deployment/velero` \# Test AWS credentials `kubectl exec -n velero deployment/velero -- aws sts get-caller-identity` ### Security Considerations 1. **Least privilege**: Only grant necessary permissions 2. **Bucket encryption**: Enable S3 bucket encryption 3. **Access logging**: Enable S3 access logging ### Cost Optimization 1. **Lifecycle policies**: Set S3 lifecycle rules 2. **Snapshot retention**: Configure deletion policies 3. **Storage classes**: Use appropriate S3 storage classes #### Comparison: Installation Methods | Aspect | Helm | Terraform | | -------------- | -------------- | ----------------- | | Complexity | Simple | Medium | | Infrastructure | Manual | Automated | | Multi-cluster | Manual | Easy to replicate | | Best for | Single cluster | Multiple clusters | ### Support For issues with this setup: 1. Check Velero documentation: [https://velero.io/docs/ ](https://velero.io/docs/) 2. Review AWS EKS documentation: 3. Check Velero GitHub issues: ### Additional Notes * To enable CSI snapshots, follow Section 2.3 (controller, VSC, label) and set `configuration.features = "EnableCSI"` in Terraform. * For file-system backups, enable `deployNodeAgent = true` and `uploaderType = "kopia"` ; opt-in per PVC with `velero.io/fs-backup=true` . * If not on AWS, use the cloud plugin for your provider instead of the AWS plugin: * GCP: `velero/velero-plugin-for-gcp` * Azure: `velero/velero-plugin-for-microsoft-azure`