Incremental Kubernetes backups need a tested restore path

A backup finishes. The dashboard turns green. During an outage, the team discovers that recovering the application requires a base snapshot nobody retained, credentials nobody can retrieve, or a restore procedure written for the previous software version. These are the failure cases to test before changing an incremental backup system.

For platform and storage teams, Kubernetes changed block tracking raises a useful question: can the backup application identify the data it needs without repeatedly scanning everything? The next question belongs in the same change review: can the team recover a usable application from what it kept? This article proposes a review and rehearsal, not a claim that Valen Systems has tested your storage stack.

A September explanation of a March API change

Kubernetes published its beta migration explanation on September 14, 2026. The milestone itself dates to the external-snapshot-metadata v1.0.0 release in March. The post describes promotion of SnapshotMetadataService from v1alpha1 to v1beta1, removal of the old served version, and no automatic conversion. Alpha users need to update the CRD, their resource manifests, and clients that address it. The post limits this feature to block volumes, not changed-file lists for network shares.

There is also a compatibility discrepancy worth resolving before installation. The September post lists Kubernetes 1.33 and CSI 1.10 as minima; the CSI developer table retrieved on September 21 lists Kubernetes 1.36 and CSI 1.12 for v1.0.0. Don't choose between those numbers by preference. Confirm the supported combination with the maintainers of the actual driver and backup application, then pin the versions used in your test.

Metadata is one part of the recovery path

The CSI documentation describes an authenticated service that streams snapshot metadata from the driver to a backup application. Availability depends on driver support; deploying Kubernetes alone doesn't provide it. The client also needs the advertised endpoint, certificate trust, and the appropriate token and permissions.

Our recommendation is to follow that chain all the way to restored application behavior. Listing changed blocks doesn't demonstrate that the backup application copied the right data, retained every dependency, or can reconstruct an older recovery point. Those need their own evidence. Ask for the backup product's supported recovery procedure rather than treating an API demonstration as a finished backup service.

Choose a recovery point that makes the test meaningful

Start with an approved, isolated test environment and a representative application. Before the exercise, write down the recovery point, its dependent artifacts, and the transaction or query that will establish whether the application is usable. Agree what data may be lost and how long recovery may take. These are proposed acceptance criteria for your team to set, not performance promises from the upstream project.

Include an older retained recovery point as well as the newest one. If the backup product uses a base plus subsequent changes, identify everything that point requires. Rehearse against a protected test copy. Don't delete production snapshots or expire real retention chains to manufacture a failure.

• Record the Kubernetes, CSI driver, metadata sidecar and backup application versions, plus the relevant configuration revision.
• Identify where the recoverable data and required metadata live. Check access from the recovery environment, not just from the healthy source cluster.
• Use the product's supported restore procedure. Preserve the commands or job identifiers, timestamps, warnings and actual result.
• Run the agreed application check after storage recovery. A mounted volume or a running pod isn't the same result as a valid business transaction.
• Record elapsed recovery time and the age of the recovered data separately. Keep planned targets separate from measurements.

Test the upgrade boundary, not just a clean installation

For an existing alpha installation, the valuable test is the transition. Can the updated client discover the service? Can it use the intended identity? Can the backup application still recover a point created before the change, where that workflow is supported? A successful fresh installation leaves those questions open.

Ask the driver and backup maintainers what rollback actually preserves. A saved manifest doesn't establish that older clients, stored resources and recovery artifacts remain compatible after an upgrade. Keep the known-good recovery procedure and artifacts available until the agreed checks pass. If a supported reversal isn't established, record that as a release decision instead of describing the change as reversible.

Buy the work that is actually missing

If the gap is a scattered recovery procedure, document it. If the gap is an untested restore, scope an authorized exercise with the people who own the application and storage. Those are different jobs, and a document shouldn't be sold as evidence that the exercise happened.

Valen Systems' Backup Recovery Runbook is a smaller starting point: supplied-evidence documentation for one host and one existing backup system. It does not include a live restore, backup installation, disaster response, or a recovery guarantee. A multi-cluster backup migration needs a separate scope discussion. Tell us what you operate and which recovery question you need answered.

Sources

1. Kubernetes: Changed Block Tracking API, Beta DifferencesSource published September 14, 2026. Retrieved September 21, 2026.
2. Kubernetes CSI developer documentation: external-snapshot-metadataPublication date not stated. Retrieved September 21, 2026.