Verify offline data volume rollback with LVM snapshots

An application upgrade can change more than the executable: it may also modify indexes, resource files, and data formats. Keeping the old installer does not guarantee that the data volume can be rolled back. On a dedicated server already using LVM (Logical Volume Manager), you can stop all writes, create a short-lived snapshot, and restore the pre-upgrade state through an offline merge. What matters is not simply whether the snapshot command succeeds, but whether write activity is controlled, capacity remains available, and the restored state passes validation.

1. Confirm the target volume and rollback scope

This guide uses Debian 13, an ext4 file system, and a classic LVM data volume. It does not cover root file systems, boot volumes, or thin pools. CloudUDP's standard configuration includes two directly attached 1TB NVMe drives, with no hardware RAID. That does not mean the system is already configured with LVM; you must verify the actual storage layout on the server.

First, record the mount source, file system type, logical volume attributes, and free space in the volume group:

findmnt -T /srv/appdata -o SOURCE,TARGET,FSTYPE,OPTIONS
sudo lvs -a -o vg_name,lv_name,lv_attr,segtype,origin,data_percent
sudo vgs -o vg_name,vg_size,vg_free
sudo lsblk -f

If the target is an ordinary partition, stop here: this procedure does not apply. Do not format existing data just to enable snapshots. If the output shows thin volumes, a caching layer, or another complex layout, follow the procedure for that storage configuration instead.

The following examples assume that the data volume is vgdata/appdata, its mount point is /srv/appdata, and the associated service is app.service. Replace these names with the actual values on your server, and confirm that there are no nested mounts or other processes writing to the volume. Run the commands in stages; do not copy and execute the entire article as a script.

A snapshot is not a backup

A classic snapshot uses copy-on-write to preserve data blocks before they are overwritten on the origin volume. It depends on that volume and the underlying storage. It neither protects against drive failure nor includes configuration files, application files, or remote data outside the volume.

Merging a snapshot discards every write made to the volume after the snapshot was created. If the upgraded application has already accepted production requests, first export any new data that must be retained and confirm that it is compatible with the old application version. Do not roll back directly.

2. Stop services and create a consistent recovery point

Before upgrading, prepare an independent backup and record the application version, a summary of its configuration, and its health status. A snapshot preserves only the block-level state of the specified volume. If multiple volumes or external databases are involved, you still need to define a consistency boundary across those components.

Before stopping the service, pause incoming requests, queue consumers, and scheduled tasks so that no other process continues modifying data afterward. The commands below use successful unmounting as the condition for proceeding; do not use forced or lazy unmounting:

sudo systemctl stop app.service
sudo fuser -vm /srv/appdata
sudo sync
sudo umount /srv/appdata

If unmounting fails because the volume is busy, check working directories, open files, and other services. Do not continue until all writers have been stopped. Then create the snapshot:

sudo lvcreate --snapshot --size 20G \
  --name preupgrade /dev/vgdata/appdata
sudo lvs -o lv_name,lv_attr,origin,data_percent vgdata
sudo mount /srv/appdata

The 20G snapshot allocation is an example, not a product specification or a universal recommendation. Before running the command, confirm that the volume group has enough free space and that preupgrade does not already exist. Size the snapshot based on the data blocks likely to be overwritten during the rollback observation period, rather than simply allocating a fixed percentage of the origin volume's total capacity.

After remounting the origin volume, create a small validation file, then carry out a controlled upgrade. Continue blocking real production writes at this stage; otherwise, a later rollback will require additional steps to preserve and transfer new data.

3. Inspect the snapshot read-only and monitor usage

After creating the snapshot, verify that its contents are readable. The following options apply only to the ext4 example in this guide: ro mounts the file system read-only, and noload prevents journal replay during mounting. Do not apply these options directly to other file systems.

sudo mkdir -p /mnt/app-preupgrade
sudo mount -t ext4 -o ro,noload \
  /dev/vgdata/preupgrade /mnt/app-preupgrade
sudo sha256sum /mnt/app-preupgrade/manifest.json \
  | sudo tee /root/app-preupgrade.sha256
sudo umount /mnt/app-preupgrade

Here, manifest.json stands for a stable application data file chosen for verification. Confirm beforehand that it exists and is meaningful for validation. Store the checksum record outside the target volume. After rollback, compare only the checksum field, not the mount path.

Assess risk from the rate of change

sudo lvs -a -o lv_name,lv_attr,origin,data_percent vgdata
sudo vgs -o vg_name,vg_free

Data% is the percentage of classic snapshot space in use, not the file system usage of the origin volume. Sample it regularly before and after the upgrade to track how quickly it grows. Operations such as rebuilding indexes in bulk or overwriting large files can consume snapshot space rapidly.

As usage approaches your predefined warning threshold, pause further writes and assess whether to extend the snapshot. If the volume group still has free space and the snapshot remains valid, increase its capacity according to that assessment:

sudo lvextend -L +10G /dev/vgdata/preupgrade

Adding space does not extend the rollback window permitted by your operational requirements. If the snapshot is already invalid, extending it is not a repair; switch to the independent backup recovery procedure. Snapshot monitoring without a way to receive alerts and respond to them does not provide reliable protection.

4. Merge offline and verify the restored state

Once you decide to roll back, first block incoming production requests and save evidence of the failure along with any new data that must be retained. Restore application files and configuration outside the volume to matching versions as well. Restoring only the data volume can leave old data paired with a new application version.

Confirm that the snapshot is valid and its mount point has been unmounted. Then stop all associated services and unmount the origin volume. The following operations change data, so check the volume names again before running them:

sudo systemctl stop app.service
sudo umount /srv/appdata
sudo lvchange -an /dev/vgdata/appdata
sudo lvconvert --merge /dev/vgdata/preupgrade
sudo lvchange -ay /dev/vgdata/appdata
sudo lvs -a -o lv_name,lv_attr,origin,data_percent vgdata

If the merge is scheduled for the next activation, monitor its status until it completes. Do not restart production workloads merely because the command returned successfully. Once the merge succeeds, the snapshot volume disappears. Remount the origin volume and check it:

sudo mount /srv/appdata
sudo sha256sum /srv/appdata/manifest.json
sudo cat /root/app-preupgrade.sha256

The file checksum should match the baseline, and the validation file created after the snapshot should be gone. These checks demonstrate only that the specified files were rolled back, not that the application is functioning correctly. Also verify the application version, counts of key records, read results, and one controlled write. Then start the service and review its logs.

5. Turn the rehearsal into an upgrade checklist

First, rehearse the complete procedure on a non-production data volume. Record snapshot creation, capacity growth, merge status, and application validation results. Before a production upgrade, confirm at least the following:

• The target is an existing classic LVM data volume, and the mount source matches the volume name.
• All writers can be stopped, the origin volume can be unmounted normally, and the independent backup has been verified.
• Snapshot sizing is based on expected block changes, with free space reserved in the volume group for expansion.
• The old application version, configuration outside the volume, and recovery checklist are all available.
• The method for retaining new writes and the conditions for resuming production service are clearly defined.

After the upgrade passes validation and the rollback observation period ends, confirm that the snapshot is not mounted, then use lvremove /dev/vgdata/preupgrade to remove it if it still exists. A snapshot that has already been merged does not need to be deleted again.

The final deliverable should be an operations record containing the volume layout, recovery point, capacity measurements, and application validation results. A snapshot is only a short-term rollback tool. Reliability comes from a repeatable recovery process, not from keeping an extra volume name around.

Frequently asked questions