Stupid sys-admin’ing, and hooray for LVM and unnecessary partitions

The scenario is: almost finished migrating from an old server to a new server. Only a few steps remain, namely

• change DNS to point to new server
• wipe disks on old server

Done in this order, one might be unlucky that ssh’ing in to wipe the disks lands you on the new server. If you don’t discover this and run the command
dd if=/dev/zero of=/dev/md2
to wipe the disks you might run it on the new server instead. BAM: you just wiped the data of your new server. Fortunately, I realised my mistake after a few seconds, and was able to recover from this with only unimportant data loss, and a little panic.

/dev/md2 is actually a LVM physical partition, of which the first part now only contains zeroes; hereunder all the meta-data. LVM reported no volumes of any kind:
root@server# pvs -v
Scanning for physical volume names
root@server# lvs -v
Finding all logical volumes
No volume groups found
root@server # vgs -v
Finding all volume groups
No volume groups found
After a bit of Google’ing while panicking I found out LVM keeps a backup of all its metadata in /etc/lvm/backup, and that it could be rewritten to the physical volume using:
pvcreate -ff -u XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX --restorefile /etc/lvm/backup/vg0 /dev/md2
where XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX is the physical UUID from the backup file:
...
physical_volumes {
pv0 {
id = "XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX"
device = "/dev/md2" # Hint only ... 
Unfortunately, I got an error:
root@server# pvcreate -ff -u XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX --restorefile /etc/lvm/backup/vg0 /dev/md2
Couldn't find device with uuid XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX.
Can't open /dev/md2 exclusively. Mounted filesystem?
lsof showed nothing accessing md2, but I guess something still had references to the logical volumes. I was unable to get it to work, so I decided to reboot (while making sure the system would come up, and nothing would try to mount the (non-existent) LVM volumes). After a reboot the command worked, but still no logical volumes:
root@server # pvcreate -ff -u XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX --restorefile /etc/lvm/backup/vg0 /dev/md2
Couldn't find device with uuid XXXXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXXXX.
Physical volume "/dev/md2" successfully created
root@server # sudo pvs
PV VG Fmt Attr PSize PFree
/dev/md2 lvm2 a- 648.51g 648.51g
root@server # sudo lvs
No volume groups found
Now I had a physical volume, and could use the vgcfgrestore command:
root@server # vgcfgrestore -f /etc/lvm/backup/vg0 /dev/vg0
Restored volume group vg0
root@server # sudo lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
...
home vg0 -wi-a- 20.00g
I now had my logical volumes back! Now to assess the data loss… The backup file /etc/lvm/backup/vg0 lists which order the logical volumes are stored. The first volume in my case was the “home” volume. Sure enough, it would no longer mount:
root@server # mount /dev/vg0/home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg0-home,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
I actually had no important information on that file system, so I re-created it. Luckily the next volume was intact, and I could bring up the rest of the system without problems.

So to sum up: LVM keeps a backup of all it’s metadata that can be restored (after a reboot), but if your keyboard is faster than your brain it can be a good idea to keep an unused volume as the first logical volume 🙂