How about a little tale of upgrades for a Friday?
On my dev machine here at the house I was running Debian 8. A few weeks ago, hoping to get more coherent support for things I needed like MySQL 5.7 and PHP 7, I did an in place upgrade to Debian 9. The upgrade proceeded without any issues even, through a couple of reboots and some tests. Finally, with a flourish, there was a final reboot that should have brought the machine back online as a Debian 9 PC.
It crashed.
I was watching a console full of errors of all sorts scroll by. All I could get was an emergency failure prompt. That means something had gone horribly wrong during the boot process. I was quickly able to determine that the issue was in the RAID1 array. I was able to manually mount the RAID volumes and see that the data was still intact. But, something was wrong. I shut it down and let it sitting there.
Since Comcast was up and down today I figured this was a good time to get it that box back up and running. With root access from the emergency prompt I was able to comment out the broken mounts for the RAID volumes. This allowed me to get it to boot but it was unhappy about missing the /var and /home mounts from the RAID.
I quickly discovered the I could use mdadm /dev/md1 -manage to get the RAID device up and then I was able to do the mounts for /var and /home. Using Webmin I checked the status of the RAID noted that both RAID volumes were flagged as degraded and each one only showed a single physical device attached. I readded the necessary physical disks to the RAID and waited for it to resync. Then I put the mounts back in fstab and rebooted. It fired right up and all is well with the world.
It appears that during the upgrade process from Debian 8 to 9 the system chose to only recognize one partition on each physical drive as part of the RAID1 setup. That broke the RAID and disrupted the boot process. I doubt I’ll ever see this sort of thing again.
Just a little hiccup in upgrading from Debian 8 to 9
