All Hard Drives Go To Heaven (How To Recover From Drive Failure)

One neat thing about Raspberry Pi's is they use SD cards for storage. They seem rather reliable; they die when their memory cells start going kerplooey. Unfortunately SD cards tend to provide little in the way of storage capacity.

For this reason I added a scrap 500 gig laptop hard disk to the Pi I am using to experiment with Go. It's connected to a SATA-USB converter, which in turn connects to a USB hub. Spin drives, though remarkably reliable today, still apparently have limits.

The other day I was working on fixing a problem with a missing godoc install when the external drive wouldn't let me write to it. I checked with 'mount' and, sure enough, it was mounted as a read-only drive.

This should have triggered an alarm bell...after all, fstab lists the drive as mounted in RW, and I hadn't remounted it, but in my distracted mind I chalked it up to a fluke and decided to reboot it and let that sort it out. 

sudo shutdown -r

Oh, side note of some importance. I access this Pi through ssh. 

After giving it a few minutes to boot up, I tried reconnecting. Nothing.

Maybe it needed a hard restart? I pulled power to the USB hub, causing it all to power down. Count to ten. Plug it back in. After a bit of time, try connecting to it again. Nothing.

I pull the unit out and plug in an HDMI monitor and boot again. Doesn't take long to see the problem. The screen fills with errors as the kernel tries mounting the external drive.

Aw, damn.

Bad news, all my source code experimentation is gone. Good news is I had a spare drive to replace the dead one. Here's what you do.

First, the Pi eventually logged in Root at the console without the network launching. I copy /etc/fstab to /etc/fstab.old, then opened fstab and removed the UUID line that mounted the external drive. Then I shut down the Pi and switch out the drive.

Power it back up, and now networking, along with ssh, is back. 

Dmesg tells me the new drive, like the old, is on /dev/sda. 

I open fdisk and removed all the existing partitions with the "d" command, then create one big partition with "n". Create it with "p"rimary, 1, and accept the defaults for start and end sectors. Exit fdisk with "w" command so fdisk writes changes to the drive.

Next I formatted the drive with:

sudo mkfs.ext4 /dev/sda1

At this point you'd better be really sure you're working with the right drive or you're a little more than screwed.

At this point I want it to set the new drive to mount automatically at boot. I run:

sudo blkid

...and copy the UUID to my clipboard (I'm running iTerm to ssh to the Pi, so copy and pasting is really easy.) I then re-create the fstab entry using a lazy method.

cd /etc

mv fstab fstab.orig

mv fstab.old fstab

I open fstab in a text editor and replace the dead drive's UUID with the new UUID in the clipboard. Save the file (you did launch the editor using su, right? You need root privileges to edit fstab...)

Reboot. Did it mount? If yes, you're ready for the next step.

I re-created my go_projects folder with sudo, and chowned the directory to my user.

Now experimenting with git pays off (and this is why the post is also tagged with "programming."

git init

git remote add origin git@<my git server repo>:/<gitpath>

git pull origin master

This only pulled a ./src folder. I added the other two workspace folders..

mkdir bin pkg

Yay! Back in business!

Source control is not only great for keeping your source code protected as you make changes and work in groups, but it also allows you to have a kind of distributed backup!