Migrating a running Fedora install to BtrFS root

This post documents the steps to transfer a running distro (with a recent kernel) to a new disk with a new file system, in this case, BtrFS.

Preparing new disk

Attach the disk to the running system, either physically as a secondary disk, or by a USB adapter, or by connecting the disk image to the VM. In my example, I am using a disk image in a virtual machine that maps to /dev/vdb. The new disk now needs to be partitioned.

Partitioning

You can use fdisk or gparted, or whatever tool you normally would use. Typically I make the new layout very similar to the old one. For our demonstration purposes, I'll use the following scheme, which does not put home in a separate partition from the root file system

Partition	Size	Type	Purpose or mount point
1	0.5 to 1 GB	Linux EXT4-formatted	Boot partition where grub lives
2	2-4 GB	Linux Swap	Swap partition
3	the rest	Linux BtrFS-formatted	Root and Home will share this main file system

I'm eschewing LVM in favor of raw partitions here, mainly because BtrFS offers many of the same benefits as LVM, so really LVM would be redundant for what we're doing. BtrFS can grow volume to include other devices just like LVM, can do striping and RAID as well. Also normally I'd always recommend putting /home into its own partition to make reinstalling the operating system easier. But for demonstration purposes I'm putting everything in one partition. We will make use of BtrFS subvolumes, though, to allow more flexibility with snapshots.

I'm not showing the fdisk commands I used in my example here. If you're not familiar with fdisk or parted command-line tools, you can use the gparted tool that most distros package. If you do use gparted, you can skip the section on formatting, since gparted does that for you.

Formatting the Partitions

If you used fdisk to create the partitions, you must format the partitions using commands such as the following, as root (sudo if necessary):

# mkfs.ext4 /dev/vdb1
# mkswap /dev/vdb2
# mkfs.btrfs /dev/vdb3

Nowadays most distros use UUIDs to identify the partitions when mounting in /etc/fstab, rather than labels or device paths. This is so that the partitions can still be found even if the order of the disks changes physically. So really labeling the file systems when formatting isn't that essential. In fact it can even be confusing, so I haven't labeled any of the file systems we just made. Later on when we are editing fstab on our new disk, we will retrieve the UUIDs of the file systems we created.

Mounting and working with the new file systems

We can now mount our new file systems and start setting up things like subvolumes and copying files. First we can focus on the partition that will contain /boot:

# mkdir /mnt/newboot
# mount /dev/vdb1 /mnt/newboot
# rsync -raHX /boot/ /mnt/newboot/
# umount /mnt/newboot

That should get /boot copied over. The '-X' argument to rsync is supposed to preserve selinux labels. '-H' preserves hard links, if any. We will also need to re-install grub2 on the new disk a bit later.

The BtrFS file system needs a bit of work before we start copying files over. You could just copy files right into it, but to be able to manipulate snapshots, it's best if we create at least one subvolume to place our actual files into. As this demonstration progresses I think the reason for that will become evident.

So let's mount the file system and start working with it:

# mkdir /mnt/newdisk
# mount /dev/vdb3 /mnt/newdisk
# btrfs subvolume create /mnt/newdisk/ROOT
# btrfs subvolume create /mnt/newdisk/HOME

Those commands should have mounted the file system and created two subvolumes in it, one for the root file system, and one for the home directory files. We gave them arbitrary names "ROOT" and "HOME." They could have been anything we want. And when we are looking at the BtrFS volume, these names will also form a directory hierarchy. When we go to actually use these subvolumes, though, we will actually mount the subvolumes to specific locations in the file system. Subvolume "ROOT" will be mounted to /. "HOME" will be mounted to /home. Different distros will have different default layouts when installing fresh to BtrFS. SuSE uses "@" for the root subvolume and "@HOME" for /home by default I believe. But as we'll see, we can actually change this around down the road using snapshots.

Copying files

Copying files is pretty straightforward with rsync. Just be aware that copying off of a running system results in some files potentially being dirty. If you have any daemons running that maintain open files in /var, such as MariaDB, or any other database system, you will want to shut them down before doing the file copy. Other than this caveat, I've cloned systems off of running distros for many years. Let's start with the root file system, excluding /boot and /home. We've already taken care of /boot, and we'll do /home later. Note the '-x' flag to rsync. This is a critical flag because it tells rsync not to cross filesystem boundaries. We really don't want the contents of /proc copied, for example, although we do want the directory itself created. '-x' does that. So we can copy the root file system by doing:

# rsync -raHX -x / /mnt/newdisk/ROOT/

Pretty simple, but will take a fair amount of time depending on how big your installation is. Now to do /home:

# rsync -raHX /home/ /mnt/newdisk/HOME/

Making it bootable

Now that the files are copied, we need to mount everything in one place so we can chroot into it, fix up fstab, and write the boot loader. My instructions from here on are somewhat Fedora and RHEL -specific, but should apply to other distros that use grub2. We need to unmount our file systems, and remount them all together as they will be when linux boots. /home is not required for this step, but I include it for testing and completeness purposes.

# umount /mnt/newdisk
# mount /dev/vdb3 /mnt/newdisk -o subvol=ROOT
# mount /dev/vdb1 /mnt/newdisk/boot
# mount /dev/vdb3 /mnt/newdisk/home -o subvol=HOME

As you can see we are mounting /dev/vdb3 twice, though specifying the subvol option. It's possible to mark a particular subvolume in BtrFS as the default volume, which means that subvolume will be mounted if no subvol option is passed. This can be useful, but I've found it's just unnecessary.

Before we can chroot into the new root file system, we have to mount a few kernel file systems into it so that grub's installer can work:

# mount /dev /mnt/newdisk/dev -o bind
# mount /proc /mnt/newdisk/proc -o bind
# mount /sys /mnt/newdisk/sys -o bind

We bind /dev to the new image's dev, just so that any dev entries udev created on the host system are available in the chroot. Otherwise, grub's installer cannot function. Also we need information from /proc and /sys. Now we can chroot into the new root file system setup.

# chroot /mnt/newroot

Fixing /etc/fstab

In our chroot, /etc/fstab is just a copy of our running distro's fstab, which is no good. We want it to mount our new file systems. So we'll have to edit fstab to refer to the new file systems, and also subvolumes. First we need to know the UUIDs of each of our file systems. This is done with the blkid command:

# blkid /dev/vdb1
/dev/vdb1: UUID="f10ec507-331a-4e12-b3a6-4d6495f5eae4" TYPE="ext4"
# blkid /dev/vdb2
/dev/vdb2: UUID="eed32947-7c45-4293-b6f1-7a4cc7b613cc" TYPE="swap" 
# # blkid /dev/vdb3
/dev/vdb3: UUID="0710e029-f65b-4a40-b992-2b5f256d5a65" UUID_SUB="f37b82c9-4cf1-479b-9cbc-e0136f96b9b8" TYPE="btrfs"

Make a note of these UUIDs. Maybe you can cut and paste them from another window. On the BtrFS file system, we only care about the UUID, not the UUID_SUB. Now we can edit /etc/fstab (in the chroot--make sure you are in the chroot and not in your running distro's root!). Stick in the new UUIDs and put in the subvol options like this:

UUID=0710e029-f65b-4a40-b992-2b5f256d5a65 /     btrfs defaults,subvol=ROOT 1 1
UUID=f10ec507-331a-4e12-b3a6-4d6495f5eae4 /boot ext4  defaults             1 2
UUID=eed32947-7c45-4293-b6f1-7a4cc7b613cc swap  swap  defaults             0 0
UUID=0710e029-f65b-4a40-b992-2b5f256d5a65 /home btrfs defaults,subvol=HOME 1 3

Basically we just cut and pasted the UUIDs in, overwriting the ones that were already there. Also for / we added the subvol, and then for /home, the line is identical to the line for /, except that we added the "subvol=HOME" option. That's it for fstab, really.

Installing grub2 to the new disk

With fstab in place, it's time to create a new initial ramdisk (which needs to contain the btrfs module), recreate the grub2 configurations, and install grub2 to the boot sector of the new disk.

Initial Ramdisk

On Fedora boxes, rewriting the initial ramdisk is quite easy. All you need to do is this:

# dracut -f --add-drivers "btrfs"

Note that to be completely correct, you should do this for every kernel installed in /boot, just so they are all bootable. Otherwise you might find a kernel panic when the root file system cannot be mounted. I'm actually not sure if the "btrfs" argument is needed to dracut. It may be smart enough to examine root's file system and include the "btrfs" module on its own. But to be safe, specify it.

Updating the grub configuration and menu

Let's see if grub can figure out our new BtrFS configuration properly:

# grub2-mkconfig  | grep subvol
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.10.0-54.0.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-54.0.1.el7.x86_64.img
 linux16 /vmlinuz-3.10.0-54.0.1.el7.x86_64 root=UUID=0710e029-f65b-4a40-b992-2b5f256d5a65 ro rootflags=subvol=ROOT vconsole.font=latarcyrheb-sun16 crashkernel=auto vconsole.keymap=us rhgb quiet 
Found linux image: /boot/vmlinuz-0-rescue-e4bf2781a7cc4bfbe926f01508881e83
Found initrd image: /boot/initramfs-0-rescue-e4bf2781a7cc4bfbe926f01508881e83.img
 linux16 /vmlinuz-0-rescue-e4bf2781a7cc4bfbe926f01508881e83 root=UUID=0710e029-f65b-4a40-b992-2b5f256d5a65 ro rootflags=subvol=ROOT vconsole.font=latarcyrheb-sun16 crashkernel=auto vconsole.keymap=us rhgb quiet 
done

See the "rootflags=subvol=ROOT" part of the linux16 line? That means grub is picking up the BtrFS filesystem and will have the kernel mount the right subvolume that we want. Let's write that configuration to to our menu config file in /boot:

# grub2-mkconfig > /boot/grub2/grub.cfg 
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.10.0-54.0.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-54.0.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-e4bf2781a7cc4bfbe926f01508881e83
Found initrd image: /boot/initramfs-0-rescue-e4bf2781a7cc4bfbe926f01508881e83.img
done

Installing the boot loader

With grub2 configured now, we can write the actually boot loader to the new disk's boot sector, making it bootable.

# grub2-install /dev/vdb
Installation finished. No error reported.

Selinux

If you are using selinux, despite the -X argument to rsync, selinux labels have to be fixed up before it will boot properly. You can try running "fixfiles relabel" or just touch ".autorestart" in the root of your new disk, and then at boot it will relabel. Or edit /etc/sysconfig/selinux on the new disk and set it to "permissive" until you get it up and running successfully.

Cleanup

Now to just clean up and unmount everything:

# exit
# umount /mnt/newdisk/dev
# umount /mnt/newdisk/sys
# umount /mnt/newdisk/proc
# umount /mnt/newdisk/home
# umount /mnt/newdisk/boot
# umount /mnt/newdisk

Conclusion

That's it. The new disk is bootable. Remove the old disk, or make a new vm with the new image and fire it up. Everything should run exactly as it did on the original system, but the underlying file system is BtrFS, and we can now go on to do fun things with like, like snapshots. I'll explore this more in an upcoming blog post.

Blogger's broken editor

Well, except for easily inserting and editing links and images, the WYSIWYG editor built into blogger is useless. It doesn't use paragraph (<p>) tags! What is Google thinking? As a result my paragraphs look awful next to headings. What a mess. So from now on I'm just composing posts largely in HTML mode. It's not too bad and at least it looks just like I expect it should.

Even worse, when you switch back to compose mode, it strips all the <p> back out! Yikes. Blogger is badly broken in this regard. Where can I post a bug report? Sadly with Google there is no customer interface, as we're not the customers; We're the product.

Fixing Kingsoft Writer's default template

Introducing Kingsoft Office

Kingsoft Office is an up-and-coming contender in the field of office desktop productivity software that's dominated by Microsoft.  It's currently available for Linux and Windows.  While it doesn't quite compare to OpenOffice or LibreOffice in terms of breadth of features, it does focus on complete compatibility with the ubiquitous, but not at all standardized, MS Office file formats.  And it's also available in a free version that should fit most home users rather well, if you don't need Visual Basic for Applications macro support.

Unlike OpenOffice and LibreOffice, who are focused on compatibility while promoting their open, and very good in their own right, file formats, Kingsoft is focused nearly completely on bug-for-bug compatibility with MS Office. This means that documents created in MS Office should open in Kingsoft Office and look identical.  Also documents generated in Kingsoft Office and opened in MS Office should open with high fidelity.  Does this mean that you can use Kingsoft Office to edit documents from co-workers without fear of losing vital formatting?  I don't know.  But initial reports are promising.

Kingsoft Office is not open source, though it is freely downloadable.  It's produced commercially by a company in China, where Kingsoft Office is widely used, from what I hear.  It has only recently been becoming known in the English-speaking west, partly due to its well-received Android version of Kingsoft Office.Maybe it's a Chinese plot to steal all our data. Or maybe they are just tired of Microsoft running the show.  They are good at reverse-engineering and duplicating our products and technologies!  In this case I think we all benefit (unless it is a dastardly plot!).

Changing Kingsoft Writer's Defaults

The Kingsoft Writer US english default template is a little weird compared to MS Word and LO/SO.  For one, when you turn on the ruler, the default scale is in "characters."  Now I'm not sure what a character is precisely.  Is it an "en" or an "em" unit, dependent on the current font?  Is it based on the old standard of 10 or 12 characters per inch from mono-spaced, dot matrix printer days?  I haven't check it out.  As well the default tab stop is "2 characters," not the traditional 1/2" that we're used to.  So here I provide instructions on changing the default unit for display, and the default tab stop.  If you don't use inches and 1/2" stops, you can use whatever units you want, and whatever tab stops you want.

Kingsoft Writer's default blank document with strange units and the 2 character tabs

Changing the default units

As mentioned the default units are "characters."  To change this to something more sane, click on the little drop-down arrow next to the blue "Writer" button in the upper left-hand corner of the window if you're running the default UI. This accesses the classic menu, which is already on the screen if you're running in classic theme.  Select "Tools->Options."  You'll be presented with a little dialog box with lots of options.  We want to select "General and Save" on the left side.

Here you can select the "Measurement units" to be something you're more familiar with, like inches.  The option is a bit deceiving, though.  The ruler will still show "characters" unless you also uncheck the box that says, "Use character units."

Now you at least have a more familiar-looking document.  Be sure to cleanly shut down Kingsoft Writer, and these settings will be saved.

Editing the default template

Like in MS Office, the default template gives your new documents their starting settings, such as page size, margins, tab configuration, etc.  The easiest way to change this for all future documents is to edit this default template.  To do that, open the default template as if it were a normal document.  On Linux the default template is located in $HOME/.kingsoft/templates/wps/normal.wpt.  On Windows it is located in your user folder, C:\Documents and Settings\<username>\Application Data\Kingsoft\office6\templates\Normal.wpt  on Windows XP, or C:\Users\<username>\AppData\Kingsoft\office6\templates\Normal.wpt. Unfortunately once you open this template as a document, Kingsoft Office will not let you save it back to the same place. So save it somewhere else and then use Windows Explorer to copy your newly modified Normal.wpt over the one in the AppData or Application Data directory that I mentioned previously.

Opening the default template on Linux

Once the template is open, any change you make to the format of the document, when saved, will become the default.

Changing the tab stops

Even if you had previously change the display units, the default tab stops are still going to be "2 characters," which is a bit unusual.  Now that you're editing the default template, it's easy to change them.

Not sure where to find it on the ribbon, but it's in the conventional menu

In the Tabs dialog you can change the unit and the width of the default tab stops.  Be sure to save the template you're working on (save normally), and it will be the new default for blank documents.

Changing the default margin

If you don't like the default margins, you can change them in the template as well.  I prefer 1" margins, myself.  If you're using conventional menus, that's under File->Page Setup, just like in older versions of MS Office. The ribbon option is under "Page Layout."

As you can see it's about the same as it would be in MS Office or LibreOffice/OpenOffice. The only difference is that you can set the unit here and the value.  If it's not inches (or your preferred unit), feel free to set it.  Changing the unit does not convert the existing value, so if the margin was 25.4 mm, and you change it to inches, the value remains 25.4.  So you'll have to change the value after you change the units.

Save the template, and you're set!