Migrating an ext3/4 filesystem to LVM with minimal downtime

There have been a few posts asking about moving a Linux file system from a raw partition onto an LVM. The advantages to using LVMs are numerous, but the advice has generally been to reinstall from scratch.

The problem is, several of the steps must be performed while the file system is unmounted. Specifically, both reducing the ext2/3/4 file system size and moving it from the raw partition onto the logical volume. Additionally, the /boot directory must be moved to a separate physical partition, and the initramfs it contains needs to have the lvm2 tools installed.

The following steps can be followed to migrate from a single monolithic partition (+ swap) to a typical LVM configuration where root and swap are on logical volumes. It may be possible to encrypt the LVM physical volume during this process, and I may explore that in a later post.

Assumptions:
/dev/sda1 contains a monolithic / file system
/dev/sda5 contains the swap partition
The boot loader is GRUB2

Step 1: Make a backup

Seriously. Don't have any critical data that's only available on this system. In fact, you may want to take an older, idle system and practice this whole procedure on it. There are plenty of opportunities to screw up, and this guide assumes you're already pretty familiar with the LVM2 and ext tools and HDD partitioning.

Step 2: Prepare the system

Add lvm2 and dependant packages if needed. On Ubuntu, this would be:
# apt-get -y install lvm2

Determine whether you will need a swap file. Take a look at the output of top to see if you've got enough physical memory that you can just disable swap for the duration of the migration. If you will not need swap, skip ahead to plan your partition size. If you will need it, migrate your swap from a partition to a swap file using the following steps (example below creates a 256MB file; increase as you see fit):
# dd if=/dev/zero of=/swapfile bs=1048576 count=256
# mkswap /swapfile
# echo "/swapfile none swap sw 0 0" >> /etc/fstab
# swapon -a
# cat /proc/swaps
Verify that /proc/swaps shows that /swapfile is active and the right size.

Step 3: Make a plan

You need to determine the size and location of a temporary partition to hold your file system while you're making the conversion. You will need the file system "block size" from:
# dumpe2fs -h /dev/sda1
On my test plaform, it is 4096, but may vary depending on your disk size.

You will also need the sector size from:
# fdisk -lu /dev/sda
It is almost always 512. Using these two values, calculate the sectors per block by dividing block size by sector size (eg. 8)

From the fdisk output, you will also need the starting sector of partition 1 (usually either 63 or 2048), and the count of total sectors displayed in the header above the sector size.

You will also need to know the amount of space used on your root filesystem:
# df -h /

Now determine your temporary file system size. Take your used space, and add a comfortable margin of 10-20%, and round up. For instance, if you're using 1.5GB, you may want to make it 3GB. If you're using 45GB, you may want to keep at least 50GB, perhaps more.

In any case, since you'll be duplicating it across your internal disk, your target size should be a little less than 50% of your total disk size. If you cannot shrink the file system to less than 50% of the HDD size, you may have luck with an external eSATA or USB drive, but test it first.

Step 4: Boot into LiveCD

Now we get to the scary part. Since you're bringing the system down, if there are business critical apps running on it, you'll have to do this during a maintenance window.

Step 5. Re-size the file system

This is the main reason the file system must be unmounted. This command is not supported on a mounted ext2/3/4 file system. Substitute the value of 50G in the example below with the size you determined above.
# e2fsck -f /dev/sda1
# resize2fs /dev/sda1 50G

Determine the number of sectors used by the re-sized file system. Do this by multiplying the "Block count" from the following command with the sectors per block calculated above:
# dumpe2fs -h /dev/sda1
Add about 10000 sectors to this for the LVM headers, and subtract it from the drive's total sector count recorded above to determine the starting sector for partition 3.

Step 6. Repartition the drive

This will make room for the temporary partition
# fdisk -cu /dev/sda
Press 'p' to print the current partition table.
Press 'd' and '5' to delete the swap partition, 'd' and '2' to delete the extended partition table, and 'd' to delete the last remaining partition.
Create partition 3 to hold your temporary file system by pressing 'n', 'p', '3', "<start sector P3>", "" (New Partition, Primary, partition 3, starting sector, max size). Substitute the starting sector with what was calculated above.
Recreate partition 1 with the size determined above by pressing 'n', 'p', '1', "<start sector P1>", "" (New partition, Primary, partition 1, starting at 63 (or 2048), max size).
Finally, press 'w' to write your changes to disk and recreate the device nodes.

If ioctl is unable to re-read the partition table after writing to disk, you must reboot into the LiveCD to recreate the proper block devices.

Step 7. Install LVM tools.

You'll need internet connectivity for this, so if you're using a fixed IP, you'll need to first configure the interface and default route manually. Then install the LVM2 tools using a command such as:
# apt-get -y install lvm2

Step 8. Create your logical volume.

Use the following sequence of commands to create a mapped device for your root file system:
# pvcreate /dev/sda3
# vgcreate os /dev/sda3
# lvcreate -l 100%VG -n root os

Verify there are enough sectors to hold existing file system
# blockdev --getsize /dev/mapper/os-root
The number of sectors returned must be more than the file system size calculated above. If not, subtract a few thousand sectors from your partition 3 starting sector, repartition, and try again.

Step 9. Copy your root file system.

This is the other command that must be done while the file system is unmounted:
# dd if=/dev/sda1 of=/dev/mapper/os-root

Step 10. Repartition the drive again

Deactivate the volume group to allow the next command to succeed:
# vgchange -a n os

Up until this point, your system would still boot if you lost power. From this point until GRUB is reinstalled, you'll need the LiveCD to recover your system.
# fdisk -cu /dev/sda
Delete partition 1 by entering 'd', '1'.
Create a new 300MB partition 1 for the /boot file system by entering 'n', 'p', '1', "", "+300M".
Create an extended partition table with the remaining space by entering 'n', 'e', '2', "", "".
Create a logical partition filling the extended partition by entering 'n', 'l', "", "".
Change the file system type value on partition 5 by entering 't', '5', "8e".
Write changes to disk by entering 'w'.

Step 11. Separate /boot to its own partition

The /boot directory contains the kernel and initramfs as well as all of GRUB's modules and config files. It must be available as a separate file system in order to map the logical volume and access root.
# vgchange -a y os
# mount /dev/mapper/os-root /mnt

# mke2fs /dev/sda1
# mkdir /newboot
# mount /dev/sda1 /newboot
# cd /mnt/boot
# tar cf - . | (cd /newboot; tar xvf - )
# echo "/dev/sda1 /boot ext2 defaults 2 0" >> /mnt/etc/fstab

Step 12. Reinstall GRUB

These steps are for GRUBv2. Substitute appropriate commands for GRUB Legacy.
# for mp in dev sys proc; do mount -o bind /$mp /mnt/$mp; done
# chroot /mnt
# update-grub
# grub-install /dev/sda
Handle any errors you see at this point or your system will not boot.

Step 13. Reboot

You should be able to boot into your original system at this point, albeit with reduced file system space.
If you get a GRUB rescue prompt, something went wrong with your GRUB install.
If you get a GRUB error about no disk found, the GRUB configuration was not built correctly.
If you get the {initramfs} prompt, it could not find the root file system in the logical volume, was lvm2 installed before booting into the LiveCD?
I got a blank screen the first boot, but it appeared to be an X.org issue. I hit Ctrl-Alt-Del, and it came up fine upon rebooting. It scared me, but turned out to be nothing.

Step 14. Move your logical volume.

If you're going to encrypt the resulting system, here is where you would prepare /dev/sda5. I may try this in the future, but for now, you're on your own if you attempt this.

These commands add the new space to the volume group and move root to it:
# pvcreate /dev/sda5
# pvscan
# vgextend os /dev/sda5
# pvmove -i 60 -n root /dev/sda3 /dev/sda5

Then you can remove the unused physical volume:
# pvscan
# vgreduce os /dev/sda3
# pvremove /dev/sda3

Step 15. Repartition the drive one final time

Now that we no longer need partition 3, we delete it and extend the logical partition all the way.
# fdisk -cu /dev/sda
Note the starting sector of P2 when you enter 'p'.
Remove partition 3 by entering 'd', '3'.
Remove partitions 5 and 2 by entering 'd', '5', 'd', '2'.
Recreate extended partition 2 by entering 'n', 'e', '2', "", "".
Recreate logical partition 5 by entering 'n', 'l', '5', "", "".
Write changes to disk by entering 'w'.
Note: you will see a warning about ioctl being unable to reload the partition table. This is expected, since the file system is using it.

Step 16. Reboot into your final configuration.

If your reboot is to be delayed, you can buy some time by growing the root logical volume and file system to more usable sizes until you can reboot. If you can reboot now, don't bother with resizing here, as you'll be able to grow it to its final size after rebooting.
# lvresize -l 100%VG /dev/mapper/os-root
# resize2fs /dev/mapper/os-root

Step 17. Grow root to its final size.

Now is your chance to determine how much swap space you really want. If you don't need swap at all, use '-l 100%VG" to take all available space. Otherwise, leave space for a swap volume added in the next step.

Use a combination of the "-l nn%VG" and "-L +n.nG" options to bring your root volume up to the desired size. Do not shrink it unless you know for sure you're not releasing areas that your file system is using.
# pvresize /dev/sda5
# pvs
# lvresize -l 95%VG /dev/mapper/os-root
# lvresize -L +1.24G /dev/mapper/os-root
# resize2fs /dev/mapper/os-root

Step 18. Restore your swap partition

Create your swap partition and restore swap to it.
# lvcreate -n swap -l 100%FREE os

# mkswap /dev/mapper/os-swap
# swapon -a
or, to activate it automatically, use the UUID specified in your fstab file as:
# mkswap -U /dev/mapper/os-swap

Verify your swap partition is active before disabling the swap file with:
# cat /proc/swaps
# swapoff /swapfile
# rm /swapfile

Finally, edit your /etc/fstab file to remove the line starting with /swapfile.

Step 19. Breathe

You've done it! I believe this is the best way to perform this task with the least amount of down time. If you have suggestions or other thoughts, let me know.

Using ssh-agent with an encfs encrypted home folder

Passwords and pass-phrases are an authentication method; they authenticate the user as the only one, at least in theory, who knows the correct phrase. With encryption tools, they serve an additional purpose; they either generate the symmetric key directly through one or more hashes (eg. cryptsetup), or they decrypt an existing stored key (eg. LUKS, GnuPG).

SSH2 public key authentication is purely an authentication method. There is no symmetric key involved. The requester (eg. sshd) generates a challenge, sends it to the client or agent for a signature using the private key, and verifies the signed response using the public key. If the public key successfully decrypts the signature, the private key must have generated it, and thus the user, as the only one with access to the private key, in theory, is authenticated.

I recently worked with someone who was encrypting their home folder on a remote system with pam-encfs, but wanted to use public key authentication for the initial SSH authentication, rather than an interactive password. The first problem with remotely accessing encrypted home folders via SSH is with the authorized_keys file. This file is normally in the user's home directory, and thus inaccessible until the user logs in and decrypts it. There are known ways around that, at least, such as placing the file outside of the user's home directory.

The second problem is generating the decryption key. When using SSH public key authentication, there is no password for pam-encfs to use to generate the symmetric key. Since the authentication task is passed back to the ssh client or agent, the password is never present on the remote system.

Stepping back for a moment, pam-encfs does seem to be the logical place to decrypt and mount the file-system. It has the advantage of PAM's modular design, and knows when the session start and session stop events occur to mount/umount it. Plus, at least with interactive logins, PAM already has access to the user's pass-phrase, and can pass that to encfs without needing a second prompt.

Not so with public key authentication, unfortunately. There is no way to uncover the symmetric key without this pass-phrase or the key material that is generated from it. So the question becomes, while authenticating the user via the SSH agent, how can we access a pass-phrase for encfs? It must be stored somewhere, with agent authentication used to authorize access to it. I've come up with a couple of ideas, neither of which I'm thrilled with.

Hold password in trusted keyring with access authorized by agent

This would require a keyring of some sort on the remote system which maintains the password. If it encrypts the password, it would still need to be able to decrypt it locally upon authorized requests, so has all the problems of a classic DRM system.

Pro: No calls to external systems needed
Con: Passphrase or hash is stored on the system which uses it
Con: Requires a trusted keyring on the system

Retrieve the password from an SSH "key store" using SSH agent

The system holding the private key is the most trusted platform in this scenario. This platform could also host the password, and release it on an authorized SSH connection authenticated using its own private key. This key storing/serving SSH server could run on a non-standard port and be limited to just the one host and key and just return the required passphrase instead of opening a shell.

Encfs has a parameter to specify a command which returns the plaintext password on stdout. With a little patching and scripting, pam-encfs could be modified to make a call back to the originating server for the password.

Pro: Can be compatible with interactive logins by storing same password
Pro: Only requires a pam-encfs patch and a few scripts
Con: Password exists in plaintext on the key store
Con: Additional complexity, with no fallback on failure

Modify encfs to take raw key option, and retrieve from "key store"

Since the only thing the remote system needs the pass-phrase for is to generate the symmetric key, we could avoid exposing the pass-phrase entirely by storing that key instead of the pass-phrase. Then, we retrieve that key and apply it directly in order to mount the encrypted file-system.

Pro: Can be compatible with interactive logins by using same password
Pro: Password is not stored in any format, only its hash
Con: Likely also requires encfs patch to take raw symmetric key as an option
Con: Even more complexity, still no fallback on failure

Either of the above two, but with encrypted store
We could require two things for access to the plaintext pass-phrase/key: the encrypted pass-phrase/key from the keystore, and a symmetric key stored on the system which uses it. This would prevent a compromise of the originating machine from directly leading to the plaintext, at the cost of more complexity.

Pro: Pass-phrase/key not stored in plaintext
Con: Even more complexity

Modify PAM (conf?) to require password after public key

This may turn out to be the best solution. If the existing modules can be configured such that sshd authenticates via public key, but PAM still requires a password, then pam-encfs has what it needs to create the symmetric key, and all is well. If this is not a configuration issue, it may require a patch to one of the PAM modules, sshd, or login. It requires two passwords: one to unlock the private key (or login), and one to decrypt the file system.

Pro: 100% compatible with interactive logins
Pro: Password is not stored in any form, no less secure than interactive login
Pro: At best, a config option. At worst, one of the PAM modules may need a patch
Pro: Increased security, as a key AND a pass-phrase is required for remote access
Con: defeats SSO capabilities of SSH agent authentication

The user's solution

The solution that was implemented is to mount the file-system outside of PAM using the shell's login/logout scripts. This has a few potential problems, such as concurrency issues if the user connects more than once, and the inability to execute commands directly from the ssh client. However, it gives a uniform experience whether connecting remotely or at the console, and in fact allows different passwords for login and decryption.

Using an SSH agent for sudo authentication

I've discovered a great PAM module for using an SSH agent to authenticate the user for sudo access. I'm planning on installing it on all of my Linux servers.

To build in Ubuntu 8.04.4 LTS, you'll need to install a couple of development packages:

sudo apt-get install libpam0g-dev libssl-dev

Then download the source from
http://sourceforge.net/projects/pamsshagentauth/

The build process is your typical "./configure; make; sudo make install" but requires a little tweeking before install. So first run

./configure; make

The manpage is NROFF formatted, but installed as a text page. To overcome this, I changed both references in the Makefile from "cat" to "man" with my favorite editor.

Additionally, the PAM modules in Ubuntu are installed in /lib/security, whereas make installs by default to /usr/local/libexec. You can manually modify the libexecdir value in the makefile, or set the variable when calling make install, such as

sudo make libexecdir=/lib/security install

I used sudo checkinstall instead in order to build a simple binary .deb file for distributing among my servers. See here for details.

Once installed, change the line in /etc/pam.d/sudo from

@include common-auth

to

auth sufficient pam_ssh_agent_auth.so \
file=%h/.ssh/authorized_keys
auth requisite pam_unix.so nullok_secure

That allows PAM to use the SSH agent. Finally, in order to pass the agent socket to sudo, add the following line before the env_reset line in /etc/sudoers

Defaults env_keep += SSH_AUTH_SOCK

To test, you will have to be running an agent such as Pageant (part of the PuTTY tools) or ssh-agent from OpenSSH, and you will have to connect with the agent forwarding flag (-A). To verify you're reauthenticating instead of using cached sudo credentials, run

sudo -K

to clear the user timestamp. Then, run

sudo id

and check the /var/log/auth.log file for a line like

sudo[5678]: pam_ssh_agent_auth: Authenticated: `and1' as `and1' using /home/and1/.ssh/authorized_keys

Now, why did we do all of this?

Authenticating through an SSH agent means the password is never entered on the remote system. The authentication request is sent back to the originating computer running the agent. The request signed by the user's private key which is held by the agent, and then returned to the remote system to verify using the user's public key.

If the remote system is compromised, using this module will prevent the attacker from ever seeing your password while you access the system and escalate your privileges to admin.

If the PAM module is set to sufficient, it will fall back to prompting for a password. If you set it to required and remove the pam_unix.so line, it should then only allow agent authentication, much like setting "PasswordAuthentication no" in sshd_config. WARNING: I haven't verified this, so make sure you have an alternative means of gaining root access before you fiddle with it...

Once this module is in place, there is no reason to have a password on the remote account. You could set it to '*' or a long, random string and forget it. In fact, it also gives the ability to issue privileged commands remotely, such as

ssh -A myRemoteSystem sudo shutdown -r now

Note that you have to use an agent; specifying the identity on the ssh command line will not work, as the followup request from sudo will not be serviced. The -i command to ssh will only authenticate the original login request.

There is one caveat, true of all agent forwarding. If the remote system is compromised, the attacker will have access to your authentication socket as long as you're logged in. This means he would be able to connect to systems where that key is authorized using your agent forwarding socket.

You can mitigate this using OpenSSH by using the -c option to ssh-add when adding your private key. When used this way, your local system will popup a dialog asking you to authorize each use of the private key. I don't know of a way to do this with PuTTY without patching Pageant.