Recently I got a new job as a sysadmin in a company running Solaris 10 servers. I’m not really used to this OS, so I’ve ordered some books and read a ton of articles about it. As always, it’s quite a challenge to switch from one OS to another; you have to learn its way of doing things, its admin tools, its legacy…
Solaris 10 feels particularly old compared to modern Linux distros like Fedora or enterprise-class Linux distros like CentOS/RHEL, but there are also a couple of very cool technologies included. This blog post is probably the first of a series dedicated to Linux -> Solaris migrations, and how we can learn things from Solaris to improve Linux.
So, the first thing I want to talk about is the updates management.
In Fedora, we have yum/PackageKit to nicely handle the whole process: download the updates and install them. In Solaris 10, there is a jungle of mostly useless applications to download the updates; obtaining them is bit like being in hell, only worse. The most convenient way is to use an application called Patch Check Advanced. But once you have downloaded the updates for your system, you can apply them in a WAY cooler way than in Linux. Let me explain how it works…
First, you have to know that Solaris now comes with a file system called ZFS, which we can’t have in Linux for legal reasons as far as I know. This file system is very similar functionality-wise to Btrfs that we should see in Linux in a couple of months (oddly enough, both “belong” to Oracle now); ZFS supports something called “snapshots”.
Snapshots give you the opportunity to create an exact copy of any file system on the fly in a matter of seconds, without any performance degradation nor disk space use (best case). You can see it as a backup, it’s very similar to Apple’s Time Machine. To handle updates, Solaris 10 uses a set of utilities called “Live Upgrade”, what they do is:
- Create a snapshot (at the file system level, with ZFS) of you whole root partition
- Apply the patches you have downloaded to this snapshot
- Create an entry in GRUB to boot from this “new root” snapshot at the next reboot
The end result is that at the next reboot, you can see if your system still works well with the applied updates.
- It you are happy, you stay with this new root and delete the old one
- If you are unhappy, you reboot to the old root and your system is in its old state
Simply imagine how cool this is for critical servers, or even the end user. You don’t mess at all with the software currently installed, there is no risk as you can roll back and the only downtime you have is the reboot.
With Btrfs coming to Linux, we could absolutely do the same thing, and I’d like to see that. I’m just useless to make that happen in Fedora, but I can at least raise awareness on this with this blog post. I know that this requires a massive paradigm shift for the way we handle updates, we need to move from a “per application update” to a “complete OS state update”. Instead of selecting a specific kernel to boot in GRUB, you would select a system state.
On a side note, OpenSolaris’ package manager (similar to PackageKit) does exactly that.
Here is a complete Live Upgrade sequence I did on a Solaris 10 box
[root@sol10 /]$ lucreate -n s10Aug2209 (We create a / ZFS snapshot called s10Aug2209)
Checking GRUB menu…
Analyzing system configuration.
No name for current boot environment.
INFORMATION: The current boot environment is not named – assigning name <root-dataset>.
Current boot environment is named <root-dataset>.
Creating initial configuration for primary boot environment <root-dataset>.
The device </dev/dsk/c0d0s0> is not a root device for any boot environment; cannot get BE ID.
PBE configuration successful: PBE name <root-dataset> PBE Boot Device </dev/dsk/c0d0s0>.
Comparing source boot environment <root-dataset> file systems with the
file system(s) you specified for the new boot environment. Determining
which file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment <s10Aug2209>.
Source boot environment is <root-dataset>.
Creating boot environment <s10Aug2209>.
Cloning file systems from boot environment <root-dataset> to create boot environment <s10Aug2209>.
Creating snapshot for <rpool/ROOT/root-dataset> on <rpool/ROOT/root-dataset@s10Aug2209>.
Creating clone for <rpool/ROOT/root-dataset@s10Aug2209> on <rpool/ROOT/s10Aug2209>.
Setting canmount=noauto for </> in zone <global> on <rpool/ROOT/s10Aug2209>.
Creating snapshot for <rpool/ROOT/root-dataset/var> on <rpool/ROOT/root-dataset/var@s10Aug2209>.
Creating clone for <rpool/ROOT/root-dataset/var@s10Aug2209> on <rpool/ROOT/s10Aug2209/var>.
Setting canmount=noauto for </var> in zone <global> on <rpool/ROOT/s10Aug2209/var>.
Saving existing file </boot/grub/menu.lst> in top level dataset for BE <s10Aug2209> as <mount-point>//boot/grub/menu.lst.prev.
File </boot/grub/menu.lst> propagation successful
Copied GRUB menu from PBE to ABE
No entry for BE <s10Aug2209> in GRUB menu
Population of boot environment <s10Aug2209> successful.
Creation of boot environment <s10Aug2209> successful.
[root@sol10 /]$ zfs list (we look at the ZFS partition list, we can see that we have snapshots)
NAME USED AVAIL REFER MOUNTPOINT
rpool 10.4G 446G 38.5K /rpool
rpool/ROOT 1020M 446G 18K legacy
rpool/ROOT/root-dataset 1020M 446G 804M /
rpool/ROOT/root-dataset@s10Aug2209 64.5K - 804M -
rpool/ROOT/root-dataset/var 216M 446G 216M /var
rpool/ROOT/root-dataset/var@s10Aug2209 98K - 216M -
rpool/ROOT/s10Aug2209 92K 446G 804M /
rpool/ROOT/s10Aug2209/var 0 446G 216M /var
rpool/dump 4.88G 446G 4.88G -
rpool/export 48K 446G 19K /export
rpool/export/home 29K 446G 29K /export/home
rpool/swap 4G 450G 16K -Download the patches in /var/tmp/10_Recommended with Patch Check Advanced.
[root@sol10 /]$ luupgrade -n s10Aug2209 -s /var/tmp/10_Recommended -t `cat patch_order` (we apply the patches to the snapshot)
[root@sol10 /]$ luactivate s10Aug2209 (we activate the updated snapshot for the next boot)
Generating boot-sign, partition and slice information for PBE <root-dataset>
Saving existing file </etc/bootsign> in top level dataset for BE <root-dataset> as <mount-point>//etc/bootsign.prev.
A Live Upgrade Sync operation will be performed on startup of boot environment <s10Aug2209>.Generating boot-sign for ABE <s10Aug2209>
Saving existing file </etc/bootsign> in top level dataset for BE <s10Aug2209> as <mount-point>//etc/bootsign.prev.
Generating partition and slice information for ABE <s10Aug2209>
Copied boot menu from top level dataset.
Generating multiboot menu entries for PBE.
Generating multiboot menu entries for ABE.
Disabling splashimage
Re-enabling splashimage
No more bootadm entries. Deletion of bootadm entries is complete.
GRUB menu default setting is unaffected
Done eliding bootadm entriesModifying boot archive service
Propagating findroot GRUB for menu conversion.
File </etc/lu/installgrub.findroot> propagation successful
File </etc/lu/stage1.findroot> propagation successful
File </etc/lu/stage2.findroot> propagation successful
File </etc/lu/GRUB_capability> propagation successful
Deleting stale GRUB loader from all BEs.
File </etc/lu/installgrub.latest> deletion successful
File </etc/lu/stage1.latest> deletion successful
File </etc/lu/stage2.latest> deletion successful
Activation of boot environment <s10Aug2209> successful.[root@sol10 /]$ init 6 (we reboot)
propagating updated GRUB menu
Saving existing file </boot/grub/menu.lst> in top level dataset for BE <s10Aug2209> as <mount-point>//boot/grub/menu.lst.prev.
File </boot/grub/menu.lst> propagation successful
File </etc/lu/GRUB_backup_menu> propagation successful
File </etc/lu/menu.cksum> propagation successful
File </sbin/bootadm> propagation successful
[root@sol10 /]$ lustatus (we check the boot environments we have)
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
————————– ——– —— ——— —— ———-
root-dataset yes no no yes -
s10Aug2209 yes yes yes no -[root@sol10 /]$ ludelete root-dataset (we delete the old root)
System has findroot enabled GRUB
Checking if last BE on any disk…
BE <root-dataset> is not the last BE on any disk.
Updating GRUB menu default setting
Changing GRUB menu default setting to <0>
Saving existing file </boot/grub/menu.lst> in top level dataset for BE <s10Aug2209> as <mount-point>//boot/grub/menu.lst.prev.
File </etc/lu/GRUB_backup_menu> propagation successful
Successfully deleted entry from GRUB menu
Determining the devices to be marked free.
Updating boot environment configuration database.
Updating boot environment description database on all BEs.
Updating all boot environment configuration databases.
Boot environment <root-dataset> deleted.
Yes. BTRFS support too please.
You can already do this with LVM snapshots.
Snapshots yes, but can you apply the RPM updates to a root that isn’t in production, and then reboot to it to minimize downtime? I didn’t see this in the doc. The more global thing I wanted to point out is that OpenSolaris does this whole process transparently for the end user, THIS is the real gain instead of having to type a myriad of commands.
I’m not convinced I’m in favor of this idea. For something like a server, perhaps. But for a desktop, your view of the system is not one giant system that you receive updates to in bulk, but a bunch of apps that you use. Sometimes, upstream comes up with a new version of some program you use, and you’d like the next features. I shouldn’t have to reboot just because I’ve upgraded GIMP. I think, at least as far as the desktop goes, Linux needs to move in the other direction; decentralize package management, make distro-agnostic third-party packages first-class citizens, etc.
Sure, you pointed out the main drawback of this solution for the end user…you need a reboot for every update, and a file system that supports snapshots. On the other side, you are guaranteed to be able to roll back. I don’t say it’s perfect, it’s another vision on how to handle updates safely. In the current manner we handle updates in Fedora, there is absolutely no safety net.
Ultimately, I believe this will be an option, like Time Machine for Mac.
I wonder if it could be done as a yum plugin?
1. Every time yum starts a transaction, snapshot the state of the machine with LVM.
2. Create an interface that lists snapshots that’s accessible thru some yum command.
Maybe it’s time to read up on how LVM snapshots work…
Hm. Actually, this looks like it might be quite straightforward:
http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html
The big question is, how much room would the snapshot take up, and how many snapshots should you save? Just the last one? The last several? And how reliable is the restore?
Interesting stuff.
This is a question I don’t have an answer to…all I know is that with ZFS it’s a matter of 5-10 seconds to create a snapshot. I really lack an advanced LVM knowledge, but IMO we need Btrfs for that to be efficient. The key there is really how the FS works, with “copy-on-write” which guarantees reliability.
I’m quoting wikipedia “An advantage of copy-on-write is that when ZFS writes new data, the blocks containing the old data can be retained, allowing a snapshot version of the file system to be maintained. ZFS snapshots are created very quickly, since all the data composing the snapshot is already stored; they are also space efficient, since any unchanged data is shared among the file system and its snapshots.”
You should look at the way they do “Time Machine” in Open Solaris on http://blogs.sun.com/erwann/ (the screencast explains it very well).
The Gnome integration bits are already there, the rest depends on the file system.
I have not actually used ZFS, it does sound very cool. Does it have a gui management interface?
Actually it even has a web-based management GUI, but these are Sun’s tools. With Btrfs, we should see the same functionality in Linux.
lvm snapshot merging would make the above very similar in linux. unfortunately, i really haven’t heard anything more about it than when the patches were posted like last year i believe.