Some Speed Numbers

A major reason for freebsd-rustdate in the first place is that freebsd-update is slow. How slow, and how much faster can we get?

Spoiler alert: freebsd-rustdate wins, mostly by a whole lot. There are some reasons why. It’s not a fair fight.

These tests all start from a clean 13.2-RELEASE. See at the bottom for some details on the system and setup.

Summary table

Here’s a quick summary of the simple warm tests; see below for details. install -s is freebsd-rustdate’s install with fsync() disabled. -j enables multithreaded install. Times in min:sec.

Now, more detail, plus some extras.

All Warm

These tests are done with all caches warm and an already full files/ dir. That means there’s nothing that needs to be fetched from the server, and no local files that need to be stashed; that’s all already done and waiting. So this is a best-case for speed on both sides.

fetch updates

This goes from 13.2-RELEASE to 13.2-RELEASE-p12, touching ca. 286 paths.

For freebsd-update:

• fetch takes 13 seconds.
• install takes 10 seconds.

For freebsd-rustdate:

• fetch takes 0.4 seconds.
• install takes 7 seconds.
• install -s (no fsync) takes 6.8 seconds.

Note that over 6 of those seconds for both are taken up by the “post-world” steps (certctl rehash, etc). So the purely freebsd-update timings are more like 3.5 seconds for freebsd-update, and 0.5/0.3 for freebsd-rustdate.

upgrade 13.2 -> 14.0

From 13.2-RELEASE to 14.0-RELEASE-p10, touching around 24k paths. Note that for freebsd-rustdate we’re now using install -a to run through all 3 steps straight through.

freebsd-update:

• -r 14.0-RELEASE upgrade takes 1:50.
• install takes 3:43.
• These numbers are for 14.0-RELEASE-p9; there’s less than 1% difference in the number of touched files, so I’m not bothering to re-run it.

freebsd-rustdate:

• upgrade -r 14.0-RELEASE takes 2.5 seconds.
• install -a takes 23 seconds.
• install -as takes 12 seconds.
• install -aj takes 12 seconds.
• install -asj takes 9.5 seconds.

Here, with the much larger number of files touched, we can see the speed gains from avoiding all the fsync() calls. Running with -j effectively hides the fsync() cost without disabling it, and using both can chop another couple seconds off.

As mentioned above, remember we’re spending north of 6 seconds on post-install stuff like certctl, so freebsd-rustdate’s times are more like 17/6/6/3.5; -j or -s cut the time nearly in third, and combining them cuts it almost in half from there.

upgrade 13.2 -> 14.0 (with src)

OK, let’s get serious now. Add /usr/src into the mix. Again, going from 13.2-RELEASE to 14.0-RELEASE-p10, but now we have to touch the filesystem a lot more. North of 90k paths.

freebsd-update:

• -r 14.0-RELEASE upgrade takes a painful 5:51.
• install is a staggering 24:38.
• As above, these numbers are for 14.0-RELEASE-p9; it’s close enough.

freebsd-rustdate:

• upgrade -r 14.0-RELEASE takes 5 seconds.
• install -a takes 1:41.
• install -as takes 29 seconds.
• install -aj takes 34 seconds.
• -j12 install -aj takes 25 seconds.
• install -asj takes 19 seconds.

More files, more fsync() slowdown.

We can see that enabling parallel installation with -j does a good job of hiding the sync penalty in the parallelism; it’s within a few seconds of the no-sync variant even though it’s doing fsync()’s. Doubling the number of jobs makes a pretty small improvement to the total.

Combining parallelism and no-fsync() (unsurprisingly) gives the best speedup; the upgrade and install together add up to less than 30 seconds; you could do 14 complete system upgrade/install runs with freebsd-rustdate in the time it took freebsd-update to do one upgrade without even installing.

From a UX perspective too, freebsd-update mostly just leaves the user sitting there waiting and hoping it’s nearly done, while with freebsd-rustdate you have a progress bar giving you a sense of how far it has to go.

Here’s an example of the show-install output for this upgrade from freebsd-rustdate:

Summary of pending upgrade from 13.2-RELEASE to 14.0-RELEASE-p10
    (use `freebsd-rustdate show-install -v` to show full details)
 16460 files to add.
 6708 files to remove.
 70315 files to update.
 43 paths with changed type.
 No merged files.
 No conflicts to resolve.

Release vs. debug builds

If there’s one thing anybody who’s ever mentioned benchmarking rust code publically knows, it’s that you test in release mode. How much difference does that make here?

One answer is slightly less than it should. 2 of the most expensive things freebsd-update does is de/compressing files, and doing SHA256 hashes. For freebsd-rustdate, I’ve configured the crates involved in those actions (sha2, flate2, miniz-oxide) to be built with full optimizations even when building freebsd-rustdate in debug mode. I’m not developing them, so I don’t care about the details of them, or make changes and have to rebuild often. And those operations can eat a lot of time if unoptimized, so it slowed down working on the code. So, this is somewhat less than a full debug build. But everything else (particularly freebsd-rustdate itself) is still debug-y.

For comparison, I’ll redo the 13.2 -> 14.0 upgrade with source. And I’ll just do the no-fsync variant, to maximize the time spend in freebsd-rustdate rather than waiting on the system.

Compared to the release-build freebsd-rustdate numbers above:

• upgrade -r 14.0-RELEASE takes 3.3 seconds.
• install -as takes 28 seconds.

The debug build looks like:

• upgrade -r 14.0-RELEASE takes about 10 seconds.
• install -as takes 41 seconds.

If we do it without the src/ tree around, so we have a much smaller set of files to deal with,

• upgrade goes from the earlier 1.2 seconds to about 4.
• install -as goes from 11 seconds to 16.

So, doing --release builds definitely yields performance differences. However, it’s not gigantic. And even debug freebsd-rustdate builds are night and day faster than freebsd-update.

As another data point, running show-install requires parsing out the saved state, which is notable slower in non-release builds. A release build show-install of the with-src upgrade (as pasted above) takes about half a second; a debug build takes close to 2.5.

Empty files/ Directory

How about if I empty out the files/ dir in between runs? This means files have to be re-fetched from the server (on localhost) or re-stashed from the local filesystem. That involves some de/compression and hashing, plus bspatch invocations, etc. Let’s see what we can do.

fetch on 13.2 (no src)

Going from 13.2-RELEASE to 13.2-RELEASE-p12

• freebsd-update: 28 seconds.
• freebsd-rustdate: 4.5 seconds.

For comparision, remember that with files/ already populated above, freebsd-update took 13 seconds, and freebsd-rustdate 0.4. So the process of fetching down the files took freebsd-update about 15 seconds, and freebsd-rustdate about 4.

Context & Notes

These trials are done on my workstation, which is a reasonably powerful few-year-old machine. It has more cores than freebsd-rustdate will be trying to use (6), many gigabytes of free RAM, the filesystems the upgrade is working on are all on PCIe NVME drives, and all caches are warm so the drive speed doesn’t much matter for reads anyway. The system is otherwise quiet, though it is sitting on the normal desktop. The freebsd-update server is an Apache mod_cache running on localhost.

I’m starting out with a clean 13.2-RELEASE base/kernel untar’d in the dir.

freebsd-update numbers sometimes take a little doing, since I have to bypass or try to speed through the interactive bits. Minor tweaks were made to the script to facilitate e.g. doing all 3 install steps at once and bypassing interaction, as well as letting me run it as a regular user.

Note also that I’m running freebsd-rustdate via cargo run (with --release of course; see toward the end on that), not via directly running the built binary. That adds a little more overhead since it’s checking things are up-to-date etc before thunking through, but a little testing shows that’s easily < 0.1 seconds of overhead.

Of course, on a slow system with few CPUs and a cold cache, these times end up much longer. This site is currently running on a tiny single-core VPS with fairly slow everything. It took nearly 9 minutes to build freebsd-rustdate. Doing an upgrade from 14.0 to 14.1 took 2:10. The install took 0:05 (kernel) and 0:44 (world). I don’t have times lying around for the last freebsd-update, but it was certainly a lot longer.

These numbers are taken as of freebsd-rustdate 0.7.0 with rustc 1.81.