Compressing distribution tarballs for release with xz

[Mike Massonnet]

2010/8/9 Joshua Saddler <nightmorph at gentoo.org>:
> On Mon, 09 Aug 2010 15:16:43 +0200
> Yves-Alexis Perez <corsac at debian.org> wrote:
>
>> On lun., 2010-08-09 at 14:09 +0200, Nick Schermer wrote:
>> > That doesn't really matter I think, my whole point is: are our
>> > packages large enough to get a *real* benefit from xz, because not
>> > much is downloaded from the Xfce servers (ie. compared to
>> > distributions, who really benefit from xz compression).
>>
>> TBH, when Xfce only provided .bz2 files and Debian didn't accept them,
>> we retarballized everything in .tar.gz. Now that dpkg/dak and the whole
>> Debian infrastructure accepts .bz2, we're fine with that, but it won't
>> accept .orig.tar.xz right now, so we'll have to repackage that.
>>
>> Ubuntu had the same problem with Launchpad, I think it accepts bz2 now,
>> not sure about xz. Not sure either for Red Hat/Fedora, Suse etc.
>> Basically I think it's still a bit soon for distros, so they shouldn't
>> be the main argument for changing, at least.
>
> We don't have a problem with xz archives in Gentoo.

It works fine since a few months on Archlinux. That's a perfect case
where it makes sense to decrease the bandwidth.

lftp archive.xfce.org:/xfce/4.6.2/fat_tarballs> get xfce-4.6.2.tar.bz2
23687449 octets transférés en 58 secondes (397.5Ko/s)

This would have been a 38 seconds if it was compressed with xz
actually. This may be the only point why we should care.

	75KB/s
100MB	1365s
75MB	1024s (-5,6min)
20MB	273s
15MB	232s (-41s)
5MB	68s
3MB	40s (-28s)
	
	400KB/s
100MB	256s
75MB	192s (-64s)
20MB	51s
15MB	43s (-8s)
5MB	12s
3MB	7s (-5s)
	
	800KB/s
100MB	128s
75MB	96s (-32s)
20MB	25s
15MB	21s (-4s)
5MB	6s
3MB	3s (-3s)

As you can see, there is a real benefit.

> What about decompression times for end users?

xz is said to be faster (and it is proven) than bzip2 for
decompressing (a lot faster actually). For the rest I wouldn't care
this since we make twice a release in every decade (heck, even if we
would make the tarball every day it will be fine).

> Are the differences between xz and bz2 negligable?

Here are some unprofessional benchmarks:

* Decompressing

  $  time tar xf xfce-4.6.2.tar.bz2
real    0m10.685s
Mem: 4MB

  $  time tar xf xfce-4.6.2.tar.xz
real    0m0.707s  <--  yes, this is a _freaky_ _one_ _second_ (no
wait, shorter than a second)
Mem: 8MB

* Compressing

  $  time tar cfj xfce-4.6.2.tar.bz2 src
real    0m46.042s
Mem: 6MB

  $  time tar cfJ xfce-4.6.2.tar.xz src-xz/
real    0m29.071s  <--  faster!
Mem: 90MB  <--  that's the huge difference between both formats



Cheers
-- 
Mike