Compressing distribution tarballs for release with xz
mmassonnet at gmail.com
Mon Aug 9 17:40:54 CEST 2010
2010/8/9 Joshua Saddler <nightmorph at gentoo.org>:
> On Mon, 09 Aug 2010 15:16:43 +0200
> Yves-Alexis Perez <corsac at debian.org> wrote:
>> On lun., 2010-08-09 at 14:09 +0200, Nick Schermer wrote:
>> > That doesn't really matter I think, my whole point is: are our
>> > packages large enough to get a *real* benefit from xz, because not
>> > much is downloaded from the Xfce servers (ie. compared to
>> > distributions, who really benefit from xz compression).
>> TBH, when Xfce only provided .bz2 files and Debian didn't accept them,
>> we retarballized everything in .tar.gz. Now that dpkg/dak and the whole
>> Debian infrastructure accepts .bz2, we're fine with that, but it won't
>> accept .orig.tar.xz right now, so we'll have to repackage that.
>> Ubuntu had the same problem with Launchpad, I think it accepts bz2 now,
>> not sure about xz. Not sure either for Red Hat/Fedora, Suse etc.
>> Basically I think it's still a bit soon for distros, so they shouldn't
>> be the main argument for changing, at least.
> We don't have a problem with xz archives in Gentoo.
It works fine since a few months on Archlinux. That's a perfect case
where it makes sense to decrease the bandwidth.
lftp archive.xfce.org:/xfce/4.6.2/fat_tarballs> get xfce-4.6.2.tar.bz2
23687449 octets transférés en 58 secondes (397.5Ko/s)
This would have been a 38 seconds if it was compressed with xz
actually. This may be the only point why we should care.
75MB 1024s (-5,6min)
15MB 232s (-41s)
3MB 40s (-28s)
75MB 192s (-64s)
15MB 43s (-8s)
3MB 7s (-5s)
75MB 96s (-32s)
15MB 21s (-4s)
3MB 3s (-3s)
As you can see, there is a real benefit.
> What about decompression times for end users?
xz is said to be faster (and it is proven) than bzip2 for
decompressing (a lot faster actually). For the rest I wouldn't care
this since we make twice a release in every decade (heck, even if we
would make the tarball every day it will be fine).
> Are the differences between xz and bz2 negligable?
Here are some unprofessional benchmarks:
$ time tar xf xfce-4.6.2.tar.bz2
$ time tar xf xfce-4.6.2.tar.xz
real 0m0.707s <-- yes, this is a _freaky_ _one_ _second_ (no
wait, shorter than a second)
$ time tar cfj xfce-4.6.2.tar.bz2 src
$ time tar cfJ xfce-4.6.2.tar.xz src-xz/
real 0m29.071s <-- faster!
Mem: 90MB <-- that's the huge difference between both formats
More information about the Xfce4-dev