- Fixed some memleaks in libxbps and its clients, found by valgrind.
Tuesday, December 28, 2010
XBPS 0.7.2 released
A new version has been released with the following changes:
Thursday, December 23, 2010
XBPS 0.7.1 released
A new version has been released with the following changes:
- xbps-dgraph: fixed small build error on x86 shown in buildbot tests.
- libxbps: xbps_repository_pool_init(): don't loop forever if repository pkg-index.plist files cannot be internalized.
- libxbps: xbps_fetch_file(): set a 30 seconds timeout for transfers.
- xbps-repo(8): fixed the 'genindex' target for the API changes in 0.7.0.
- xbps-uhelper: fixed the 'unregister' target for the API changes in 0.7.0.
XBPS: VirtualBox 4.0.0 + DKMS
I've spent the past two days in packaging VirtualBox 4.0.0, that was released just yesterday. The problem is that VirtualBox uses some kernel modules for the hosts and guests and I didn't want really to create a package with the binary modules hardcoding a specific kernel package version.
What's the solution to this? DKMS. DKMS stands for Dynamic Kernel Modules Support, and is being used in Debian and RedHat derivatives. Why DKMS is useful? because it will rebuild/install some external kernel modules for all kernels available on the system or just a specific one. Let's show a real example:
My machine is currently running the Linux kernel 2.6.37-rc7, if DKMS is not used, I'd have to force the VirtualBox package that comes with the kernel modules to depend on a specific kernel version, the same that was used to build them. Actually the XBPS distribution uses the latest stable kernel, 2.6.36.2, so the XBPS package with the VirtualBox modules wouldn't work on my system. DKMS solves this by building/installing those external modules in the specific kernel version that you specify to it.
Hope you understood why DKMS is useful, and because those "foo-modules-version" packages in most distros are really not very useful, and becase there are tons of them.
VirtualBox 4.0.0 is really awesome, I'm impressed how fast it is compared to QEMU (without KVM). It's really working well on my XBPS x86_64 distro. You can find VirtualBox in the "xbps-templates" mercurial repository on the XBPS project site. Below are two screenshots of my system running VirtualBox.
DKMS: http://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
VirtualBox: http://www.virtualbox.org/
What's the solution to this? DKMS. DKMS stands for Dynamic Kernel Modules Support, and is being used in Debian and RedHat derivatives. Why DKMS is useful? because it will rebuild/install some external kernel modules for all kernels available on the system or just a specific one. Let's show a real example:
My machine is currently running the Linux kernel 2.6.37-rc7, if DKMS is not used, I'd have to force the VirtualBox package that comes with the kernel modules to depend on a specific kernel version, the same that was used to build them. Actually the XBPS distribution uses the latest stable kernel, 2.6.36.2, so the XBPS package with the VirtualBox modules wouldn't work on my system. DKMS solves this by building/installing those external modules in the specific kernel version that you specify to it.
Hope you understood why DKMS is useful, and because those "foo-modules-version" packages in most distros are really not very useful, and becase there are tons of them.
VirtualBox 4.0.0 is really awesome, I'm impressed how fast it is compared to QEMU (without KVM). It's really working well on my XBPS x86_64 distro. You can find VirtualBox in the "xbps-templates" mercurial repository on the XBPS project site. Below are two screenshots of my system running VirtualBox.
DKMS: http://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
VirtualBox: http://www.virtualbox.org/
Sunday, December 19, 2010
XBPS: XBMC 10.0 available!
So a few hours ago I've just updated the xbmc package to 10.0. Some days ago I was looking for a free media center that is able to stream content to my XBOX 360 and found XBMC. What can I say about it? it's simply amazing, this version is able to install multiple add-ons to improve its capabilities, i.e for viewing youtube videos, listening streams over shoutcast, etc.
Below are two screenshots of my desktop system running XBMC 10.0 and GNOME 2.32.1. Both available from XBPS in binary packages and sources in the xbps-templates repo. Enjoy!
Below are two screenshots of my desktop system running XBMC 10.0 and GNOME 2.32.1. Both available from XBPS in binary packages and sources in the xbps-templates repo. Enjoy!
Wednesday, December 15, 2010
XBPS 0.7.0 released
A new version has been released with the following changes:
- xbps-repo(8): implemented the 'find-files' target to print which packages own a file, link, directory, etc. Analogue to the one implemented in xbps-bin(8). Works for local and remote repository, fetching info on-the-fly.
- Fixed a typo in the configure script that broke detection of strlcat(), reported and patch provided by Min Sik Kim in issue 2.
- xbps-bin(8): added -F flag for the remove target. Unless it's set, packages that are dependencies of other installed packages won't be removed. This flag overrides this behaviour and forces the package removal.
- xbps-{bin,repo}(8): indent uniformly all lines while listing packages in xbps-bin or while searching for packages in xbps-repo.
- xbps-repo(8): while searching for packages also try to find strings in the short description object.
- xbps-{bin,repo}(8): added -d arg to enable debugging output to stderr.
- While updating packages, obsolete directories are also removed when necessary. This helped to remove some empty dirs for python 2.6 > 2.7 update, just to mention a real example.
- Cleaned up the API. Still not finished and there's a long way to have this finished and ready for public consumption.
- xbps-dgraph: new utility to generate graphviz' dot(1) graphs for package metadata properties, such as dependencies, reverse dependencies, etc. Extracts the info from installed package metadata plist files.
- Performance improvements in libxbps and all utilities, by avoiding unnecessary access(2) and chdir(2) calls while executing the INSTALL/REMOVE scripts at pre/post (de)install time.
- Fixed some memleaks on libxbps found while working on the xbps-dgraph utility.
Monday, November 8, 2010
XBPS: creating dot(1) graphs for metadata properties on installed pkgs
For the past days (well, only Sunday to be exact) I've been coding an application to automate the generation of dot(1) graphs for installed packages with XBPS, exactly for package's metadata properties; those such as dependencies, reverse dependencies, configuration files, etc. The code is 500 LOC and generates customized dot files through properties set in a configuration plist file. I'm sorry if you don't like XML, I don't care :-)
Basically the steps you have to do are the following:
$ xbps-dgraph -g [to generate the config file]
$ xbps-dgraph pkgname [to generate the .dot file]
$ dot -Tpng pkgname -o pkgname.png [to generate the png image]
You can also add/change/remove properties for graph, edges and nodes through the configuration file (xbps-dgraph.conf by default).
The following are some graphs from packages installed on my system, please note that some packages will genererate a really LARGE image if it contains many nodes. I hope you enjoy the graphs as I do; xbps-dgraph will appear in the next upcoming XBPS release... perhaps in a few days :-)
Basically the steps you have to do are the following:
$ xbps-dgraph -g [to generate the config file]
$ xbps-dgraph pkgname [to generate the .dot file]
$ dot -Tpng pkgname -o pkgname.png [to generate the png image]
You can also add/change/remove properties for graph, edges and nodes through the configuration file (xbps-dgraph.conf by default).
The following are some graphs from packages installed on my system, please note that some packages will genererate a really LARGE image if it contains many nodes. I hope you enjoy the graphs as I do; xbps-dgraph will appear in the next upcoming XBPS release... perhaps in a few days :-)
Sunday, October 31, 2010
XBPS 0.6.2 released!
A new version of the X Binary Package System, also known as XBPS has been released with the following changes:
- libxbps: xbps_repository_unregister(): in remote repositories, also remove the pkg index file and its directory.
- libxbps: xbps_get_pkg_index_plist(): fixed a memleak.
- xbps-bin(8): modify the 'list' target to accept an optional argument to list packages in the specified state, so that you can list packages that need to be purged, configured and broken (will be used in the future).
- xbps-bin(8): added a new target 'find-files' to find which installed package(s) own a file. Exact matches "/bin/mount" or patterns "/usr/lib/libb[ao]b\*" can be specified.
- When updating a package and removing obsolete files, don't forget to remove those directories if they were empty.
Wednesday, October 27, 2010
XBPS templates month news: October 2010
A lot of updates including Linux kernel to 2.6.36, GNOME 2.32, xorg 1.9.1, etc. Here's the full changelog for this month, it can be easily taken from Mercurlal repository with the following command:
$ hg log --template "{desc|firstline}\n\n" --date "2010-10-01 to 2010-10-31"
Summary:
$ hg log --template "{desc|firstline}\n\n" --date "2010-10-01 to 2010-10-31"
Summary:
- sylpheed: update to 3.0.3.
- squashfs-tools: update to 4.1.
- sed: use --without-included-regex.
- groff: fix issues with some UTF-8 manpages (from Arch), added -doc subpkg.
- texinfo: ship compressed info files, bumprev.
- fuse: remove static dev nodes, bumprev.
- sysvinit: update to 2.88.
- ConsoleKit: update to 0.4.2.
- glibc: update to 2.12.1.
- module-init-tools: update to 3.12.
- psmisc: update to 22.13.
- qemu: update to 0.13.0.
- libarchive: bumprev for xz -> liblzma update.
- Removed xz-devel symlink.
- xz: update to 5.0.0, provide liblzma{,-devel} subpkgs. Bump abi_depends.
- pam: update to 1.1.2.
- libsoup: move gobject-introspection stuff into a new subpkg, bumprev.
- udev: update to 164.
- avahi: update to 0.6.28.
- dconf: needs gio-modules trigger, move dconf bin into main pkg.
- Added dconf-0.5.1 build template.
- xbps-triggers: update to 0.4, improve gsettings-schemas trigger.
- gnome-bluetooth: needs gsettings-schemas trigger, bumprev.
- xbps-triggers: update to 0.3, added a gsettings-schemas trigger.
- pygtk: update to 2.22.0.
- pygobject: update to 2.26.0.
- ntfs-3g: update to 2010.10.2.
- fuse: update to 2.8.5.
- network-manager-applet: update to 0.8.1.999.
- NetworkManager: update to 0.8.1.999.
- gnome: update to 2.32.0, finally!
- gnome-bluetooth: update to 2.32.0, bump abi_depends.
- asciidoc: update to 8.6.2.
- kernel: modify kernel config one more time, bumprev.
- xrandr: update to 1.3.3.
- libXi: update to 1.3.2.
- xcb-util: needs libxcb-devel at build time.
- libX11: update to 1.3.99.902 (1.4.0 RC2).
- xf86-video-intel: update to 2.13.0.
- xf86-video-ati-git-snapshot: needs libudev.
- xorg-util-macros: update to 1.10.1.
- llvm: update to 2.8.
- xkeyboard-config: update to 2.0.
- libxml2: add a patch to fix segfaults with yelp and others, from Arch.
- NetworkManager: update to 0.8.1.
- gnome-panel: update to 2.30.0.2.
- gdm: update to 2.32.0.
- gedit: update to 2.30.4.
- eog: update to 2.32.0.
- gnome-netstatus: update to 2.28.2.
- gnome-system-monitor: update to 2.28.2.
- cairomm: require latest cairo that doesn't depend on glitz, bumprev.
- gnome-utils: update to 2.32.0.
- gnome-power-manager: update to 2.32.0.
- gnome-terminal: update to 2.32.0.
- vte: update to 0.26.0.
- epiphany: update to 2.30.6.
- webkit-gtk: update to 1.2.5.
- gnome-user-docs: update to 2.32.0.
- nautilus: update to 2.32.0.
- yelp: update to 2.30.2.
- gnome-themes: update to 2.32.0.
- gnome-media: update to 2.32.0.
- gnome-icon-theme: update to 2.31.0.
- gnome-control-center: update to 2.32.0.
- metacity: update to 2.30.3.
- zenity: update to 2.32.0.
- gnome-settings-daemon: update to 2.32.0.
- nss-devel: fix Cflags in nss.pc file, bumprev.
- libgnomekbd: update to 2.32.0.
- gnome-backgrounds: update to 2.32.0.
- gnome-applets: update to 2.32.0.
- gst-plugins-base: update to 0.10.30.
- libgtop: update to 2.28.2.
- gstreamer: update to 0.10.30.
- gucharmap: update to 2.32.0.
- gnome-panel: update to 2.32.0.
- GConf: added gir-gconf subpkg, bumprev to -devel subpkg.
- gtk+: added gir-gtk+ subpkg, bumprev to -devel subpkg.
- atk: update to 1.32.0.
- pango: update to 1.28.3.
- gnome-disk-utility: add patch missed in previous update.
- gdk-pixbuf: add gir-gdk-pixbuf subpkg, bumprev to -devel subpkg.
- libgnomecanvas: update to 2.30.2.
- libgnome: update to 2.32.0.
- gvfs: update to 1.6.4.
- gnome-disk-utility: update to 2.32.0.
- gnome-vfs: update to 2.24.4.
- libbonobo: update to 2.32.0.
- evolution-data-server: update to 2.32.0, bump abi_depends.
- mapping_shlib: added libgdata.
- Added libgdata-0.6.5 build template.
- libgweather: update to 2.30.3.
- libsoup: update to 2.32.0.
- libgnome-keyring: update to 2.32.0.
- gnome-desktop: update to 2.32.0.
- gobject-introspection: update to 0.9.10.
- gnome-menus: update to 2.30.4.
- libwnck: update to 2.30.5.
- librsvg: update to 2.32.0.
- adobe-flash-plugin: update to 10.1 preview 2.
- firefox: finish update to 3.6.11.
- xulrunner: finish update to 1.9.2.11.
- sqlite: update to 3.7.3.
- eject: added a patch to handle mountpoints with spaces, from Gentoo. bumprev.
- kernel-libc-headers: remove .install/..install.cmd files, install scsi/scsi.h. Bumprev.
- ifenslave: sync code with 2.6.36, bumprev.
- tzdata: update to 2010m.
- libarchive: update to 2.8.4.
- gnome-session: update to 2.32.0.
- GConf: update to 2.32.0.
- xorg-server: update to 1.9.1.
- glib: move some bins from -devel to main pkg, bumprev.
- polkit-gnome: bump api_depends.
- polkit-gnome: update to 0.99.
- polkit: update to 0.99, bump abi_depends.
- xbps: update to 0.6.1.
- gtk+: update to 2.22.0.
- xbps-triggers: update to 0.2.
- Added gdk-pixbuf-2.22.0 build template.
- dbus: update to 1.4.0.
- upower: update to 0.9.6.
- polkit: update to 0.98.
- lvm2: update to 2.02.74.
- openssh: update to 5.6p1.
- cronie: update to 1.4.6.
- logrotate: update to 3.7.9.
- sudo: update to 1.7.4p4.
- util-linux-ng: update to 2.18, added libmount{,-devel} subpkgs.
- man-pages: update to 3.29.
- grep: update to 2.7.
- rsyslog: update to 5.6.0.
- kernel: improve x86_64 options, remove kernel-manpages subpkg.
- Split kernel-manpages from the 'kernel' pkg, to speed up the process.
- dhcpcd: update to 5.2.8.
- coreutils: update to 8.6.
- busybox-initramfs: update to 1.17.3.
- make: add two patches from archlinux, bumprev.
- nss: update to 3.12.8.
- nspr: update to 4.8.6.
- Added talloc-2.0.1 build template.
- xf86-input-evdev: update to 2.5.0.
- xf86-video-ati: update to 6.13.2.
- xorg-server: update to 1.9.0, bump abi_depends.
- libpciaccess: update to 0.12.0.
- libdrm: update to 2.4.22.
- libXfont: update to 1.4.2.
- xproto: update to 7.0.18.
- libICE: update to 1.0.7.
- libfontenc: update to 1.1.0.
- freetype: update to 2.4.3.
- cairo: update to 1.10.0.
- pixman: update to 0.18.4.
- kernel-libc-headers: install drm headers, bumprev.
- kernel-libc-headers: update to 2.6.36.
- glib: update to 2.26.0.
- bzip2: update to 1.0.6.
- mercurial: update to 1.6.4.
- xbps-base-system: update to 0.32, do not depend on radeon-firmware.
- kernel: update to 2.6.36.
- wifi-firmware: update to 1.1: some deps now come from linux-firmware pkg.
- Added linux-firmware replacing some firmware packages.
- gcc: update to 4.5.1.
- mpc: bump revision due to mpfr ABI change.
- Backed out changeset a850a857fb8a. Human error...
- mpc: bump revision due to mpfr ABI change.
- mpfr: update to 3.0.0. Bump abi_depends.
- udev: update to 163.
- xbps-src: do not overwrite PATH, just append it to the new one.
- make: update to 3.82.
XBPS: upcoming features for 0.6.2
I've been working in some missing features for the next XBPS version, and here they are:
http://groups.google.com/group/xbps-commits/browse_thread/thread/cf450a3c6da4c09c
The second item adds code to find files owned by a package, it's well explained
here:
http://groups.google.com/group/xbps/browse_thread/thread/db5923dff1f42dac
Last item helps to remove obsolete and empty directories that aren't used anymore by any installed package.
Development won't stop and I'm looking for more ideas and features to be implemented for the next upcoming XBPS version, 0.6.2.
- xbps-bin(8): modify the 'list' target to accept an optional argument to list packages in the specified state, so that you can list packages that need to be purged, configured and broken (will be used in the future).
- xbps-bin(8): added a new target 'find-files' to find which installed package(s) own a file. Exact matches "/bin/mount" or patterns "/usr/lib/libb[ao]b\*" can be specified.
- When updating a package and removing obsolete files, don't forget to remove those directories if they were empty.
http://groups.google.com/group/xbps-commits/browse_thread/thread/cf450a3c6da4c09c
The second item adds code to find files owned by a package, it's well explained
here:
http://groups.google.com/group/xbps/browse_thread/thread/db5923dff1f42dac
Last item helps to remove obsolete and empty directories that aren't used anymore by any installed package.
Development won't stop and I'm looking for more ideas and features to be implemented for the next upcoming XBPS version, 0.6.2.
XBPS 0.6.1 released
A new version has just been released with the following changes:
XBPS 0.6.1 - Released October 23, 2010.
XBPS 0.6.1 - Released October 23, 2010.
- When replacing a package that is going to be updated in the transaction do not remove it, just overwrite its files and continue. The updated package will find that old files didn't match the SHA256 hash and will skip them. This solves the issue of new package updates requiring new dependencies with files that were previously stored in the old version. For example gtk+-2.20 containing gdk-pixbuf, and gtk+-2.22 requiring gdk-pixbuf externally.
Thursday, June 3, 2010
XBPS 0.6.0 released!
A new version has just been released with the following changes:
xbps-0.6.0 (2010-06-03):
xbps-0.6.0 (2010-06-03):
- Synced libfetch code with NetBSD pkgsrc, updated to 2.31.
- libxbps: improved xbps_get_file_hash() to mmap(2) rather than read(2) the file we have to process. With mmap'ed files the hash can be processed 5x faster than before, or even more in some cases.
- libxbps: switch to OpenSSL SHA256 implementation and remove the one previously used. It's faster and OpenSSL is required for libfetch so there is not point in using it.
- Added strlcpy() and strlcat() from OpenBSD. Use them if they weren't found by the configure script on the system.
- Added a configure script that emulates GNU autoconf but simplified, to simplify customization for build options, flags, etc.
Monday, May 17, 2010
XBPS week news: 11/05/2010 - 17/05/2010
This week there have been a lot of changes both in packages and xbps-src and its helpers.
Package upgrades:
xbps-src now also checks for required utilities in the host system before doing
any operation. That simply has been added to avoid errors from new or existing users while building the bootstrap with 'xbps-src bootstrap'.
Package upgrades:
- vim: updated to 239 patch level, added icons and .desktop file for gvim.
- initramfs-tools: merged in 0.94.4 from Debian, now hibernation works properly.
- xbps-casper: merged in 1.236 from Ubuntu, fixed some errors detecting filesystems.
- kernel: updated to 2.6.34.
- udev: updated to 154.
- xbps-mklive: updated to 0.3.3, multiple improvements.
- lame: updated to 3.98.4.
- mercurial: updated to 1.5.3.
- ntfs-3g: updated to 2010.3.6.
- pixman: updated to 0.18.2.
- lcms: updated to 1.19. 2.0 is ABI incompatible, it will have to wait.
- transmission: updated to 1.93.
- git: updated to 1.7.1.
- curl: updated to 7.20.1.
- qemu: updated to 0.12.4.
- xorg-server: updated to 1.8.1.
- upower: updated to 0.9.3.
- syslinux: updated to 3.86.
- gettext: updated to 0.18.
- libffi: updated to 3.0.9.
- gawk: updated to 3.1.8.
- gcc-multilib: updated to 4.5.0.
- glibc32: updated to 2.11.1.
xbps-src now also checks for required utilities in the host system before doing
any operation. That simply has been added to avoid errors from new or existing users while building the bootstrap with 'xbps-src bootstrap'.
Tuesday, May 11, 2010
XBPS 0.5.1 released!
Today I had to release the XBPS utilities version 0.5.1 to fix a bug that caused some errors in packages using 'epoch' in its version string. The release plan used in XBPS is a time based release, a new version every 2 months of development.
XBPS 0.5.1 - Released May 11, 2010
XBPS 0.5.1 - Released May 11, 2010
- Fixed build issues in RHEL5, Ubuntu and others distributions.
- libxbps: changed the epoch component in version strings to the ':' character to be differentiated from '-' in pkgname, and '_' in revision.
- xbps-bin(8): added a new target 'show-orphans' that lists all orphaned packages currently installed.
- Removed asciidoc build dependency, simply use the troff manpages.
Sunday, May 2, 2010
XBPS: project moved to googlecode
These days I've been working in packaging MoinMoin to create a wiki for XBPS, and two days ago that was accomplished. I created the wiki and after using it a few minutes I asked to myself: why do I have to host it? I don't currently own a domain, nor do I have a static IP and the last point, I don't want to have a server running all year.
So I started looking at the free alternatives, none convinced me; I started to look again at googlecode, and now they allow Mercurial repositories. They also have issue tracking, wiki and mailing lists for free. After some hours of work, I had ready the project, with documentation and howtos in the wiki, two mailing lists (one for only commits and another one for discussing user/development questions). I also moved the two repositories for XBPS: xbps utilities and xbps-src with the build templates to Mercurial.
Now it's all working and I'm really happy with the result, it is available at http://code.google.com/p/xbps
To fetch the xbps utilities repository you now do:
and to fetch the xbps-src with the build templates repository you do:
All commits to the repositories are sent to the xbps-commits group.
So I started looking at the free alternatives, none convinced me; I started to look again at googlecode, and now they allow Mercurial repositories. They also have issue tracking, wiki and mailing lists for free. After some hours of work, I had ready the project, with documentation and howtos in the wiki, two mailing lists (one for only commits and another one for discussing user/development questions). I also moved the two repositories for XBPS: xbps utilities and xbps-src with the build templates to Mercurial.
Now it's all working and I'm really happy with the result, it is available at http://code.google.com/p/xbps
To fetch the xbps utilities repository you now do:
$ hg clone http://xbps.googlecode.com/hg/
and to fetch the xbps-src with the build templates repository you do:
$ hg clone http://xbps-src.xbps.googlecode.com/hg/ xbps-templates
All commits to the repositories are sent to the xbps-commits group.
Friday, April 30, 2010
XBPS: version 0.5.0 released!
The XBPS 0.5.0 version has just been released. Release notes:
https://launchpad.net/xbps/head/0.5.0
- xbps-repo(8): fixed the 'search' target to also match patterns against its description, and not only from the package/version touple.
- Cleaned up the code by using the LLVM's clang static analyzer, found two possible NULL pointer dereferences in error branches and dead code.
- xbps-bin(8): added new flag '-p' for the 'remove' and 'autoremove'
targets, to also purge the package(s) after successful removal. - xbps-repo(8): print a warning when registering a repository that has been already added previously.
- Add proplib-0.4.1 source and use it in XBPS. This is to avoid
an external dependency, so that we depend on the features of the
internal library. This also means that proplib is not required anymore. - Added support to read/write gzip compressed plists by default, thanks to proplib-0.4 that gained new functionality.
https://launchpad.net/xbps/head/0.5.0
XBPS: LLVM/Clang, Apache packages
This is the list of recently added packages into the x86_64 official repository for XBPS:
Along with some apache required dependencies (apr, apr-util), and some GNOME 2.30 updates, are
all related changes from this week in the xbps-src git repository.
I'm working in packaging MoinMoin with Apache/mod_fastcgi that I plan to use to document XBPS related stuff really soon.
- llvm-2.7 (provides some subpkgs: clang, clang-devel, llvm-docs). The clang static analyzer helped me to found some warnings on XBPS and Portable Proplib.
- apache-2.2.15 with the prefork MPM. A -devel subpkg to build modules is also provided.
- apache-mpm-{worker,event}-2.2.15: Apache httpd binaries that provide different MPMs.
- mod_fastcgi-2.4.6 - Apache 2.x module for the FastCGI protocol.
Along with some apache required dependencies (apr, apr-util), and some GNOME 2.30 updates, are
all related changes from this week in the xbps-src git repository.
I'm working in packaging MoinMoin with Apache/mod_fastcgi that I plan to use to document XBPS related stuff really soon.
Wednesday, April 28, 2010
XBPS: xbps-src gains new functionality
For the past days I've been working in bringing multiple improvements to the build system: xbps-src. It is a POSIX shell script with some more shell components that will fetch, check and compile source distribution files, to finally make XBPS binary packages.
Basically xbps-src works like the ports system on FreeBSD/OpenBSD or pkgsrc from NetBSD; but rather than running a make(1) command in a directory, you run the xbps-src command in a specific directory. The build template files, that specify how to fetch and build a binary package are shell scripts that understand a few variables (I shall document this in the future).
The structure to build packages is explained in the "building packages from source" document here:
http://xbps.nopcode.org/doc/building-from-source.html. Previously (before the last set of changes) there was a small shell script that required superuser permissions to bind mount /sys, /proc, /dev and the xbps-src main directory; that changed until the other day where I started using POSIX.1e capabilities support from the Linux kernel. They are explained in detail on its manpage capabilities(7).
I made three small C programs to bind mount and umount the required filesystems/mountpoints and another one to chroot into the master directory (the directory to chroot). The two mount programs use the CAP_SYS_ADMIN capability, and the last one uses the CAP_SYS_CHROOT. The programs need to have set those capabilities through the use of setcap(8). I added some security checks to the chroot code to make sure that you can't change root to / or to any other directory that the user doesn't own. With these changes in place you don't need to be the superuser anymore, the only problem is that you have to use a system that supports POSIX.1e capabilities and a filesystem with Extended Attributes... I'm sorry NetBSD you won't be able to use xbps-src.
The other set of changes implements a stow-alike approach while installing the packages in the masterdir (chroot directory), and all package files are now symlinked from its destination directory (if possible). That saves a lot of required space and helps to catch file conflicts and other problems. The master directory is where all packages are built, and stowned. The package is installed into its destination directory (/pkg-destdir) and stowned (symlinked) into the masterdir to be able to resolve dependencies.
xbps-src allows you to unstow (remove its symlinked files from the masterdir, and unregister package from the package database), as well as stowning the package multiple times, the only requirement is that the package must be installed into its destdir (destination directory), acomplished with the 'xbps-src install-destdir' command.
You can see the source code for the three C small programs here (yes, they are in the public domain): chroot.c, mount.c, umount.c.
Some more improvements were made but I'll explain (perhaps) them in another post... enjoy it.
Basically xbps-src works like the ports system on FreeBSD/OpenBSD or pkgsrc from NetBSD; but rather than running a make(1) command in a directory, you run the xbps-src command in a specific directory. The build template files, that specify how to fetch and build a binary package are shell scripts that understand a few variables (I shall document this in the future).
The structure to build packages is explained in the "building packages from source" document here:
http://xbps.nopcode.org/doc/building-from-source.html. Previously (before the last set of changes) there was a small shell script that required superuser permissions to bind mount /sys, /proc, /dev and the xbps-src main directory; that changed until the other day where I started using POSIX.1e capabilities support from the Linux kernel. They are explained in detail on its manpage capabilities(7).
I made three small C programs to bind mount and umount the required filesystems/mountpoints and another one to chroot into the master directory (the directory to chroot). The two mount programs use the CAP_SYS_ADMIN capability, and the last one uses the CAP_SYS_CHROOT. The programs need to have set those capabilities through the use of setcap(8). I added some security checks to the chroot code to make sure that you can't change root to / or to any other directory that the user doesn't own. With these changes in place you don't need to be the superuser anymore, the only problem is that you have to use a system that supports POSIX.1e capabilities and a filesystem with Extended Attributes... I'm sorry NetBSD you won't be able to use xbps-src.
The other set of changes implements a stow-alike approach while installing the packages in the masterdir (chroot directory), and all package files are now symlinked from its destination directory (if possible). That saves a lot of required space and helps to catch file conflicts and other problems. The master directory is where all packages are built, and stowned. The package is installed into its destination directory (
[juan@nocturno ~]$ ls -l /storage/masterdir/
total 132
drwxr-xr-x 2 juan juan 4096 abr 28 01:29 bin
drwxr-xr-x 2 juan juan 4096 abr 26 21:52 boot
drwxr-xr-x 2 juan juan 4096 abr 26 05:29 dev
drwxr-xr-x 52 juan juan 4096 abr 28 00:45 etc
drwxr-xr-x 2 juan juan 4096 abr 21 15:26 home
drwxr-xr-x 11 juan juan 4096 abr 28 12:09 lib
lrwxrwxrwx 1 juan juan 3 abr 21 15:26 lib64 -> lib
drwxr-xr-x 2 juan juan 4096 abr 21 15:26 media
drwxr-xr-x 2 juan juan 4096 abr 21 15:26 mnt
drwxr-xr-x 2 juan juan 4096 abr 21 15:26 opt
drwxr-xr-x 4 juan juan 4096 abr 22 16:29 pkg-binpkgs
drwxr-xr-x 2 juan juan 4096 abr 28 12:09 pkg-builddir
drwxr-xr-x 1020 juan juan 40960 abr 28 12:09 pkg-destdir
drwxr-xr-x 2 juan juan 4096 abr 28 01:46 pkg-srcdistdir
drwxr-xr-x 2 juan juan 4096 abr 26 05:29 proc
drwxr-x--- 2 juan juan 4096 abr 21 15:26 root
drwxr-xr-x 2 juan juan 4096 abr 28 12:09 sbin
drwxr-xr-x 2 juan juan 4096 abr 26 05:29 sys
drwxrwxrwt 3 juan juan 12288 abr 28 12:09 tmp
drwxr-xr-x 3 juan juan 4096 abr 22 06:19 tools
drwxr-xr-x 10 juan juan 4096 abr 26 21:51 usr
drwxr-xr-x 14 juan juan 4096 abr 28 00:45 var
[juan@nocturno ~]$
[juan@nocturno ~]$ ls -l /storage/masterdir/pkg-destdir/dash-0.5.5.1/
total 28
drwxr-xr-x 2 juan juan 4096 abr 22 12:22 bin
-rw-r--r-- 1 juan juan 399 abr 22 12:22 files.plist
-rwxr-xr-x 1 juan juan 690 abr 22 12:22 INSTALL
-rw-r--r-- 1 juan juan 590 abr 22 12:22 props.plist
-rwxr-xr-x 1 juan juan 689 abr 22 12:22 REMOVE
drwxr-xr-x 3 juan juan 4096 abr 22 12:22 usr
drwxr-xr-x 3 juan juan 4096 abr 22 12:22 var
[juan@nocturno ~]$
[juan@nocturno ~]$ ls -l /storage/masterdir/bin/dash
lrwxrwxrwx 1 juan juan 34 abr 22 12:22 /storage/masterdir/bin/dash -> /pkg-destdir/dash-0.5.5.1/bin/dash
[juan@nocturno ~]$
xbps-src allows you to unstow (remove its symlinked files from the masterdir, and unregister package from the package database), as well as stowning the package multiple times, the only requirement is that the package must be installed into its destdir (destination directory), acomplished with the 'xbps-src install-destdir' command.
You can see the source code for the three C small programs here (yes, they are in the public domain): chroot.c, mount.c, umount.c.
Some more improvements were made but I'll explain (perhaps) them in another post... enjoy it.
XBPS Library API documentation
Two months ago I started working in documenting the XBPS library API with doxygen and graphviz. It took me 1 week of work to properly document the modules and create the graphs of the plist files and its dictionaries.
Yesterday I uploaded it to the XBPS server, and here are the results:
http://xbps.nopcode.org/doc/api/html/
Please note that the API is still not finished and I could change in the future, and probably adding more high level functions to acomplish specific tasks. In the meanwhile, I've updated the XBPS main page with notes, tables, etc. Enjoy it!
Yesterday I uploaded it to the XBPS server, and here are the results:
http://xbps.nopcode.org/doc/api/html/
Please note that the API is still not finished and I could change in the future, and probably adding more high level functions to acomplish specific tasks. In the meanwhile, I've updated the XBPS main page with notes, tables, etc. Enjoy it!
Tuesday, April 20, 2010
Portable proplib: rw support for gzipped plist files
Today I released version 0.4.1 of the Portable proplib implementation, that is used extensively in XBPS for package metadata, package databases and repository metadata. This new version implements support for reading and writting from/to compressed gzip files with zlib.
The ABI remains compatible and the API has been modified with the
following functions for this task:
prop_{array,dictionary}_internalize_from_zfile():
Internalizes a compressed or uncompressed plist file.
prop_{array,dictionary}_externalize_to_zfile():
Externalizes an array/dictionary into a compressed gzip plist file.
The _zfile variants of these functions support gzip and uncompressed files, so you can safely always use the them and support both types of files.
The following source code in C internalizes a plist file (compressed or uncompressed),
externalizes the result into a compressed file (if argv[1] == 'ext') and prints
the number of objects in the dictionary. If argv[1] == 'int' it's not externalized.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <prop/proplib.h>
int main(int argc, char **argv)
{
char path[256];
prop_dictionary_t d;
if (argc != 3)
exit(EXIT_FAILURE);
/* Externalize a plist file into a gzipped file */
if (strcmp(argv[1], "ext") == 0) {
d = prop_dictionary_internalize_from_zfile(argv[2]);
snprintf(path, sizeof(path), "%s.gz", argv[2]);
prop_dictionary_externalize_to_zfile(d, path);
/* Internalize a gzip plist file */
} else if (strcmp(argv[1], "int") == 0) {
d = prop_dictionary_internalize_from_zfile(argv[2]);
fprintf(stderr, "%s", prop_dictionary_externalize(d));
}
fprintf(stdout, "Total objs in dictionary: %u\n",
prop_dictionary_count(d));
exit(EXIT_SUCCESS);
}
I got two files, one compressed and another one uncompressed with the same
dictionary (and data) on them:
[juan@nocturno test]$ file pkgidx.plist*
pkgidx.plist: XML document text
pkgidx.plist.gz: gzip compressed data, from Unix, max compression
[juan@nocturno test]$
Now running the example code and redirecting stderr output to /dev/null
(which is too large to show) is able to read the data automagically:
[juan@nocturno test]$ time ./a.out int pkgidx.plist.gz 2>/dev/null
Total objs in dictionary: 3
real 0m0.041s
user 0m0.038s
sys 0m0.002s
[juan@nocturno test]$ time ./a.out int pkgidx.plist 2>/dev/null
Total objs in dictionary: 3
real 0m0.033s
user 0m0.029s
sys 0m0.003s
[juan@nocturno test]$
We can use the 'ext' argument to the example code to magically convert
a non compressed plist file into a compressed gzip file, like:
[juan@nocturno test]$ rm pkgidx.plist.gz
[juan@nocturno test]$ time ./a.out ext pkgidx.plist 2>/dev/null
Total objs in dictionary: 3
real 0m0.111s
user 0m0.096s
sys 0m0.005s
[juan@nocturno test]$
[juan@nocturno test]$ ls -l pkgidx.plist*
-rw-r--r-- 1 juan juan 1009406 abr 20 15:22 pkgidx.plist
-rw-r--r-- 1 juan juan 130546 abr 20 15:49 pkgidx.plist.gz
[juan@nocturno test]$
The portable proplib implementation, maintained by myself, is available from
http://code.google.com/p/portableproplib and will be used in the upcoming XBPS version 0.5.
Saturday, April 17, 2010
XBPS: bootchart with GNOME
So here are the results of running bootchart in a GNU/Linux system built with XBPS, on a GNOME desktop system. Please note that this can be improved vastly if I change ntpd, and other services to be started later, but I don't really care... the chart is only here to compare with other distributions (hi Ubuntu!).
The results indicate that it took 22 seconds to start (even with the password prompt) to be in the user desktop system; probably next time I should enable autologin and compare. Really, 22 seconds is fucking great for a system built for home use!
The results indicate that it took 22 seconds to start (even with the password prompt) to be in the user desktop system; probably next time I should enable autologin and compare. Really, 22 seconds is fucking great for a system built for home use!
Thursday, April 15, 2010
XBPS: working in GNOME 2.30, xorg-server-1.8.0 ready...
Long time without posting to the blog, but really I have stuff to do everyday much like every other person in the world. So much stuff happened over this time that I cannot remember, but briefly, XBPS is currently at version 0.4.1 and the number of available packages and quality of the build system has improved vastly.
Long ago I added support to build packages in a chroot (with xbps-src) without root privileges; this works thanks to Linux supporting POSIX File capabilities. You still need root permissions to be able to bind (or null in BSD world) mount some directories. This is the only step requiring the permission and it's all encapsulated in a small shell script.
I've been working for the past days in bringing GNOME 2.30 to XBPS, there are zillions of dependencies as you might already know, but it's really progressing. The number of packages in XBPS has increased vastly for these months:
Every template file results in a binary package. Some highlights include latest Xorg (with the xorg-server-1.8.0 that doesn't need HAL and uses UDEV), latest stable and main (HEAD in BSD world) kernel, latest XFCE, latest GNOME (a minimal desktop, not fully finished), etc.
I wanted to show everyone some mandatory screenshots of GNOME running in XBPS (really I should give the distribution a name, but every time I think about it I'm out of ideas)... enjoy.
Long ago I added support to build packages in a chroot (with xbps-src) without root privileges; this works thanks to Linux supporting POSIX File capabilities. You still need root permissions to be able to bind (or null in BSD world) mount some directories. This is the only step requiring the permission and it's all encapsulated in a small shell script.
I've been working for the past days in bringing GNOME 2.30 to XBPS, there are zillions of dependencies as you might already know, but it's really progressing. The number of packages in XBPS has increased vastly for these months:
[juan@nocturno srcpkgs]$ find . -type f -name \*template|wc -l
1121
[juan@nocturno srcpkgs]$
Every template file results in a binary package. Some highlights include latest Xorg (with the xorg-server-1.8.0 that doesn't need HAL and uses UDEV), latest stable and main (HEAD in BSD world) kernel, latest XFCE, latest GNOME (a minimal desktop, not fully finished), etc.
I wanted to show everyone some mandatory screenshots of GNOME running in XBPS (really I should give the distribution a name, but every time I think about it I'm out of ideas)... enjoy.
Subscribe to:
Posts (Atom)