Portable proplib 0.5.1 released

A new version was released with the following changes:

• Added pkg-config support and install a proplib.pc file.
• Merged stuff from old subversion repo, that re-added the change to emit base 10 rather than base 16 for unsigned numbers contributed by Adam Hoka.

Please see http://code.google.com/p/portableproplib for more information.

Portable proplib 0.5.0 released!

I've just released a new version of the portable proplib implementation: 0.5.0. This version has the following changes:

• Synced proplib code with NetBSD HEAD, which replaces the red-black tree implementation and checks for NULL pointers passed in to the _dict/_array functions.
• The _zfile functions in the API are now encapsulated into its own file, that way upstream changes are merged easily.
• The list of public API exported symbols is now handled via libtool's -exported-symbols option so that local symbols don't pollute the API.
• Explicitly require a C99 compiler. Don't want to waste time in non C99 compilers.
• Build with Stack Smashing Protection if supported by the compiler, and a bunch of compiler warnings.

Next stable XBPS version will link to the external dependencies for proplib and libfetch by default just to avoid namespace pollution in libxbps. Enjoy!

Portable proplib: rw support for gzipped plist files

Today I released version 0.4.1 of the Portable proplib implementation, that is used extensively in XBPS for package metadata, package databases and repository metadata. This new version implements support for reading and writting from/to compressed gzip files with zlib.
The ABI remains compatible and the API has been modified with the
following functions for this task:


prop_{array,dictionary}_internalize_from_zfile():
Internalizes a compressed or uncompressed plist file.


prop_{array,dictionary}_externalize_to_zfile():
Externalizes an array/dictionary into a compressed gzip plist file.


The _zfile variants of these functions support gzip and uncompressed files, so you can safely always use the them and support both types of files.

The following source code in C internalizes a plist file (compressed or uncompressed),
externalizes the result into a compressed file (if argv[1] == 'ext') and prints
the number of objects in the dictionary. If argv[1] == 'int' it's not externalized.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <prop/proplib.h>

int main(int argc, char **argv)
{
 char path[256];
 prop_dictionary_t d;

 if (argc != 3)
  exit(EXIT_FAILURE);

 /* Externalize a plist file into a gzipped file */
 if (strcmp(argv[1], "ext") == 0) {
  d = prop_dictionary_internalize_from_zfile(argv[2]);
  snprintf(path, sizeof(path), "%s.gz", argv[2]);
  prop_dictionary_externalize_to_zfile(d, path);

 /* Internalize a gzip plist file */
 } else if (strcmp(argv[1], "int") == 0) {
  d = prop_dictionary_internalize_from_zfile(argv[2]);
  fprintf(stderr, "%s", prop_dictionary_externalize(d));
 }
 fprintf(stdout, "Total objs in dictionary: %u\n",
     prop_dictionary_count(d));

 exit(EXIT_SUCCESS);
}

I got two files, one compressed and another one uncompressed with the same
dictionary (and data) on them:

[juan@nocturno test]$ file pkgidx.plist*
pkgidx.plist:    XML  document text
pkgidx.plist.gz: gzip compressed data, from Unix, max compression
[juan@nocturno test]$

Now running the example code and redirecting stderr output to /dev/null
(which is too large to show) is able to read the data automagically:

[juan@nocturno test]$ time ./a.out int pkgidx.plist.gz 2>/dev/null
Total objs in dictionary: 3

real 0m0.041s
user 0m0.038s
sys 0m0.002s
[juan@nocturno test]$ time ./a.out int pkgidx.plist 2>/dev/null
Total objs in dictionary: 3

real 0m0.033s
user 0m0.029s
sys 0m0.003s
[juan@nocturno test]$

We can use the 'ext' argument to the example code to magically convert
a non compressed plist file into a compressed gzip file, like:

[juan@nocturno test]$ rm pkgidx.plist.gz 
[juan@nocturno test]$ time ./a.out ext pkgidx.plist 2>/dev/null
Total objs in dictionary: 3

real 0m0.111s
user 0m0.096s
sys 0m0.005s
[juan@nocturno test]$
[juan@nocturno test]$ ls -l pkgidx.plist*
-rw-r--r-- 1 juan juan 1009406 abr 20 15:22 pkgidx.plist
-rw-r--r-- 1 juan juan  130546 abr 20 15:49 pkgidx.plist.gz
[juan@nocturno test]$

The portable proplib implementation, maintained by myself, is available from
http://code.google.com/p/portableproplib and will be used in the upcoming XBPS version 0.5.

Some mistakes in shell programming

I've been seeing too many mistakes recently related to shell programming, due to some implemented features in GNU Bash (that are really cool if it were the only available shell on the planet). Let me tell you that it's really bad, because if those are used in some critical code to detect some implemented features, they can lead to serious problems in the build stage of the application, as I've seen with GNU autoconf generated configure scripts.

I'm posting this because I've spent a while detecting why libusb didn't have a symbol required by a dependent application; surprisingly that was due to the first mistake I'm explaining below in a configure script.

One of the most famous that you are probably aware of is the == operator used in conditionals. Remember that GNU Bash is the only shell that accepts the following:

if [ "$foo" == "$blah" ]; then
    ...
fi

and another one that I've just seen coming from the xorg detection block in GNU configure scripts (didn't see it until now):

BLAH="goo foo boob"
BLAH+="baobab anotherone"

This is a great mistake! those two examples (surely there will be many more) are relaying in GNU Bash for correct operation! think about it, relaying in such specific features won't do any good for all us.

libarchive-2.7.0 and openssl requirement

So libarchive 2.7.0 was released today, and after updating the package I noticed that the library required libcrypto, WTF? with XBPS that means it's not good, because libarchive is one of the few packages required to build the xbps-base-chroot meta-pkg. The configure script looks for some OpenSSL headers/functions and if they are found, OpenSSL is linked automagically.

I want to avoid having extra dependencies for a core library as libarchive is, even more if the feature that this dependency requires is rarely used (such as in my case)... so I made a simple patch that adds an option (--without-openssl) and reported it in http://code.google.com/p/libarchive/issues/detail?id=22

I hope that the developers do understand this and commit it just in time for the next version!

git: pushing to update remote branch refs

These past days I've been working on my hobby project xbps (xtraeme's build package system), which will be a package manager for GNU/Linux (not ready yet!). I'm handling the repository with the git VCS (Version Control System) because it gives me just what I want: performance, disconnected mode, etc, etc; also creating branches is cheap, you should be using them!

Back to the point that I was thinking for this post... I wanted to commit some experimental changes to test them before merging into master and I created a new branch; tested the changes, merged them into the master branch again, removed the now unused experimental branch and pushed the changes to the remote repo. After pushing the changes, the remote repo still showed up that the branch that I had just removed was still there... why's that?

Of course the answer was that you have to update the refs in remote, otherwise it doesn't know what changes you did locally! in short, to push your branch in remote repo:

$ git push origin branch

and to remove your local and remote branch (once merged into master or other branch):

$ git push origin :branch

Knowing this surely will help me in the future! I have still to learn a lot about it.

mkisofs(8) and HEAD

For reasons that I cannot remember, I needed to make a basic livecd from NetBSD -current to test some behaviour with the new modules style. Interestingly this triggered a problem with the bootloader not finding the modules at its standard path: /stand/arch/version/modules.

On HEAD for a while now, the GENERIC kernel hasn't built in the file systems by default; instead they are provided as modules and NetBSD's bootloader autoloads the required one. That works very fine currently for ffs, but not for ISO 9660 images built with mkisofs(8) with default options!

The root of the problem comes from the ISO 9660 level: by default mkisofs(8) uses level 1 and it only allows 8.3 file names (8 + 3 for the extension), as well as only a single dot in between; a leading dot is not allowed either. Also internally the ISO 9660 file system, converts all dots to underscores, so with default modules path (/stand/i386/5.99.4/modules), the bootloader is not able to find the module and the livecd is not able to mount the root file system.

After looking at makefs(8) I've found that to build the installcd, the allow-multidot option is required, as well as ISO level 2. Knowing this I've looked at the mkisofs(8) manpage and there's an option for exactly specify the ISO level; NOTE: if using mkisofs(8) you should only use -iso-level 2 exactly, don't use any other, it won't work.

In summary, if you want to build an ISO image with mkisofs(8) for booting NetBSD, now it's required to use the following options: -allow-multidot -iso-level 2.

Fixed ld@cac and null/overlay/umap modules

Between yesterday and today I fixed two problems on NetBSD. The former is still reproducable in the netbsd-5 branch (and HEAD as well), because AFAIK nobody submitted a ticket yet; the latter only applies to HEAD.

The first problem was introduced by myself some months ago, when I added support for JMicron and Intel RAID to ataraid(4). A did an extensive review before submitting the patch, but sometimes you always fail to catch an error... after all I'm a stupid human. Anyway, yesterday it took me 30 seconds to found it, because more or less I'm familiar with this code.

Info: http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=40099

The second problem is triggered every time that you use mount_nullfs(8), mount_overlay(8) or mount_umapfs(8), so I think it's important. The problem was that (again!) those missing symbols that the module requires are not there and some path in the code is causing a NULL pointer dereference. I've submitted a quick patch to fix them.

Info: http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=40110

compat_linux32 module broke my NetBSD!

I wanted to build the latest GNOME package available from pkgsrc for NetBSD, and I started removing all previous old packages to start with a clean set of new ones. All fine until Perl-5.10.0 executed a test via its configure script and NetBSD tried to autoload some modules for binary compatibility, that resulted in a panic while loading the compat_linux32 module due to a bunch of missing symbols.

The list of missing symbols were related to the fact that the module was missing to build some additional files from compat/linux, as well as symbols provided by the compat_netbsd32 module.

I still think that something is not working fine in how the kernel is handling those missing symbols, because IMHO, they should be treated as errors and not warnings!

Anyway I contributed a patch that fixed all issues, so that I can continue building GNOME again:

http://mail-index.netbsd.org/netbsd-bugs/2008/12/03/msg007545.html

UPDATE

The patch has been applied by Andrew Doran and therefore problem has been fixed.

The Portable proplib

Past month I decided that I'd like to do something in other OS != NetBSD, and by lack of development tools, performance and support, I decided to go with Linux (X window sucks, but what can we do?). So I was working in a new package system made from scratch with ideas from pkgsrc, ports, Compile from Gobolinux, Archlinux, etc.

The first implementation was called pkgfs, and it was only meant for NetBSD. It used db(1) files to handle build dependencies and registration of installed packages; this also used the native ftp(1) command to fetch source distribution files... after 1 week or so, I had a full X window system with latest Xorg, Gkrellm, GTK+, Firefox and others built with pkgfs.

Thanks to this I found a major bug in NetBSD's pthread library and ported the latest libpciaccess version from OpenBSD to NetBSD (that allows you to use Xorg's xserver 1.5.x on x86), amongst other things. Things were working wonderfully and even pkgfs was something like 30% faster than pkgsrc building from source... of course that was due to only add stuff for NetBSD and not other OSes.

But after some days and thinking about it, NetBSD wasn't suited for what I wanted to do. It's a marvellous OS but still needs many years of development to compare to current's Linux state. So with that idea in mind and some comments from a friend, I decided to give up definitely from NetBSD and switched to Linux.

I had to make radical changes to pkgfs to make it work on Linux, basically replacing the need for db(1) and ftp(1) , that aren't available in most Linux distributions. Talking with the same friend, we decided to go with SQLite to handle dependencies, and the idea wasn't really bad... but thinking about what I needed exactly once again, I thought the best was to use something simpler, without going to the SQL path.

db4 from Sleepycat (now Oracle), seemed ok but after looking at its API it was too complicated for my taste; so I decided to start with NetBSD's proplib! because I'm experienced with this library (see sysmon_envsys(9) and envstat(8) on NetBSD) and it's a very good piece of code. I started removing NetBSD related stuff, plus minor changes like strlcat() -> strncat() and voila, proplib was working fully in userspace on Linux; just what I wanted!

Because I feel bored I thought it was cool to make it build through GNU autotools (autoconf, automake and libtool) and I did it. With those changes the Portable proplib 0.1 version was released via Google Code.

That's the short history of the Portable proplib version. Today I incorported a bugfix that was committed some days ago to NetBSD's tree, changed the autoconf scripts to detect some more pthread required functions and released version 0.2. I'm using it currently for my new build package system for Linux xbps. I will talk about xbps in the next post perhaps.

If you want to use proplib on any other OS than NetBSD, try it! and let me know if it doesn't build or there is any problem, I'll fix it ASAP. Here are the links to the stuff I've been talking about:

The Portable proplib: http://code.google.com/p/portableproplib/
xtraeme's build package system: http://repo.or.cz/w/xbps.git