git: pushing to update remote branch refs

These past days I've been working on my hobby project xbps (xtraeme's build package system), which will be a package manager for GNU/Linux (not ready yet!). I'm handling the repository with the git VCS (Version Control System) because it gives me just what I want: performance, disconnected mode, etc, etc; also creating branches is cheap, you should be using them!

Back to the point that I was thinking for this post... I wanted to commit some experimental changes to test them before merging into master and I created a new branch; tested the changes, merged them into the master branch again, removed the now unused experimental branch and pushed the changes to the remote repo. After pushing the changes, the remote repo still showed up that the branch that I had just removed was still there... why's that?

Of course the answer was that you have to update the refs in remote, otherwise it doesn't know what changes you did locally! in short, to push your branch in remote repo:

$ git push origin branch

and to remove your local and remote branch (once merged into master or other branch):

$ git push origin :branch

Knowing this surely will help me in the future! I have still to learn a lot about it.

Booting Linux with 2048 bytes per sector USB flash devices

For some days I've been wanting to try a Linux distribution on my Nocturno computer (Pentium 4 3GHz, 1GB DDR2-667 RAM, 750GB in a RAID5 Array with an Areca ARC-1210 PCI-e RAID card), because I wanted something that is really stable and fast to make it a file server. The problem with that machine is that it doesn't have a CD/DVD-ROM drive, so my choices for installing are via Network or USB.

I've never bothered to do a full network installation (via TFTP with NFS root or such), and therefore I used the USB method. I own a Creative ZEN Stone with 1GB Flash device, that I used for this. The problem is that for some time now, some USB flash devices now use a 2048 Bytes per sector, instead of the common 512 bytes per sector; this device is one of them.

Once important issue with this layout is that GNU Grub (legacy) and ISOLINUX do not support these drives by default. For ISOLINUX it seems that the only way to make them work is to modify the source code, as specified by its Author; for GNU Grub perhaps there's a patch, but I haven't investigated enough.

Now you might be thinking, how do you boot a live system from these devices then? good question. The solution seems to come from the fact that these devices share a common detail with CD/DVD-ROM drives, they all use this layout. So simply using dd(1) to copy an ISO 9660 image directly into the device, works.

To verify this, I tried with Fedora 10, OpenSuse 11.0 and a custom NetBSD LiveCD built by my own application mklivecd(8). It's important that dd(1) reads and writes blocks of exactly 2048 bytes. In short, if you want to boot from USB on any of these drives, use something like:

$ dd if=/iso9660-image.iso of=/dev/sdd bs=2048

The image must be written into the whole device, and not a partition on it; just treat those USB flash drives as if they were CD/DVD-ROM discs.

mkisofs(8) and HEAD

For reasons that I cannot remember, I needed to make a basic livecd from NetBSD -current to test some behaviour with the new modules style. Interestingly this triggered a problem with the bootloader not finding the modules at its standard path: /stand/arch/version/modules.

On HEAD for a while now, the GENERIC kernel hasn't built in the file systems by default; instead they are provided as modules and NetBSD's bootloader autoloads the required one. That works very fine currently for ffs, but not for ISO 9660 images built with mkisofs(8) with default options!

The root of the problem comes from the ISO 9660 level: by default mkisofs(8) uses level 1 and it only allows 8.3 file names (8 + 3 for the extension), as well as only a single dot in between; a leading dot is not allowed either. Also internally the ISO 9660 file system, converts all dots to underscores, so with default modules path (/stand/i386/5.99.4/modules), the bootloader is not able to find the module and the livecd is not able to mount the root file system.

After looking at makefs(8) I've found that to build the installcd, the allow-multidot option is required, as well as ISO level 2. Knowing this I've looked at the mkisofs(8) manpage and there's an option for exactly specify the ISO level; NOTE: if using mkisofs(8) you should only use -iso-level 2 exactly, don't use any other, it won't work.

In summary, if you want to build an ISO image with mkisofs(8) for booting NetBSD, now it's required to use the following options: -allow-multidot -iso-level 2.

Fixed ld@cac and null/overlay/umap modules

Between yesterday and today I fixed two problems on NetBSD. The former is still reproducable in the netbsd-5 branch (and HEAD as well), because AFAIK nobody submitted a ticket yet; the latter only applies to HEAD.

The first problem was introduced by myself some months ago, when I added support for JMicron and Intel RAID to ataraid(4). A did an extensive review before submitting the patch, but sometimes you always fail to catch an error... after all I'm a stupid human. Anyway, yesterday it took me 30 seconds to found it, because more or less I'm familiar with this code.

Info: http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=40099

The second problem is triggered every time that you use mount_nullfs(8), mount_overlay(8) or mount_umapfs(8), so I think it's important. The problem was that (again!) those missing symbols that the module requires are not there and some path in the code is causing a NULL pointer dereference. I've submitted a quick patch to fix them.

Info: http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=40110

compat_linux32 module broke my NetBSD!

I wanted to build the latest GNOME package available from pkgsrc for NetBSD, and I started removing all previous old packages to start with a clean set of new ones. All fine until Perl-5.10.0 executed a test via its configure script and NetBSD tried to autoload some modules for binary compatibility, that resulted in a panic while loading the compat_linux32 module due to a bunch of missing symbols.

The list of missing symbols were related to the fact that the module was missing to build some additional files from compat/linux, as well as symbols provided by the compat_netbsd32 module.

I still think that something is not working fine in how the kernel is handling those missing symbols, because IMHO, they should be treated as errors and not warnings!

Anyway I contributed a patch that fixed all issues, so that I can continue building GNOME again:

http://mail-index.netbsd.org/netbsd-bugs/2008/12/03/msg007545.html

UPDATE

The patch has been applied by Andrew Doran and therefore problem has been fixed.

My lovely daughter: Leyre

This post is dedicated to my lovely and first daughter Leyre. She was born a Monday 21st in August of 2006. She's almost two years and half already! time has passed really fast since then...

Here are some pictures that I like so much that I wanted to share publicly. Of course she's very pretty and really clueful for its age; the pictures are from past summer, but I don't have recent enough ones with enough quality to put here right now.

Half Life 2 Mod: Black Mesa

So I was watching one of my favourites videogames related sites (3Djuegos.com) and I've found something that I've been really surprised with. A new video of a game called Black Mesa. After reading a bit on its page, I discovered that it's just a HL2 Mod!

The video is amazing, and it shows that this is more than a yet another Mod for Valve's Source engine. It seems that Black Mesa is a remake of the first Half Life, released in 1998. They are using the Source engine really at its max power, and added other cool features like cooperative multiplayer, etc. The Source engine is showing its age, but still I think it's one of the best 3D engines out there!

From what I read I don't know if it will be available for free, but surely the money will be well wasted. Here's the amazing video available from 3Djuegos.com and from Developer's site: http://www.blackmesasource.com/

The Portable proplib

Past month I decided that I'd like to do something in other OS != NetBSD, and by lack of development tools, performance and support, I decided to go with Linux (X window sucks, but what can we do?). So I was working in a new package system made from scratch with ideas from pkgsrc, ports, Compile from Gobolinux, Archlinux, etc.

The first implementation was called pkgfs, and it was only meant for NetBSD. It used db(1) files to handle build dependencies and registration of installed packages; this also used the native ftp(1) command to fetch source distribution files... after 1 week or so, I had a full X window system with latest Xorg, Gkrellm, GTK+, Firefox and others built with pkgfs.

Thanks to this I found a major bug in NetBSD's pthread library and ported the latest libpciaccess version from OpenBSD to NetBSD (that allows you to use Xorg's xserver 1.5.x on x86), amongst other things. Things were working wonderfully and even pkgfs was something like 30% faster than pkgsrc building from source... of course that was due to only add stuff for NetBSD and not other OSes.

But after some days and thinking about it, NetBSD wasn't suited for what I wanted to do. It's a marvellous OS but still needs many years of development to compare to current's Linux state. So with that idea in mind and some comments from a friend, I decided to give up definitely from NetBSD and switched to Linux.

I had to make radical changes to pkgfs to make it work on Linux, basically replacing the need for db(1) and ftp(1) , that aren't available in most Linux distributions. Talking with the same friend, we decided to go with SQLite to handle dependencies, and the idea wasn't really bad... but thinking about what I needed exactly once again, I thought the best was to use something simpler, without going to the SQL path.

db4 from Sleepycat (now Oracle), seemed ok but after looking at its API it was too complicated for my taste; so I decided to start with NetBSD's proplib! because I'm experienced with this library (see sysmon_envsys(9) and envstat(8) on NetBSD) and it's a very good piece of code. I started removing NetBSD related stuff, plus minor changes like strlcat() -> strncat() and voila, proplib was working fully in userspace on Linux; just what I wanted!

Because I feel bored I thought it was cool to make it build through GNU autotools (autoconf, automake and libtool) and I did it. With those changes the Portable proplib 0.1 version was released via Google Code.

That's the short history of the Portable proplib version. Today I incorported a bugfix that was committed some days ago to NetBSD's tree, changed the autoconf scripts to detect some more pthread required functions and released version 0.2. I'm using it currently for my new build package system for Linux xbps. I will talk about xbps in the next post perhaps.

If you want to use proplib on any other OS than NetBSD, try it! and let me know if it doesn't build or there is any problem, I'll fix it ASAP. Here are the links to the stuff I've been talking about:

The Portable proplib: http://code.google.com/p/portableproplib/
xtraeme's build package system: http://repo.or.cz/w/xbps.git

Starting new public blog

Hi,

so after moving to new home a while ago (+6 months already) and loosing my old domain and server, I decided to open a new blog; but this time in shakespeare's language. Yes, my knowledge is poor, but learning is the key of all humans!

Too many things happened over this time after my last post on the previous blog... one of them being that I'm not a NetBSD developer anymore. I'm still contributing, but I don't have contacts to any other developers, or either any other problems I had in the past by being too active. Too many stuff on my head, I needed to take a rest and do something stupid... I had some reasons, due to some people being idiots, but who cares?

In the meanwhile, at the end of this month my second daughter will born. We decided to name her Sara. I'm getting old and I still think that I'm 16... too bad.

Today couldn't sleep so I decided to re-open a blog and do other stuff. Will talk about this stuff in the next post perhaps... enough for the first post :-)