I haven’t posted in a while, so I figured I’d just give a sample of the kinds of things I deal with as an Ubuntu developer.
When a Wine developer wants to implement support for playing mp3 files, the smart thing to do is to use a software library, in this case mpg123. Wine is a good example of a large and still growing piece of software with a similarly expanding list of dependencies.
It’s my job to make sure all these bits work well together so that Wine can be built in a complete way for you – without mpg123, Wine would have to be built without full support for mp3s, and sound on some apps would simply break. Sometimes this means packaging software that isn’t in the distribution yet. Usually it means fixing bugs, and here we can see the true strength of the open source approach.
It starts with a bug report
I was ready to put out a Wine release with its new mpg123 support, but Wine wasn’t building on 64 bit. This was a case where the mystery was more interesting because it was only partially broken: 32 bit worked perfectly.
I knew right away that it was probably an Ubuntu-specific issue, although other distributions (especially Debian) might also be affected. I filed a bug upstream to try and nail down the specifics. Everyone shares their relevant expertise here: a Wine developer notes that it’s a mismatched symbol-header problem, I figure out the cause is Ubuntu’s unexpected use of 32-bit headers on 64-bit systems, a community member provides a temporary workaround patch, an mpg123 developer tells why things were the way they were and how Ubuntu+Wine was the first to expose this problem, and Alexandre Julliard says what he’d ultimately like as a best practice.
This sort of collaboration is exactly what Open Source is about. Everyone involved could just go to the same bug tracker and simply get to work – it didn’t matter that all 6 of us were in different continents working for different companies on different projects. Everything is laid bare, easy to see and thus easy to fix.
Getting it done
At the end of the day, getting the fix out to users is what actually matters. It’s really disappointing when great code like mp3 support just doesn’t happens because of something silly like an out of date distribution library.
So, that’s the final step in the chain – the mpg123 developer puts out a new version of mpg123, and then I have to package that up and put it into the latest Ubuntu alpha. This is where Launchpad’s ability to have a single bug exist against multiple packages becomes especially helpful: by filing a launchpad bug I could track the changes I had to make to all 3 packages (mpg123, ia32-libs, and then wine).
Technical details for the interested
It’s worth noting that this bug could have been prevented entirely if Ubuntu had proper multiarch support. Multiarch means the ability to install 32-bit packages directly onto a 64-bit system, rather than having to use special separate 32-on-64 packages like ia32-libs. Multiarch has been a slow, gradual redesign under development in Debian and Ubuntu for years now, and it just barely missed the Lucid cycle.
This leaves us with the current implementation, which can at best be described as “a tremendous hack.” 32-bit libraries from needed packages are manually added to a giant list. Then they’re processed by a script and all thrown into one gigantic, 600+ megabyte package named ia32-libs. This means that any program which needs 32-bit compatibility (such as Wine) has to depend on this massive source package, and any time a single component changes it needs to be manually updated.
However, ia32-libs contains only the library files, not the header files, so when you build a 32-bit application on 64-bit Ubuntu you’re actually building with the header files installed from the 64-bit version of the package. This is normally fine, as most header files are identical across different architectures, however this wasn’t the case with mpg123. So we were using a 32-bit library that had different symbols than the 64-bit header file, and the build then fails with symbol errors.
There are quite a few applications for 64-bit Ubuntu that need to make use of 32-bit libraries for various reasons, however Wine is by far the most dominant. This makes me, the one responsible for Wine, also the official babysitter of ia32-libs and similarly broken packages. Wine is special in a way, though, as unlike other applications there is no way to fix it by just making it 64-bit — Wine will always need 32 bit libraries in order to run 32 bit Windows applications, which will remain incredibly common for at least another decade.
So here’s the deal: you want to help contribute to Ubuntu in some small way but don’t want to make much effort or risk your “real” computer. Enter ISO Testing.
Basically, you burn a disk and attempt to install. If it works, you click a box, and if it doesn’t, you click another and file a bug. If the installer finished but turned everything pink and that’s not really your thing, don’t check the box for “serious”.
This is one of the best ways of helping Ubuntu that doesn’t actually involve using it. So if you have a spare computer, partition, or even VM you can basically spend about 30 minutes after downloading and submit a good test report.
To get started:
Go here and make an account and then click about 4 buttons. There’s even a “I’ve started a test so others can try something else” feature.
Once enough people give successful ISO test reports, the Alpha is released and the coolest of us can start general testing. That’s where you actually use the installed system and test all the experimental new features like being slightly less brown. But before that can happen, testers need to know they can install and update without hassle – the daily CDs are frequently unbootable for one reason or another, so that means the alpha CDs need to be good enough to install from.
Sound in Wine has been a big issue. From a user’s perspective, it didn’t work well. From a technical user’s perspective, there were 3 different drivers to choose from and none of them worked well. From a developer’s perspective, no sound driver would ever work well.
Wine was a victim of the proliferation of sound drivers on Linux.
At one point, Wine had separate sound drivers for ARTS, ESD, OSS, JACK, and ALSA. None of them worked right, even if the user could figure out how to configure Wine to use the “correct” one. Subtle changes to each API and differences between distribution versions made matters even worse.
PulseAudio was supposed to be a solution to problems like this by being the one true Linux Sound System, but in some ways it made the problem worse. Now Wine needed a separate PulseAudio sound driver, and no one wanted to write sound driver code since the whole thing was a big mess to begin with. Worse, the most work had gone into making the ALSA driver better, and there were a few technical reasons why it seemed like a good choice to stick with ALSA and use it as a default.*
So, the decision was made to go with ALSA and improve it a bit, and hope that PulseAudio’s ALSA-compatibility layer would work well enough. That didn’t exactly happen – PulseAudio didn’t actually want to support some “abuses” of the ALSA API, as one dev put it. Some users took to killing PulseAudio outright every time they wanted to use Wine.
Things got a bit political. Was Wine at fault here, or PulseAudio? Should Wine include a PulseAudio driver, even if it was poorly maintained? What would the default be? Also, what the hell were we gonna do about the Mac?
The solution, which in retrospect is pretty obvious
Wine wasn’t the only program facing this problem. Anyone writing or porting a Linux game had to answer the question about how they would play sound – almost none chose to write 7 separate audio drivers. Instead they all did the smart thing, and used a higher level library.
So, that’s the plan – Wine will use a sound library. And then we’ll let them worry about actually talking to whatever sound system happens to be working best. OpenAL is the smart choice here, for much the same reason that OpenGL was the smart choice for graphics. We get Mac compatibility “for free” too.
Now, why didn’t this happen earlier? Well, in fairness to Wine, OpenAL didn’t actually exist in a usable form until Wine already had sound drivers for a few years. A similar thing happened with wikis and distributed version control systems – we eventually saw the light that new open source projects had been following for years, and the project is better because of it.
* As I understand it, a lot of Windows programs expected to be very close to the hardware and at the time many users were having issues with PulseAudio latency.
We’re serious about a 1.2 release in the next few months. As Alexandre said, “it’s time”.
64 bit support will be the major new release goal.
For most people, having no regressions and working with way more apps than 1.0.1 did will be the actual feature they care about.
Users will need to make a new Wine installation to migrate from 32-bit to 64/32-bit capable. It is left to me to figure out and code a humane way to present this. This is also a convenient time to make their Wine folder case insensitive.
We’re giving up on separate Pulse/ALSA/OSS/Jack sound driver layers and instead doing the smart thing: passing everything to OpenAL. Maarten Lankhorst will handle most of it.
Packagers like me will have to work out CJK font support for our respective distros. Wine will need manual links to whatever font is actually provided by the distro when it’s in a CJK locale.
We need some help from Freedesktop.org for solving bug 10841. Essentially we need a standard way of saying “hey reset the resolution when I’m done, even if I crash.” I was “volunteered” for the task of approaching the respective projects.
We need user help finishing up the Tahoma replacement font. It’s still missing some glyphs for certain languages and some users report it as “ugly”.
VMware is an ally of Wine. They use our test suite on Thinapp, and even contribute test fixes and new tests back to us. Tests are just as good as “real” code.
We need more users to run git update and winetest daily (with their screens unlocked).
Alexander announced a new patch tracking system to prevent patches from getting lost in the nether (and for devs to know when a rejection occurs)
By far the largest concentration of wine developers is in the netherlands. It has something to do with being below sea level, which means global warming is an ally of Wine as well.
Francois Gouget has promised to teach me every nuance to Winelib so that I can document it properly.
I need to update about 10 different wiki pages to give them more modern instructions, such as how to get a good backtrace (eg on Ubuntu you install wine1.2-dbg). A recurring theme to this conference seemed to be that if it’s written in English and concerns the Wine project, it’s probably my responsibility at this point.
Owen is going to setup planet.winehq.org, since he’s done it before for another project. There you’ll see my blog and a few others. I might actually update more than once a month too!
Andre Hentschel is working on a Wine port to ARM. I’ll see if I can get him a first generation Pandora (I had the misfortune of preordering two about a year ago, and they may actually arrive sometime next month).
Jeremy White wants help recognizing people who should come to wineconf.
I need to make a proper message box to indicate that Wine is loading immediately after a user opens an application (currently there can be about 5 seconds of nothing, resulting in either frustration or opening the application multiple times). Wine should be sending the signals it needs (with xdg message) already, we just need to have the desktop listen to them.
I need to make a “Press” page for the website, including an email address for press@winehq.org which will probably direct to me possibly several others. I’m curious how other projects have handled this – is it a good idea to have multiple press contacts?
All documentation needs to be moved to the wiki. From there it shouldn’t be too hard to setup a wiki->pdf->docbook/xml conversion script and have the documentation build automatically based on the latest wiki pages. Then we can start shipping it with packages again (since offline documentation still matters somewhat).
I’m going to experiment setting up Redmine for the Wine project. If I can get it running well on a separate server and have it fully import Bugzilla, the Wiki, and our Git repositories then we can switch to it and start using its nice management features. Budgetdedicated will provide the server.
The winehq.org Donation link needs to go to a proper web page that actually says where the money goes and what we’ll do when we raise enough. Currently it just dumps you off to paypal without explanation. Amazingly, some people actually give money (enough to partially fund WineConf). This task is also mine.
I need to finish preparing for the Ubuntu Developer Summit that starts in a few hours, so the bullet points will have to suffice for now. I’ve got my work cut out for me it seems.
Sutor said they need to focus on usability, stability, security, reliability, performance, with some cool thrown in, as well… I think making it a complete drop-in replacement is a dead-end strategy.
Sutor’s speech on desktop Linux has inspired me. Accordingly, I’d like to present the…
Bob Sutor Guide to Running a Marathon
Overall fitness is important on race day. You’ll need to focus on your endurance, your heart, your leg muscles, your feet, your abs, your waist, and some arms thrown in too.
Running form is important on race day. You’ll need to focus on your breathing, your leg motion, your arm movements, the length of your strides, and some pacing practice thrown in too.
The training routine is important on race day. You’ll need to focus on finding other runners to practice with, self discipline, getting a top personal trainer, and some knowledge of the track thrown in too.
Don’t try and win the race. That’s a dead-end strategy.
You’ll note that Bob left out “compatibility” from his list of things to focus on. A strange omission, coming from the company that witnessed IBM Compatible computers dominate the PC industry without actually being sold by IBM. Compatibility then meant being able to run Microsoft with the same hardware and software. It still does today.
We’re pretty good at hardware. My girlfriend got confused by the manual that came with her latest printer: it gave 12 steps for Windows, another 12 for Mac, and then an additional 6 for both. There were no Ubuntu instructions. Turns out all she had to do was plug it in, the first thing she would have tried if there wasn’t a scary manual making her think printer setup required some sort of shaman dance. This is a good problem to have, although having to tell your girlfriend to stop shaking her hips at the printer is perhaps an even better problem to have.
Winning
Hardware works, but we haven’t won yet. We could make every single graphics card, printer, digital camera, and music player work out of the box flawlessly and we’d still miss the market completely. We wouldn’t actually be compatible. What we have is a software problem. That’s why I work on Wine.
IBM management has never really believed in Wine as a technology, even though their own engineers find it useful. Jeremy White has even said that IBM Managers weren’t very helpful.
For my part, I’m going to ignore IBM’s failed attempt at leadership and focus on what actually matters: making things work for users. Only if we become compatible can we be at least as good. Then people give us a try, and notice we’re actually better. That’s when we win.
Here, let me highlight it for you in case you missed it.
That’s at least five different places where we tell the user to restart. This is, as I would say to my students, a redundant use of redundancy.
I think we can give our users a bit more credit here. Consider this version:
That pretty much says the same thing, with no text at all. But even that may be more than is needed. Suppose you just ran update manager, installed a system update, and then saw this:
For those of us who’ve used update manager before, this isn’t much of a puzzle to figure out. The computer is asking for permission to do something, and since we just ran update manager it’s probably a system restart.
What’s important
Whoever wrote this dialog obviously wanted to get across the message that a restart was about to happen. Well, they succeeded, but now we have a problem of wordiness. This dialog doesn’t just say restart, it also says this:
So let’s use the example above and improve this dialog a bit, emphasizing what’s actually important:
Not bad isn’t it? The important bits are in there, and they’re no longer lost within a 3 paragraph beast. Let’s try and humanize it a bit without losing these advantages.
Same message, much more readable. Removing the title is an interesting change: now it doesn’t feel like reading a document for homework, but instead something predictable and very easy to understand. This is just a first draft, of course, but since it’s something literally millions of people will see, I think it’s worth taking a bit of time to get it right.
It was a few months ago, however the videos from the Ubuntu Developer Summit have finally been made available to me in a streamable form. This was a short, unrehearsed talk I gave to briefly convince the Ubuntu developer community that Wine isn’t a threat to free software, but rather an important tool that makes migration to a free platform possible.
Advice for those giving presentations
After rewatching the above, I can say the following things with confidence.
The fewer words on your slides, the better.
Bullet points aren’t short enough.
If you start with a borrowed joke it will come off awkward.
Unless you can’t be funny, write your own jokes.
Never let someone control the slides for you.
Wine Bug Day this Sunday
Now, while I have your attention, how about you participate in the next Wine Bug Day for this Sunday? It’s very simple – mostly we’re just downloading random games and playing them for a bit to see if a bug is still there. And it all helps Wine development, so it’s good. Check out the wiki page and join us!
One of my self-appointed duties has been cleaning up the Winehq.org web site. It was, in short, failing miserably at its purpose. Users would go to the website and still not really know what the Wine project was about. If they read a lot of words, they might be able to learn what the program does and where to download it, but if they read too much they’d land on a status page that hadn’t been updated since 2004.
Documentation that makes the user feel less informed after reading it is a very special kind of awful.
If everything is broken, start with what’s most visible
The two most popular pages were the front page and the About page. They seemed like simple enough targets. The front page was just a handful of links and large, friendly buttons, however even here there was substantial room for improvement. The key is to use affirmative verbs to imply what happens when the user clicks: Overview of Wine features becomes Learn about the Wine project. By simply changing a subtitle, we’ve already given the reader confidence and come off far more sincere.
Unfortunately, if the user actually clicked the About page, they would see this:
Wine is a translation layer (a program loader) capable of running Windows applications on native programs Windows 3.1 ports for other Unixes bug tracker Win32. Here’s a link to 7 even longer articles.
It went on for 4 paragraphs – that’s an awful lot of text to have on an about page. We needed to start from scratch, and that includes figuring out what the page is actually supposed to do:
Goals of an About page:
Tell them that Wine can run their Windows programs on Linux (or Mac)
Not make it sound like Wine is complicated enough to require shaman dances
Point users to AppDB if they want to know about a particular program
Encourage users who are still on Windows that migrating to Linux is ok
Be concise enough to read
Explain the open source nature of the project, and the role of Codeweavers
I worked on it for a good number of hours. It’s surprising how long it takes to do focused writing, but when you do it right you end up saying more than twice as much in half the words. I’m quite happy with the new text on the About page, although now there’s a new problem: the text is so short it looks really flat and wide.
This is a good problem to have. The traditional answer for long text is to collapse it into a narrow column. If you read a Paul Graham essay, for instance, you’ll find that 3/4 of the page is whitespace on a typical widescreen monitor. For an About page, this is the perfect place for a picture. I don’t, however, have one just yet.
Can you think of a picture that sums up what Wine is about?
Announcing plans for a second Wine Bug Day
On another note, since the last Wine Bug Day was so successful, I’ve decided to do the exact same thing again. I’m not yet certain of the time, however. I’ve heard feedback that a weekend might be better than a Monday, but I don’t really know. If you have experience with this sort of thing, please feel free to comment. Thank you, especially if you want to participate.
I want to help Wine development in any way I can. Sometimes this means putting on pants and discussing donating money to Wine with serious people. Sometimes it means using my charming personality to persuade promising programmers into becoming Wine developers. Sometimes, however, it’s about something larger: helping existing developers work more efficiently.
Engaging the community
While Wine has over 2 million users, we’re not particularly good at marshaling the community towards helping out. The Application Database, for instance, contains test reports that are frequently out of date and not particularly useful.
Getting good involvement from the community is not a trivial task. Users need some level of expertise and confidence before they can offer useful help to others. People need to suffer through reading the same question over and over again in mailing lists and forums before they’ll volunteer to write the answer into actual documentation.
This means the most important tools are those that make it easier for community members to build that confidence and contribute. We used to require users to submit patches to update the FAQ. This meant they had to grow neckbeards, put on wizard hats, and use terminal commands just to edit a text document. Unsurprisingly, the FAQ quickly became so out of date that it was outright harmful to read.
After some nagging, the FAQ was eventually moved from the source tree into a wiki. The barrier to contributing went from learning a version control system to clicking an edit button on a web page. After years of neglect, the FAQ became genuinely useful within a single month.
I want to give Wine coders more time to, well, code. That means less time wasted doing all the other annoying tasks in life. Although I can’t do their laundry or clean their bathrooms over the internet, I can help them spend less time sifting through Bugzilla.
Is it worth triaging bugs?
Wine, like all reasonable projects, is utterly swimming in bug reports. Fortunately, we’re not quite drowning in them – while thousands remain open, we’re triaging them at about the same rate as new ones are being added, barely keeping our head above water.
Let’s think for a moment about what a developer gets from a bug tracker:
If something is broken, they learn about it so they can solve it later
Developers can collaborate with eachother on a solution
If they need motivation they can feel happy that actual users would appreciate a fix
They can consider the difficulty and importance of different bugs to prioritize work
All this breaks down when there’s a ton of out of date or otherwise bad bug reports. Instead of a useful tool, the bug tracker feels like an obstacle. Eventually it ends up like graffiti in a big city: the problem becomes so huge that any individual effort spent fixing it feels like a waste.
Working together on a bug day
The idea of a bug day is to make it much easier for community members to triage bugs. You’re all doing it together, so there’s less a feeling of solitude even if you’re alone in your parents’ basement. If you’re unsure of something, you just ask someone in chat, building up a little confidence. If you don’t have bugzilla permissions, someone else will set a bug’s status to fixed for you right there, keeping the energy up. This makes the whole process actually kind of fun – so much so that people actually organize real life parties dedicated to bug triage.
That’s why bug days work – they help bugs get triaged in the same way that wikis help documents get written. You see the collaboration and progress right before your eyes, especially when you pick a particular set of bugs as a target. There’s an easily measured goal to start with, and everyone gets excited when the group hits 50 bugs for the first time. This even works when the bugs are a relatively easy subset: for Wine’s first bug day I decided to only target bugs with free downloads that hadn’t been checked in over 7 months.
Claiming victory
To be honest, this bug day wasn’t very well planned. I basically picked a day, threw together a forum thread and an email to the developer list, and then hoped for the best. I didn’t even blog about it. Somehow, amazingly, it was still an overwhelming success:
Our bug target search (bugs unchanged since January 1st with the download tag) started at 574 bugs. Now it’s down to 508. That’s 66 bugs that were either resolved, confirmed, or at least checked in on.
In the past two days there have been 30 bugs newly resolved. By comparison, over the previous 2 weeks before then there were only 16 bugs resolved.
I’m definitely going to repeat it again and make this a regular thing. The community really does help if you simply ask.
It’s not a good situation when the most common advice to users experiencing problems is “upgrade to the unstable betas”, but that’s the reality of Wine these days. It’s also a sure sign of a release getting stagnant – when developers are actively telling people not to use it.
That’s why I’ve made the latest Wine beta releases available for Ubuntu Karmic. All you have to do is install the Wine1.2 package. No, Wine 1.2 isn’t actually out yet (the package currently has v1.1.25). But, when Wine 1.2 is out, I can easily replace the beta version with an update, complete with official-sounding release notes and documentation that no one will ever read.
This approach is very similar to what we did during the Ubuntu 8.04 cycle. Firefox 3 wouldn’t be out in time, but if we shipped Firefox 2 then our users would be stuck with a crusty web browser. That’s a mean thing to do to users, so instead we gave them a relatively stable beta version of Firefox 3 and then issued an update after release. I did the same thing with Wine – Wine 1.0 wasn’t out yet, so I shipped a pretty good beta and then patched it to 1.0.
If only there were more frequent releases
Now, I’ve been advocating Wine 1.2 come out in time for the next Ubuntu release for over a year now, but the truth is that’s going to be about as effective as my campaign to get old people to wear their diapers on the outside. Stable releases don’t just happen on their own; they need good release management, and Wine doesn’t even have a release manager at this point. In short, there’s a very good chance Wine 1.2 won’t be out in time, and at this point the 1.0 branch is as crusted as Firefox 2 was.
In search of a release manager
Wine 1.0 took 15 years to come out (incidentally, this is about as long as it took for Duke Nukem to get cancelled). We seemed to be doing a lot of development, but never actually progressed towards a stable release. This all changed during Wineconf, when a confluence of angels inspired the Wine developers to select Dan Kegel as the 1.0 release manager. He did the job wonderfully, and Wine 1.0 actually released, to everyone’s surprise. Even Linux Hater felt the need to comment.
Our mistake, however, was stopping there. Dan Kegel doesn’t have much time for Wine anymore, and last wineconf we didn’t pick a new release manager for 1.2. So the situation looks much like it did during those 15 years before Wine 1.0’s release: a bunch of developers doing their own thing while everyone else assumes that means picking our noses.
Wine still improves, of course, but without a stable release process users either run into frustrating beta regressions or try to use the old stable version only to discover that it doesn’t work half the time. This sounds awful, but it is a good problem to have – it means we’re a mature project with actual users who really want something they can depend on.
Volunteering for the job
So, I’ll be up front about this. Wine needs a new release manager, and if I can’t find a better person to do it I’m willing to tackle the job myself. Maybe that means personally setting up another server farm to run daily regression tests. Maybe it means going Indiana Jones on the darker parts of bugzilla only to discover that what I thought were useful patches were actually piles of scorpion-powered snakes.
I’ve already contacted Dan Kegel to ask for some pointers, and at his suggestion I’m going to take a much more active role in Wine bug triage. The release criterion wiki pages also need updating, and there’s likely going to be a lot of wrangling over the particulars of just what we as a project want Wine 1.2 to be. But that’s exactly the kind of discussion this community need to have – otherwise, a release simply won’t happen.
native programs 
