“Oh.”

I began to feel the consequences of the impression I’d given. “Well, we want a real developer.” Without thinking, I’d managed to frame my years of experience in precisely the wrong way. All that integration work, application testing, knowledge of scripting languages and the deep internals of Ubuntu suddenly counted for nothing. It didn’t matter how much work I’d done or how many developer summits I’d been sponsored to: in this moment I was someone who didn’t even write simple Wine conformance tests.

We talked some more, and I went back to bed to wake at midnight, technically Friday. Too late and too early to go out, everything was quiet enough to get some real work done. I thought about the earlier conversation, and while I hadn’t written C code since high school I decided to dive right back in and hack at Wine myself. Within minutes I found a bug, and four hours later I had code not only proving it, but also fixing it for good.

Test driven development

Today’s Wine consists of two equally important parts: a series of implementations pretending to be parts of Windows, and an ever-growing suite of unit tests. The implementations are straightforward once you know the right thing to do: if the waiter function in Windows’ restaurant.dll asks for a tip, then ours needs to as well. Similarly, the tests prove what the right thing actually is, on both Wine and Windows. They help us answer weird questions, like if the Windows waiter still demands a tip with a negative bill. Somewhere out there, there’s a Windows program that depends on this behavior, and it will be broken in Wine if we make the mistake of assuming Windows acts reasonably.

I asked a developer to recommend a DLL that needed better tests, picked a random C file in it, and started looking. I soon found my target, a documented and complete implementation of a function with only partial tests. This code is already written, and believed working, in Wine. I was supposed to write a bunch of tests that should pass in Wine. That’s when I learned the function is already broken.

The Wine Testbot

Wine owes a lot to the late Greg Geldorp, an employee of VMware who originally created a Windows testbot for us. Any developer can submit a test patch to it, and those tests will be run on multiple versions of Windows to make sure they pass. It saves us the trouble of having to reboot into 10 different versions of Windows when hacking on Wine.

When I used the testbot to run my new tests, however, I found that while they passed on Wine they actually failed on Windows. Since we’re supposed to do what Windows does, no matter how stupid, that meant two problems: my new tests were bad, and Wine had a bug. Fixing the tests is simple enough – you just change the definition of “pass” – but this kind of unexpected failure can also inspire even more tests. By the end of it I had 6 patches adding different tests to just one function, 3 of which were marked “todo_wine”.

Fixing Wine

While simply submitting tests would certainly be a useful contribution, I felt like I could do more. “You found the mess, you clean it up” is an annoying cliché, but here it has a ring of truth to it: my recent experience writing these tests meant that I had become the world expert on this function. At least, for the next few days, after which I planned on forgetting it forever. That’s what good tests are for – they let us confidently ignore the internals of done code. In the off chance we break it unintentionally, they tell us exactly what’s wrong.

And so I wrote what was supposed to be my final patch: one that fixed Wine and marked the tests as no longer todo. In true open source fashion, I sent it to a friend for review, where he promptly informed me that, while my new tests were passing, I’d created a place where Wine could crash. The solution is, unsurprisingly, yet more tests to see how Windows handles the situation (keeping in mind that sometimes Windows handles things by crashing). This is typical in Wine development: your first attempt at a patch often results in mere discovery that the problem is harder to solve than you thought.

The real world

None of this actually matters, of course, unless the bug I’d fixed was actually affecting a real application that someone would want to run. Did I actually fix anything useful? I don’t know. It’s not exactly easy to get a list of all Windows applications that rely on edge-case behavior of shlwapi.dll’s StrFromTimeInterval function, but at least Wine is more correct now.

Apparent correctness isn’t the end-all of software development, of course. It’s possible doing something seemingly correct in Wine can make things worse: if my initial version of a fix slipped in to the code, for instance, an application could go from displaying a slightly wrong string to flat-out crashing. That’s why unit tests are just one part of software QA – you still need peer review of code and actual application testing.

Part of something greater

Incidentally, the whole experience reminded me of a blog post I had written over a year ago about modeling Wine development. My model was telling me that what I had just done was a bit inefficient: I made a modest improvement to Wine, but it wasn’t directly inspired by a particular real world application. Perhaps it would have been better had I tackled a more salient bug in a big name application, rather than polishing off some random function in string.c. But it wasn’t random: another developer recommended this particular code section to me because it was missing tests, and he noticed this precisely because in the past some untested behavior in a similar function was breaking a real application for him.

This function is now done. The coding was relatively simple by Wine standards – no need for expertise in COM, Direct3D, OLE, or any number of Windows conventions that O’Reilly writes entire books about. Wine doesn’t need experts: it needs a lot of grunt work from people like me. People willing to tackle one function at a time until by sheer attrition we end up with a test suite so exhaustive that everything can be simply guaranteed to work. That’s how we win in the end. That’s how real developers do it.

The problem:

Sometimes one version of Wine can run an application, but fail on others.  Maybe there’s a regression in the latest Wine version, or you’ve installed an optional patch that fixes one application but breaks another.

The solution to this is to have more than one version of Wine installed on the system, and have the system determine which version of Wine is best for the application.  This implies two different levels of usability – advanced users may want to configure and tweak which Wine runs which App, but mere humans won’t even want to know such a thing exists.

This is the reason why many people have created Wine front ends: they worry about things like patches and registry hacks and native DLLs so that users won’t have to.  You just click a button for the application you want to install, put in the requisite disc or password, and it does all the shaman dances for you.  Codeweaver’s, the chief sponsor of Wine, stays in business through sales of Crossover, which is basically just a front end to a special version of Wine.

So, these front ends exist to solve some very real and important problems. But now we have a new problem, in that we might have more than one front end — playonlinux, winetricks, and others all need to deal with this too.  In true open source fashion, we need to work together and come up with a standard.

My proposed solution:

Distribution packagers like me make a separate package, say, wine-hotfixes.  This package replaces (on Ubuntu via dpkg-diversions) the /usr/bin/wine binary, moving the existing one to /usr/bin/wine-default.

The new /usr/bin/wine will pass everything to wine-default, unless it detects the environment variable WINEHOTFIXES is set to something other than null. If it is set (to, say WINEHOTFIXES=”6971,12234″), then the script will look in /etc/wine/hotfixes.d/ and /etc/wine/hotfixes.conf for alternative wine versions that might make it happy.  In the case of a partial match it will prioritize bugs in the listed order.

hotfixes.d will contain a series of config files, one for each alternative version of wine.  These could be installed manually, but generally they’ll come from special packages — say, a version of Wine built with a workaround for that annoying mouse escape bug.  Each file will give a path (say, /opt/wine-hotfixes/yokos-tweaked-wine) and which bugs it hotfixes.  hotfixes.conf can specify a list of bugs that the default Wine already fixes, as well as which bugs to ignore (eg that are probably fixed by every hotfix).

Start menu items (.desktop entries) can then work exactly as they do now, except they will have the WINEHOTFIXES environment variable set, generally as created by a front end.  If the user has no alternative wine versions, or no wine-hotfixes package, nothing different will happen and everything will still use the wine-default.  If the user upgrades Wine to a version that fixes all the worked around bugs, the default will be used again (forward-compatible) — all that’s needed is for the newer Wine package to ship an updated hotfixes.conf.

The beauty of this is that a front end can specify a list of bugs to workaround without actually having a ready hotfix – if needed that can be handled by someone else.  Similarly, the hotfixed Wines don’t actually need to know about which app they’ll be running, as wine-hotfixes will do all the matchmaking.  We also keep /usr/bin free, except for one added binary.

Wrapping it all up

The real test of a design, of course, is if it can be any simpler.  With the above, we’ve got it down to a single configuration item depending on who’s using it — hotfixes.d for the packager, hotfixes.conf for a manual tweaker, and the WINEHOTFIXES environment variable for the front end author.

It is of course worth asking if we should be doing this at all.  Zero configurations are better than one, after all, and ideally one magic Wine package would run every application flawlessly.  But that’s what we’ve been trying for years now, and people keep feeling the need to write these front ends anyway — we’re clearly not doing well enough without them, so we might as well manage them and work together.  This way, at least everything is backwards (and forwards) compatible: environment variables mean nothing without wine-hotfixes, and if we ever do invent a perfect version of Wine all the applications installed by front ends will continue to work just fine.

That should wrap it up, unless I’ve missed something.

I introduced myself, and Brian quickly recognized that I could probably help him with a Wine problem.  He needed to run Internet Explorer 6 for the most benevolent reason of all: to prove to a client it didn’t work and that they shouldn’t bother trying to support it.

Internet Explorer in Wine

The good news is these days it’s very simple on the terminal.  On Ubuntu:

• Add the Wine Team PPA:
  sudo add-apt-repository ppa:ubuntu-wine/ppa
• Install the wine1.2 and winetricks packages:
  sudo apt-get update && sudo apt-get install wine1.2 winetricks
• Run winetricks:
  winetricks ie6 or winetricks ie7
• Run IE:
  wine iexplore

Using the WINEPREFIX environment variable:

Note that, just like in Windows, you cannot have both ie6 and ie7 at the same time, at least not in the same virtual C: drive.  Fortunately, it is easy to set up multiple wine folders (“prefixes”) for your version of IE to run.  We do this with the WINEPREFIX environment variable.

WINEPREFIX=~/.wine-ie6 winetricks ie6 to install ie6 into its own virtual C: drive
WINEPREFIX=~/.wine-ie7 winetricks ie7 to install ie7 into its own virtual C: drive
WINEPREFIX=~/.wine-ie6 wine iexplore to run ie6
WINEPREFIX=~/.wine-ie7 wine iexplore to run ie7

As an aside, Internet Explorer 8 doesn’t currently work in Wine, however once it does there will also be a winetricks ie8 for you to test with.

is now

Giving Windows applications their own Icon

Today, however, I want to show you another kind of icon that until now has been represented by a boring default diamond with a question mark.

The small numbers in the corner show the version of the program when it’s available.  This is a piece of metadata that Windows has supported for years, but since it’s not visible in the user interface no one’s really known about it.

The new icons will be available tomorrow in the next Wine 1.2 release candidate package from my PPA.  My intention in Maverick is to include them by default, before Wine is even installed.

Working together

I’ll be honest: Wine needs your help.  The above work was heavily based on community input (and code by Jan Nekvasil).  I’ve got some more visual and UI changes in store for Maverick as well, but what Wine really needs is help testing its release candidate.

We need to hunt down every application that worked perfectly in older versions of Wine and make sure they haven’t broken.  Literally millions of people will be using Wine’s 1.2 release for at least the next year, and if we allow even one major regression it’ll be the digital equivalent of the vuvuzela.

But while we’ve already found and fixed hundreds of regressions, there are a whole mess of applications we still haven’t tested.  Wine doesn’t really have a QA team to handle this kind of thing – it’s just you and me, folks.

So please, join the Platinum Regression Hunt.  It’s as simple as running your applications in Wine and telling us if they don’t work anymore.

I need the words above translated.

With the exception of a couple of languages they aren’t.  The translations are Ubuntu-specific at this point: they’re neither handled upstream nor through Rosetta.  I was planning to get rid of all of them (more on this later), so I never converted the menu items into our existing translation infrastructure.  But removing them is delayed until Wine 1.2 in some months, so for now what we have needs to be localized and I need to handle the translation manually.

Please post translations here or in this bug report.  If you can do it in patch form that’s great, if not it doesn’t matter.

When a Wine developer wants to implement support for playing mp3 files, the smart thing to do is to use a software library, in this case mpg123.  Wine is a good example of a large and still growing piece of software with a similarly expanding list of dependencies.

It’s my job to make sure all these bits work well together so that Wine can be built in a complete way for you – without mpg123, Wine would have to be built without full support for mp3s, and sound on some apps would simply break.  Sometimes this means packaging software that isn’t in the distribution yet.  Usually it means fixing bugs, and here we can see the true strength of the open source approach.

It starts with a bug report

I was ready to put out a Wine release with its new mpg123 support, but Wine wasn’t building on 64 bit.  This was a case where the mystery was more interesting because it was only partially broken: 32 bit worked perfectly.

I knew right away that it was probably an Ubuntu-specific issue, although other distributions (especially Debian) might also be affected.  I filed a bug upstream to try and nail down the specifics.  Everyone shares their relevant expertise here: a Wine developer notes that it’s a mismatched symbol-header problem, I figure out the cause is Ubuntu’s unexpected use of 32-bit headers on 64-bit systems, a community member provides a temporary workaround patch, an mpg123 developer tells why things were the way they were and how Ubuntu+Wine was the first to expose this problem, and Alexandre Julliard says what he’d ultimately like as a best practice.

This sort of collaboration is exactly what Open Source is about.  Everyone involved could just go to the same bug tracker and simply get to work – it didn’t matter that all 6 of us were in different continents working for different companies on different projects.  Everything is laid bare, easy to see and thus easy to fix.

Getting it done

At the end of the day, getting the fix out to users is what actually matters.  It’s really disappointing when great code like mp3 support just doesn’t happens because of something silly like an out of date distribution library.

So, that’s the final step in the chain – the mpg123 developer puts out a new version of mpg123, and then I have to package that up and put it into the latest Ubuntu alpha.  This is where Launchpad’s ability to have a single bug exist against multiple packages becomes especially helpful: by filing a launchpad bug I could track the changes I had to make to all 3 packages (mpg123, ia32-libs, and then wine).

Technical details for the interested

It’s worth noting that this bug could have been prevented entirely if Ubuntu had proper multiarch support.  Multiarch means the ability to install 32-bit packages directly onto a 64-bit system, rather than having to use special separate 32-on-64 packages like ia32-libs.  Multiarch has been a slow, gradual redesign under development in Debian and Ubuntu for years now, and it just barely missed the Lucid cycle.

This leaves us with the current implementation, which can at best be described as “a tremendous hack.”  32-bit libraries from needed packages are manually added to a giant list.  Then they’re processed by a script and all thrown into one gigantic, 600+ megabyte package named ia32-libs.  This means that any program which needs 32-bit compatibility (such as Wine) has to depend on this massive source package, and any time a single component changes it needs to be manually updated.

However, ia32-libs contains only the library files, not the header files, so when you build a 32-bit application on 64-bit Ubuntu you’re actually building with the header files installed from the 64-bit version of the package.  This is normally fine, as most header files are identical across different architectures, however this wasn’t the case with mpg123.  So we were using a 32-bit library that had different symbols than the 64-bit header file, and the build then fails with symbol errors.

There are quite a few applications for 64-bit Ubuntu that need to make use of 32-bit libraries for various reasons, however Wine is by far the most dominant.  This makes me, the one responsible for Wine, also the official babysitter of ia32-libs and similarly broken packages.  Wine is special in a way, though, as unlike other applications there is no way to fix it by just making it 64-bit — Wine will always need 32 bit libraries in order to run 32 bit Windows applications, which will remain incredibly common for at least another decade.

Wine was a victim of the proliferation of sound drivers on Linux.

At one point, Wine had separate sound drivers for ARTS, ESD, OSS, JACK, and ALSA.  None of them worked right, even if the user could figure out how to configure Wine to use the “correct” one.  Subtle changes to each API and differences between distribution versions made matters even worse.

PulseAudio was supposed to be a solution to problems like this by being the one true Linux Sound System, but in some ways it made the problem worse.  Now Wine needed a separate PulseAudio sound driver, and no one wanted to write sound driver code since the whole thing was a big mess to begin with.  Worse, the most work had gone into making the ALSA driver better, and there were a few technical reasons why it seemed like a good choice to stick with ALSA and use it as a default.*

So, the decision was made to go with ALSA and improve it a bit, and hope that PulseAudio’s ALSA-compatibility layer would work well enough.  That didn’t exactly happen – PulseAudio didn’t actually want to support some “abuses” of the ALSA API, as one dev put it.  Some users took to killing PulseAudio outright every time they wanted to use Wine.

Things got a bit political.  Was Wine at fault here, or PulseAudio?  Should Wine include a PulseAudio driver, even if it was poorly maintained?  What would the default be?  Also, what the hell were we gonna do about the Mac?

The solution, which in retrospect is pretty obvious

Wine wasn’t the only program facing this problem.  Anyone writing or porting a Linux game had to answer the question about how they would play sound – almost none chose to write 7 separate audio drivers.  Instead they all did the smart thing, and used a higher level library.

So, that’s the plan – Wine will use a sound library.  And then we’ll let them worry about actually talking to whatever sound system happens to be working best.  OpenAL is the smart choice here, for much the same reason that OpenGL was the smart choice for graphics.  We get Mac compatibility “for free” too.

Now, why didn’t this happen earlier?  Well, in fairness to Wine, OpenAL didn’t actually exist in a usable form until Wine already had sound drivers for a few years.  A similar thing happened with wikis and distributed version control systems – we eventually saw the light that new open source projects had been following for years, and the project is better because of it.

* As I understand it, a lot of Windows programs expected to be very close to the hardware and at the time many users were having issues with PulseAudio latency.

• We’re serious about a 1.2 release in the next few months.  As Alexandre said, “it’s time”.
• 64 bit support will be the major new release goal.
• For most people, having no regressions and working with way more apps than 1.0.1 did will be the actual feature they care about.
• Users will need to make a new Wine installation to migrate from 32-bit to 64/32-bit capable.  It is left to me to figure out and code a humane way to present this.  This is also a convenient time to make their Wine folder case insensitive.
• We’re giving up on separate Pulse/ALSA/OSS/Jack sound driver layers and instead doing the smart thing: passing everything to OpenAL.  Maarten Lankhorst will handle most of it.
• Packagers like me will have to work out CJK font support for our respective distros.  Wine will need manual links to whatever font is actually provided by the distro when it’s in a CJK locale.
• We need some help from Freedesktop.org for solving bug 10841.  Essentially we need a standard way of saying “hey reset the resolution when I’m done, even if I crash.” I was “volunteered” for the task of approaching the respective projects.
• We need user help finishing up the Tahoma replacement font.  It’s still missing some glyphs for certain languages and some users report it as “ugly”.
• VMware is an ally of Wine.  They use our test suite on Thinapp, and even contribute test fixes and new tests back to us.  Tests are just as good as “real” code.
• We need more users to run git update and winetest daily (with their screens unlocked).
• Alexander announced a new patch tracking system to prevent patches from getting lost in the nether (and for devs to know when a rejection occurs)
• By far the largest concentration of wine developers is in the netherlands.  It has something to do with being below sea level, which means global warming is an ally of Wine as well.
• Francois Gouget has promised to teach me every nuance to Winelib so that I can document it properly.
• I need to update about 10 different wiki pages to give them more modern instructions, such as how to get a good backtrace (eg on Ubuntu you install wine1.2-dbg).  A recurring theme to this conference seemed to be that if it’s written in English and concerns the Wine project, it’s probably my responsibility at this point.
• Owen is going to setup planet.winehq.org, since he’s done it before for another project.  There you’ll see my blog and a few others.  I might actually update more than once a month too!
• Andre Hentschel is working on a Wine port to ARM.  I’ll see if I can get him a first generation Pandora (I had the misfortune of preordering two about a year ago, and they may actually arrive sometime next month).
• Jeremy White wants help recognizing people who should come to wineconf.
• I need to make a proper message box to indicate that Wine is loading immediately after a user opens an application (currently there can be about 5 seconds of nothing, resulting in either frustration or opening the application multiple times).  Wine should be sending the signals it needs (with xdg message) already, we just need to have the desktop listen to them.
• I need to make a “Press” page for the website, including an email address for press@winehq.org which will probably direct to me possibly several others.  I’m curious how other projects have handled this – is it a good idea to have multiple press contacts?
• All documentation needs to be moved to the wiki.  From there it shouldn’t be too hard to setup a wiki->pdf->docbook/xml conversion script and have the documentation build automatically based on the latest wiki pages.  Then we can start shipping it with packages again (since offline documentation still matters somewhat).
• I’m going to experiment setting up Redmine for the Wine project.  If I can get it running well on a separate server and have it fully import Bugzilla, the Wiki, and our Git repositories then we can switch to it and start using its nice management features.  Budgetdedicated will provide the server.
• The winehq.org Donation link needs to go to a proper web page that actually says where the money goes and what we’ll do when we raise enough.  Currently it just dumps you off to paypal without explanation.  Amazingly, some people actually give money (enough to partially fund WineConf).  This task is also mine.

I need to finish preparing for the Ubuntu Developer Summit that starts in a few hours, so the bullet points will have to suffice for now.  I’ve got my work cut out for me it seems.

Sutor said they need to focus on usability, stability, security, reliability, performance, with some cool thrown in, as well…  I think making it a complete drop-in replacement is a dead-end strategy.

Sutor’s speech on desktop Linux has inspired me.  Accordingly, I’d like to present the…

Bob Sutor Guide to Running a Marathon

• Overall fitness is important on race day. You’ll need to focus on your endurance, your heart, your leg muscles, your feet, your abs, your waist, and some arms thrown in too.
• Running form is important on race day. You’ll need to focus on your breathing, your leg motion, your arm movements, the length of your strides, and some pacing practice thrown in too.
• The training routine is important on race day. You’ll need to focus on finding other runners to practice with, self discipline, getting a top personal trainer, and some knowledge of the track thrown in too.
• Don’t try and win the race.  That’s a dead-end strategy.

You’ll note that Bob left out “compatibility” from his list of things to focus on.  A strange omission, coming from the company that witnessed IBM Compatible computers dominate the PC industry without actually being sold by IBM.  Compatibility then meant being able to run Microsoft with the same hardware and software.  It still does today.

We’re pretty good at hardware.  My girlfriend got confused by the manual that came with her latest printer: it gave 12 steps for Windows, another 12 for Mac, and then an additional 6 for both.  There were no Ubuntu instructions.  Turns out all she had to do was plug it in, the first thing she would have tried if there wasn’t a scary manual making her think printer setup required some sort of shaman dance.  This is a good problem to have, although having to tell your girlfriend to stop shaking her hips at the printer is perhaps an even better problem to have.

Winning

Hardware works, but we haven’t won yet.  We could make every single graphics card, printer, digital camera, and music player work out of the box flawlessly and we’d still miss the market completely. We wouldn’t actually be compatible.  What we have is a software problem. That’s why I work on Wine.

IBM management has never really believed in Wine as a technology, even though their own engineers find it useful. Jeremy White has even said that IBM Managers weren’t very helpful.

For my part, I’m going to ignore IBM’s failed attempt at leadership and focus on what actually matters: making things work for users. Only if we become compatible can we be at least as good. Then people give us a try, and notice we’re actually better. That’s when we win.

Advice for those giving presentations

After rewatching the above, I can say the following things with confidence.

1. The fewer words on your slides, the better.
2. Bullet points aren’t short enough.
3. If you start with a borrowed joke it will come off awkward.
4. Unless you can’t be funny, write your own jokes.
5. Never let someone control the slides for you.

Wine Bug Day this Sunday

Now, while I have your attention, how about you participate in the next Wine Bug Day for this Sunday?  It’s very simple – mostly we’re just downloading random games and playing them for a bit to see if a bug is still there.  And it all helps Wine development, so it’s good.  Check out the wiki page and join us!