Modelling Regressions

If you’re one of the 130,000 or so people who have been using Wine beta releases, then at some point you’ve probably run into a regression.  Version 1.1.12 ran TerroristExploderPro fairly well, so you upgraded to Wine 1.1.13 only to find that it functions worse than TerroristExploder Home Edition.  This is a regression.

Regressions are best caught quickly.  When we have a regression, it means some “improvement” we made is actually very wrong, and the longer we go without realizing there’s a problem the harder it is to figure out which change actually caused it.  This was the main reason why Wine first built an extensive Applications Database – not so users could find out if their application worked, but so developers could find out when it stopped working.

An example regression

Suppose we have an incomplete “stub” function in Wine for telling the time that just returns a blinking 12:00.  This could be good enough; rather than crashing outright when Photoshop asks for the time, instead the wrong time is printed in some log file the user never sees.

Now, suppose a Wine developer comes along and sees that some other application, Baby’s First Clock, doesn’t work because a blinking 12:00 isn’t good enough.  So that Wine developer tries to implement the time function.  He can’t, however, be exactly sure how to do it since Microsoft’s documentation is often inaccurate or even nonexistant.

So, he makes his best guess, and tells the function to return 6:00 PM when it’s dinnertime.  He tests Baby’s First Clock and it seems to work, so the patch gets committed and a new version of Wine is released touting advanced Baby’s First Clock support.  By doing this, however, we may have broke Photoshop: because it expected 24-hour time, Photoshop now crashes whenever it sees 6:00 PM instead of 1800.

Preventing regressions with an automated test suite

Preventing regressions has become an important part of Wine development, however we’re still pretty bad at it.  One approach that’s had partial success is to write a big automated test suite. In theory, if we had a magic way of running Photoshop every patch then regressions like this wouldn’t happen.  Currently, Wine’s test suite, while very large, is not very reflective of real-world applications; instead, individual functions are tested to make sure we do the same thing as Windows.

In theory, if we had one of these individual tests for every case for every function, Wine would never have any regressions.  In practice, we miss a few, and real world applications break every now and again.  Actually, that’s not quite true – Wine would still have regressions even with a complete test suite; some tests we expect to fail because we haven’t written the functions yet, but we might be accidentally relying on broken behavior when we write another function.  It’s like painting a wall light brown to match the ugly yellow carpet – when you finally replace it with something good you may find your wall looks as ugly as it sounds.

Regressions can also come from passing well made tests.  Performance regressions are the best example: at one point a certain direct3d shading function was “stubbed” out and did nothing.  Most gamers didn’t care, since all it meant was the color of the beards on the terrorists was a slightly duller shade of brown.  However, this stub behavior (correctly) caused a Direct3D test to fail, so the shading function was implemented and the test passed.  Unfortunately, the implementation was very slow and now instead of seeing 50 dull beards per second gamers were seeing 5 shiny beards per second, making the game an unplayable albeit beardy slideshow.

Wine’s test suite grows as Wine development progresses.  In many ways, having a “complete” test suite is almost the same thing as having a complete Wine – once we know exactly what to test for we also know how to write our implementation.

A test suite, however, can only get you so far.  Until we have a nearly complete test suite that can expose all the subtle problems in partially broken applications, we’re still going to run into periodic regressions.  For that, we need another tool – users need to actually play the shiny beard game.

Preventing regressions with stable releases

The most effective way to prevent regressions is done by all serious projects: the stable release process.  The process starts with a feature freeze: no new code features are done, and all those “implement this” regressions just can’t happen.  Baby’s First Clock stays broken, but Photoshop won’t break either.  We stay in this freeze until we’re confident enough that anything that ever worked in Wine still works in Wine – that is, we fix Photoshop.

That’s what Wine 1.0 was about: it was the first release that actually made that promise.  It took 15 years, but this made a certain amount of sense given how visibly incomplete Wine was – there was always some new feature about to go in, and so many applications were “almost” working.  A year and a half later, they’re still almost working.

Wine 1.0 came out shortly after Ubuntu Hardy (8.04), so I had to ship a Wine beta version and then later replace it with a stable release update.  As a distribution meant for human users, Ubuntu tries to only ship stable software in its releases; this way, end users don’t have to worry so much about regressions.  Attracting new users with features is important, but not scaring off our existing users is important too – that’s the kind of innovation that gets it right.

This need for stable releases is why Jaunty (9.04) will still have Wine 1.0.1, and it’s why I’ve been secretly lobbying for speeding up the next stable Wine release for over a year now.  “If Wine 1.2 can come out in the next six months,” I’d argue, “then we can ship it in the next Ubuntu and end users will be able to actually see some of these improvements you’ve been making.”  This is the same reason Mark Shuttleworth has been calling for more synchronous freezing of upstream projects – if everyone releases around the same predictable time, then we maximize the amount of new but stable software users actually get.

Well, it looks like I may finally get my way in the Karmic time frame.  Wine may actually make that 1.2 release happen, even if it’s not as “complete” as the project’s leader originally envisioned.  So look forward to Wine and Ubuntu being particularly powerful come 9.10, and I’ll do my part to help it along the way.

8 Comments

JohnApril 16th, 2009 at 12:55 am

Great explanation of the software development process and pitfalls! I love the example of a shiny beards game too :) Where can I get teh softwares?

Thanks too for your hard work on Wine. It has saved my bacon a few times in my battle to eschew Windows at work!

Vincent PovirkApril 17th, 2009 at 11:31 pm

You’re oversimplifying.

Even when changes are absolutely correct, they can cause regressions. Real life example: I implemented (in my local tree) some functions IE7 uses in its favorites implementation. The implementation is (to the best of my knowledge) completely correct. Now IE7 crashes (in my local tree) every time I try to use the favorites.

It happened that Wine is broken in other ways. Because Wine has gotten better, IE7 is actually attempting to show me favorites now, and that crashes for an entirely unrelated reason. It was always broken; I had just never seen it. Before, it had always given up and shown me a blank space.

In this case, my changes would cause a regression as a user would define it: IE7 used to not crash, and now it crashes. Objectively, Wine would be improved. I would even argue that IE7 is behaving more correctly, as it always tries to draw favorites on Windows.

The test suite can (and should) only prevent the kind of regression that happens when Wine gets worse. Every change is supposed to make Wine better than it was before. That’s an ideal we haven’t met, but the test suite makes it a lot easier to do.

That ideal is important. We have to be willing to break applications in the short run (but not during code freeze) so that in the long run Wine can improve its compatibility with Windows.

I don’t think “all the applications that used to work will work in this version” was ever promised in 1.0. If there’s a promise that makes sense, it’s “all the features that used to work correctly will work correctly in this version”, or “we don’t have any major regressions in this version”. I don’t think anyone has made that one either.

Bug 13891 is a major regression that did get into 1.0. There are probably a few others.

Regressions are rarely as easy to fix as reverting a patch. Patches that are completely wrong rarely get into Wine. If we revert a patch that’s partially wrong, we lose the part that’s right too, and Wine gets worse. So we have to fix the regression without giving up the improvements that were introduced. That can be difficult.

And by the way, your lobbying about speeding up the Wine release process so 1.2 can get into is not so secret.

YokoZarApril 18th, 2009 at 3:18 am

I sort of included your example when I said that regressions can come while still passing well made tests. In the case of your internet explorer code, we were inadvertently relying on it not trying (the broken behavior) for the rest of the places (that were also crash-broken, one of which you fixed). You’re right though – this kind of patch should go in, although perhaps it should wait to go in at the same time that the other patches that are needed to prevent the crash (or maybe they could be restubbed to different defaults).

If we had an autohotkey magic test for internet explorer, it would have caught the regression too. That would be nice to know even if we commit the patch anyway, as it tells us an important new area to work on.

As for bug 13891, it wasn’t reported until after 1.0 came out, so we didn’t knowingly release a regression ;)

“And by the way, your lobbying about speeding up the Wine release process so 1.2 can get into is not so secret.”

You don’t know about the secret bags of money I’ve been handing out.

suetTustyMay 4th, 2009 at 1:35 am

hh.. cognitively )

CoJoncoSeptember 12th, 2009 at 4:34 pm

I am considered an extremely intelligent person by my peers, yet I could not begin to feel comfortable carrying an axe in the style you so easily manage to the challenges of overcoming regression. It’s a fight that seems to have most rounds going to that good looking guy in the other corner. He always seems to win and he brings the support of international fans while you appear a bloody mess, barely able to stand. Yet when it comes to heart, you get my applause. Not something I could do. I still like to accomplish, feel good about what I have done, and walk away, on to other ventures, other challenges, always with a chance to win.

You’re gonna be on the mat, on your back, bleeding and lookin’ like you just got run over by some monster, yet you keep coming back fore more. Wow! You guys that chase regression–you are TOUGH!

Keep it up.

Kathleen MurphyOctober 31st, 2009 at 1:54 pm

Ubuntu has been a disaster for me because the only program I truly need only runs in Windows.

Several releases ago, loading it in WINE worked and I got my smile back. Wish I could remember which release but I didn’t know about ‘regressions’ then so didn’t note it.

Then another upgrade and down it went. I didn’t know then that I could have ‘frozen’ the version that worked. Never ran right again though it did run.

When I re-installed WINE after a major crash followed by a new install of Karmic, nada. Found a near equivalent (close but no cigar) in Open Source and I’ll just have to make do.

Ubuntu was sold to me as an ‘out of the box’ plug and play. It just doesn’t work that way! It’s a system for those of you who enjoy spending at least a quarter of your computer time messing around in a terminal. Next time I have a few bucks, back to Microsoft for me.

nameFebruary 8th, 2012 at 8:20 pm

comment

RolfJune 9th, 2012 at 12:24 pm

Kathleen: I feel the same way. I installed Ubuntu because I did not want to buy Windows 7, I did not want to get an illegal copy (although I could do so easily) and most of all I did not want to have to install drivers on top of that! Ubuntu just worked from the first boot.
But then, Ubuntu is OK if you don’t mind half baked solutions, if I can say so. It exhibited numerous unrelated bugs, some appearing out of nowhere, or also design idiosyncrasies (first time I use this word) and inconsistencies.
Ubuntu has come a long way. Not so long ago you had to “mess with the terminal”, edit countless config files, just to get the sound working (and that’s if you were lucky). And it was the expected, normal behavior! And even then did people tout Linux as a better alternative to Windows! Right!
It has progressed much since, but it still feels somehow “half baked”. It is not the first time I install a Linux distro, and I was a little prepared for that. But it somehow always ends up fooling me, as the first run is bedazzling and impressive and then problems start to appear. Anyway, I know that I will buy a copy of windows when I’ll have the money to. Or better, I’ll just buy myself a Mac. For now I’m sticking with Ubuntu, and I have to say, for the first time, that it’s actually acceptable, that it turned out to be more then an experiment – it is serving me this time, and being useful!
Indeed Linux has progressed a lot, while Windows not so much – I mean I still don’t see the big difference between Windows XP and Windows 7! I would have stayed with XP, but I bought myself a new laptop that only supports Windows 7. The user base of Linux has also expanded, and users are more knowledgeable and know what to expect.
I believe there is a future in Ubuntu and other distributions. The Linux model just seems to work somehow. But I’m not counting on it, and who can blame me!
Anyway Wine is very important, as it tires to bridge the gap between the two systems. You just have to wonder how a group of volunteers are going to keep up with reverse engineering proprietary protocols created by Microsoft. Maybe they’re better off investing this energy somewhere else, like in re-designing the system, or reviewing the Gnome or other user interfaces, etc. But this is open source… can’t tell people what to do!

Leave a comment

Your comment