Testing before 1.0.0

[Greendogo]

Question Zini, are we doing a Beta cycle after we are 1.0-feature complete, in order to fix up all the bugs and any performance problems?

[raevol]

Question Zini, are we doing a Beta cycle after we are 1.0-feature complete, in order to fix up all the bugs and any performance problems?

I really think we should do this, and when we do, we should do real testing with sets of test cases for each beta version, and such. Just my opinion. It'd be nice to have the 1.0 release be super polished.

[TorbenC]

Question Zini, are we doing a Beta cycle after we are 1.0-feature complete, in order to fix up all the bugs and any performance problems?

I really think we should do this, and when we do, we should do real testing with sets of test cases for each beta version, and such. Just my opinion. It'd be nice to have the 1.0 release be super polished.

Well I certainly hope we do O.O

[Greendogo]

I'm sure there's no question of making it super-polished before we release 1.0 (I hope), I was just asking if we were going to call it a Beta, and we'd make sure everyone who uses it knows that they should be bug hunting, and have speed improvement tasks on the tracker and whatnot.

[Zini]

Sorry, guys. But I think some of your ideas are a bit unrealistic. Of cause we will have an extended public test phase before releasing 1.0 (whatever you want to call it and btw. I really don't care about that). And we should certainly aim for releasing without known bugs.
But we can't expect 1.0 to be bug free. Not for something of that (mostly arbitrary) complexity. Eventually someone will run into a rarely used combination of features and things will blow up. It doesn't matter how careful we test. There will be edge cases that we miss.
Even beyond that it would be hard to have a fully polished 1.0. We would require several people doing complete playthroughs (not just the mainquest, but with coverage of most of the content). This typically takes several months and I don't see that happening with a beta.
OpenMW 1.0 should be playable for the majority of people (and that very well), but I can pretty much guarantee you that we will need a 1.0.1 release at least. Overall I think we should adopt something like the LIbreOffice release scheme when we reach 1.0.

[TorbenC]

But we can't expect 1.0 to be bug free. Not for something of that (mostly arbitrary) complexity. Eventually someone will run into a rarely used combination of features and things will blow up. It doesn't matter how careful we test. There will be edge cases that we miss.

I personally wasn't saying "no bugs at all" that would be a ridiculous goal, all games have bugs. I personally thought they meant simply doing extensive testing before 1.0 so 1.0 is as bug free as we can get it on our own before it goes out the door. Seems a bit strange to me that 1.0.1 would be your goal for this polish as apposed to 1.0.

Pretty much this is the case I see, you release version 0.99.0 or w/e is the final pre-1.0 release as beta and all the testing happens there, there is no point in jumping to a 1.0 just because you think the testing won't be done in beta, in fact it seems a bit more clear to the general public that 1.0 is the finished release imho, but looking at what you said it appears that 1.0 would technically be the beta release.

I am pretty positive there will be a 1.0.1 release too, but planning for it seems a bit Bethesda-like to me *cough* Skyrim day 1 patch *cough*.

[Zini]

If we know that it will happen, then we can as well plan for it. The 0.99 stuff never made any sense.

[psi29a]

The illusion that 1.0 means 'finished' which is proven time and again to be more like 'good enough'.

For us, 1.0 means 'feature complete'. Bugs included.

[TorbenC]

The illusion that 1.0 means 'finished' which is proven time and again to be more like 'good enough'.

For us, 1.0 means 'feature complete'. Bugs included.

Okay

[raevol]

For us, 1.0 means 'feature complete'. Bugs included.

In the industry, that's actually the definition of a beta. I don't see why we can't get to "feature complete", and then take a month to test as much as possible, promote for testers as much as possible, and generally clean it up as much as we can. Of course 1.0 isn't going to be bug free, but I don't see why we can't take a little time to put some effort into it. Slapping 1.0 on it and kicking it out the door without a little extra care is just going to give the project a bad reputation, when all people who've been waiting for release day to try it power it up and it eats their machines.

Just my opinion though, as someone who's worked in QA, and who's been using open source for quite a few years now. We already have a tradition of throwing stuff out the door untested, I guess why change that just because we'll be doing the most public release the project will ever have?