New Feature Qualification

Early this year Oracle released  MySQL 5.6 - Best MySQL Release Ever. This release delivered not only quality, but also quantity in terms of number of features. See a comprehensive list here . The blogs below also refer to the massive changes introduced in 5.6
http://www.mysqlperformanceblog.com/2013/01/27/mysql-5-6-improvements-in-the-nutshell/
http://www.flamingspork.com/blog/2013/03/05/mysql-code-size/

It is no mean task to deliver so many features with high quality that too for a feature rich product like MySQL. This was made possible by the increased focus given to testing during 5.6 development . Testing was integrated far better than in the past with MySQL processes and test team was grown significantly. After the first few milestone releases developers and other stake holders started seeing benefits of the new processes and that resulted in better cooperation between development and test teams which in turn resulted in a bunch of happy testers :). In this post I will take you through the process that was established for new feature qualification.

Features are delivered through milestones which Tomas Ulin has explained very well at http://insidemysql.com/the-milestone-release-model-revisited/. MySQL Server QA has the following goals for new features

• Complete functional and non functional test coverage of changed and new functionality
• No regressions
• More than 80% Code Coverage

QA involvement starts as soon as the requirements and specifications of the feature are finalized by the development team. QA reviews available documents and provides feedback on the design, usability, testability etc. A discussion follows with the developer and changes are made as needed to ensure that the feature can be tested.

Once the specifications and requirements are acceptable QA starts working on the test plan.  First QA documents all scenarios that needs to be tested. This includes stand alone tests, integration tests, non functional tests etc. Next step is to identify tools for automating the scenarios. Most commonly used functional test tools are

• MTR(mysql-test-run)- Used to cover simple to medium complexity test cases
• RQG(Random Query Generator) - This is a very powerful tool and is used for concurrency tests, generating large number of SQL queries etc

MTR based tests are added by default for any new feature. RQG use started from 5.5 and it gained wide acceptance in 5.6. At least 90% of the features in 5.6 have gone through RQG testing and this helped weed out many bugs which would have been nearly impossible to find using MTR.

While the developers are working on the product code , QA engineers start working on the automated tests, test infrastructure improvements etc.Final round of testing starts after the feature has passed code reviews. This phase can last anywhere between a few days to months depending on the complexity of the feature, stability of the code etc.

Features gets signed off only when the following conditions are met:

• No open bugs in the new feature - This is very strictly enforced and even minor bugs are not allowed. We believe that bugs are easiest to fix when the code is new and hence this can help us deliver features that are of high quality.
• No regressions - A feature gets developed on a tree which gets tested regularly through a continuous integration testing tool. Any regressions are detected and fixed before sign off.
• Acceptable Code coverage numbers - Code coverage report is generated for the changed lines of code and the minimum expected coverage is 80%. Most features have coverage of more than 90%. Any uncovered lines of code are analyzed and wherever possible new tests are added to increase code coverage.
• All new tests are added to the automated regression suite.
• All planned testing has completed with satisfactory results.

This process worked really well in 5.6 and we will use our experiences to improve this further and deliver an even better 5.7 !!