New Feature Qualification

Early this year Oracle released  MySQL 5.6 - Best MySQL Release Ever. This release delivered not only quality, but also quantity in terms of number of features. See a comprehensive list here . The blogs below also refer to the massive changes introduced in 5.6
http://www.mysqlperformanceblog.com/2013/01/27/mysql-5-6-improvements-in-the-nutshell/
http://www.flamingspork.com/blog/2013/03/05/mysql-code-size/

It is no mean task to deliver so many features with high quality that too for a feature rich product like MySQL. This was made possible by the increased focus given to testing during 5.6 development . Testing was integrated far better than in the past with MySQL processes and test team was grown significantly. After the first few milestone releases developers and other stake holders started seeing benefits of the new processes and that resulted in better cooperation between development and test teams which in turn resulted in a bunch of happy testers :). In this post I will take you through the process that was established for new feature qualification.

Features are delivered through milestones which Tomas Ulin has explained very well at http://insidemysql.com/the-milestone-release-model-revisited/. MySQL Server QA has the following goals for new features

• Complete functional and non functional test coverage of changed and new functionality
• No regressions
• More than 80% Code Coverage

QA involvement starts as soon as the requirements and specifications of the feature are finalized by the development team. QA reviews available documents and provides feedback on the design, usability, testability etc. A discussion follows with the developer and changes are made as needed to ensure that the feature can be tested.

Once the specifications and requirements are acceptable QA starts working on the test plan.  First QA documents all scenarios that needs to be tested. This includes stand alone tests, integration tests, non functional tests etc. Next step is to identify tools for automating the scenarios. Most commonly used functional test tools are

• MTR(mysql-test-run)- Used to cover simple to medium complexity test cases
• RQG(Random Query Generator) - This is a very powerful tool and is used for concurrency tests, generating large number of SQL queries etc

MTR based tests are added by default for any new feature. RQG use started from 5.5 and it gained wide acceptance in 5.6. At least 90% of the features in 5.6 have gone through RQG testing and this helped weed out many bugs which would have been nearly impossible to find using MTR.

While the developers are working on the product code , QA engineers start working on the automated tests, test infrastructure improvements etc.Final round of testing starts after the feature has passed code reviews. This phase can last anywhere between a few days to months depending on the complexity of the feature, stability of the code etc.

Features gets signed off only when the following conditions are met:

• No open bugs in the new feature - This is very strictly enforced and even minor bugs are not allowed. We believe that bugs are easiest to fix when the code is new and hence this can help us deliver features that are of high quality.
• No regressions - A feature gets developed on a tree which gets tested regularly through a continuous integration testing tool. Any regressions are detected and fixed before sign off.
• Acceptable Code coverage numbers - Code coverage report is generated for the changed lines of code and the minimum expected coverage is 80%. Most features have coverage of more than 90%. Any uncovered lines of code are analyzed and wherever possible new tests are added to increase code coverage.
• All new tests are added to the automated regression suite.
• All planned testing has completed with satisfactory results.

This process worked really well in 5.6 and we will use our experiences to improve this further and deliver an even better 5.7 !!

MySQL @ Bangalore

We are working towards connecting with the MySQL user community in Bangalore. Please join us on the facebook group MySQL User Camp to get updates on upcoming activities

The first meet was held @ Oracle, Kalyani Magnum on March 22nd. Sanjay Manwani and Karthik P R have blogged about it

http://remotemysqldba.blogspot.in/2013/03/bangalore-mysql-user-camp.html
https://blogs.oracle.com/mysqlenterprisebackup/entry/report_bangalore_mysql_user_camp

Next meet is planned for June 2013.

Testing @ MySQL

MySQL has always given great emphasis on quality.  This has got even better in the last few years with increased investment in the dedicated test team. The Server test team tripled in size during 5.6 development and all 5.6 features have undergone thorough testing at the hands of the QA team.

I have more than 15 years experience in this industry and have worked on different products and technologies as part of test and development teams. Compared to most others I find the developers in MySQL very involved in testing. Here the developers are expected to write functional tests for their features and bug fixes. This is how it was historically and this continues today in spite of the bigger test team. This model has the following advantages

• Code is in reasonably good shape when delivered to test team
• Testers can focus on more complex tests since the straightforward tests will be written by the developers

Listed below are some of the focus areas of the dedicated test team

• Integration tests – Tests to check how the feature works when it interacts with other features like views, triggers, stored procedures , partitioning etc
• Replication
• Backup using mysqldump
• Concurrency testing
• Exploratory testing
• Bug hunting cycles for finding crashes and result differences 
• Upgrade/Downgrade 
  
• Non functional tests like performance, stress, longevity etc

In my next post I will explain in detail about the process  for qualifying new features

Welcome to my blog. I am Anitha and  is currently leading the testing team for MySQL Server in Oracle. This is a great place to be and I love every moment of my work on this great product with a bright  group of colleagues. In this blog I plan to write about testing in MySQL , testing in general and some other topics that are of interest to me.