Are MariaDB tests adding anything extra over Oracle MySQL tests?

Posted on by

I grabbed all the tests introduced in MariaDB 5.5.32 (i.e. “bzr diff -rtag:mariadb-5.5.31..mariadb-5.5.32 mysql-test/” and some foo) and threw them in their own test file. I only kept tests for crashing bugs and ignored those that required plugins (there were two or three, but nothing major). So now I have a test file that should crash MariaDB 5.5.31 and probably before. But, the question is: does this crash Percona Server or MySQL?

While it is excellent to see the MariaDB guys including tests for their crashing bugs, are these MariaDB specific or do they affect other MySQL flavours?

I built a release build of top of trunk Percona Server and ran the test against it. I got no crashes. In a debug build, I got two. One was to do with REPAIR on an ARCHIVE table and the other was “SELECT UNIX_TIMESTAMP(STR_TO_DATE(‘2020′,’%Y’));”. I found the same thing for a debug build of top of tree MySQL.

All the other tests for crashing bugs, of which there were 14 – were MariaDB specific. So, out of 16 total, only 2 applied to Percona Server and MySQL.

6 thoughts on “Are MariaDB tests adding anything extra over Oracle MySQL tests?”

  1. I’d say your experiment shows how many upstream bugs MariaDB *fixes* rather than *creates tests for*, which might also be an interesting study, but not entirely the same. Mind you, usually we report bugs that affect MySQL at bugs.mysql.com, and wait for reasonable time to see if they are fixed. So ideally, your results would have come up empty. However, sometimes upstream bugs get stuck, in which case they might get fixed by MariaDB team, subject to available resources. That’s what your check finds. For example, the one about archive tables would be http://bugs.mysql.com/bug.php?id=69086.

    Besides, f your question is whether MariaDB tests add anything to Oracle tests, you need to compare your found X not to the number of MariaDB-specific tests, but to the number of Oracle tests. In other words, if you say that шn 5.5.32 MariaDB found and fixed 2 crashing upstream bugs, then to decide whether it adds much or not, you’d need to know how many such bugs were found and fixed by Oracle. While the “fixed” part is more or less simple, you can check release notes, the “found” part is far more complicated, since you can’t tell from the release notes who found what, even less so for crashing bugs. At least I can’t, I barely recognize mine in there.

  2. Maybe what I should do is run these tests against the MySQL version prior to the MariaDB version? That way it’d certainly be what bugs affected MySQL as well as MariaDB. (with my methodology above, if Oracle fixed very very quickly after MariaDB reported them upstream then it could have made it into the same dot release as the MariaDB fix).

  3. That being said however, the advantage of using MariaDB 5.5.32 as compared to using MySQL 5.5.32 still stand – there’s only two additional crashing bugs fixed in MariaDB rather than MySQL, with the rest of the crashing bugs being specific to MariaDB only.

  4. Stewart,
    Let me remind you that MariaDB 5.5.32 contains much more features that MySQL 5.5.32. The bugs for these feature should be checked against MySQL 5.6.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.