Update from the MySQL users conference

I just finished the MariaDB future planning BoF session, which was more about discussing the distinct flavor of the home made black vodka than about future MariaDB features. It was still a very good BoF and there is still time for those interested to add their future request to the plans.

I have never before spent such much time preparing a keynote, and it looks like my attempts to present something a bit different were appreciated. You can find a copy of the keynote and video of it at the MySQL conference web site.

I would, however, like to offer a correction. A couple of days before the conference I had been told by someone (someone that I had all reasons to believe was a reliable source) that there were only 54 of the original 400-450 MySQL Ab people left at Oracle. I asked a lot of former Oracle employees if this figure could be accurate and everyone told me that that the figure sounded low, but they could believe it. I could not find anyone in Oracle willing to comment upon it before the keynote. Now I have finally been able to verify this and there is still closer to 200 original MySQL Ab people left at Oracle working with MySQL.

My statement that it's only the InnoDB, NDB and most part of the replication team that are intact and that there is only 2 original core MySQL developers left is however accurate. My apologies for the wrong initial number of total people left, but I was in good belief regarding it!

To make things clear, I am not in any way trying to downplay the hard work the MySQL developers (and other MySQL people) are doing. They are doing an amazing job, with the resources they have at their disposal.

My point was that I am worried that the MySQL developer and support ecosystem is slowly falling apart because people are leaving Oracle and going to other companies where they are not anymore participating in the MySQL development. That is why companies like Monty Program Ab and SkySQL are important as we help keep the ecosystem together by having people working on the same thing they did before!

Apart from this, it has been a great conference and it's been a true pleasure to meet all of the 'old MySQL conference gang' yet another time!

Planning for MariaDB 5.6 at the MySQL Conference 2011

Planning for MariaDB 5.6 at the MySQL Conference 2011

A couple of weeks ago we had a Monty Program Ab and MariaDB developers meeting in Lisbon.

MariaDB 5.3 / 5.5 are now almost in beta (the last features will be pushed directly after the MySQL conference) and it was time to start considering what features should be in the next MariaDB release.

The meeting was open to anyone and we were lucky to have several other companies than just Monty Program Ab involved in the discussions!

During the meeting we created a list of features we would like to see in the MariaDB 5.6 release.

Some of the features are already allocated to developers. Others are waiting for developers willing to do the development or sponsors willing to pay for getting things done. If you are interested in doing or sponsoring some MariaDB development, check the list out and assign your name to the features!

This list is still open for new additions until 5.3 / 5.5 is in gamma. (In practice probably a few months). If you think something critical for you is missing, feel free add the things you want to see done. If you can commit to do them, then it's of course even better!

The MySQL users conference 2011 is starting next week. I plan to organize a MariaDB BoF where we can together discuss the plans for 5.6 and collect requests and suggestions. Black drinks and chocolate will be served, so come prepared!

Most of the developers of Monty Program Ab will be at the conference and we will also have a booth so you should have no problem in finding someone to talk to about MariaDB there.

Monty Program Ab is also organizing a MySQL/MariaDB/Drizzle store engine summit. Follow the link if you are interested in attending!

Don't miss my keynote on Wednesday morning! There should be a couple of surprises for everyone in it...

Looking forward to another interesting conference!

Come to Lisbon and meet MariaDB developers

The next Monty Program Ab company and MariaDB developer meeting will be in Lisbon, Portugal between Friday 11 March and Monday March 14, 2011.

The topic of the meeting is to go through what has been done in the MariaDB and MySQL code recently and to plan the features for the next main MariaDB release.

The developer meeting is open to anyone who wants to participate. If you want to know about, affect, or participate in future development of MariaDB or if you just want to hang around and have a good time with the MariaDB developers, feel free to come and join us in Lisbon!

We are just now in the process of booking a hotel and will publish the details on our MariaDB Developer Meetings page as soon as it's decided.

What is still not decided is what restaurants to visit, where to have the actual meetings (we are just now talking with some hotels about this, but we are also open to suggestions) and what to do on the Monday "excursion day'. If you know Lisbon and have a suggestion that would work with a group of 20-40 people who enjoy good food, good company, and want to experience 'new (unexpected?) things', please let us know!

You can reach us at 'community at askmonty dot org'.

SQL 99 Complete, Really now freely available

I am very happy that Daniel Bartholomew recently finished his task of getting the excellent book SQL 99 Complete, Really into the askmonty.org Knowledge base (KB). Even if the book is a bit old, it's still one of the best (maybe even the best?) reference books about SQL.

A big thanks to my friends Peter Gulutzan and Trudy Pelzer for making this possible!

This is however just the beginning. Now we will start cross referencing the KB and the book and also moving appropriate sections from the book to the KB to create better KB articles. (We also need to add some links to make it easier to read trough the book online).

One of the goals with the KB is that over time it will become a full reference and user manual, not only about MariaDB and MySQL, but also about other databases, database systems and database languages.

If you own the rights to some database documentation that you think would benefit the users of the KB, please contact us about getting it included. We are especially interested in MariaDB and MySQL related books and are even willing purchase the rights for these to get them published in the KB.

One of the great benefits of the KB is that if you can't find an answer, you can ask a question at precisely the point where you think the answer should be and get an answer there from the people working on or using the KB. This will allow others to find the answer without having to ask the same question again. The KB is thus not only a manual, but also a replacement for forums.

When it comes to MariaDB in the KB, I am quite happy about the progress. The documentation is growing rapidly and we have been able to quickly create new articles to answer most questions that have been asked in the KB, in the #maria IRC channel or on the Launchpad MariaDB mailing list.

If you want to participate in creating free database documentation and/or help others by answering database questions please join us in working on the KB.

Proposal for MariaDB trademark policy

Within Monty Program Ab we have during this year had a lot of discussions about how to go forward with the MariaDB trademark. It's been clear that everyone wants to have something that is substantially freer than the MySQL trademark to ensure the survival of MariaDB whatever happens to Monty Program Ab.

We wanted to make something that should work well, both for open source and commercial usage (and yes, I know that in some cases these are one and the same), which is not very common with many other trademark policies. My belief is that by having a very liberal trademark policy we will create a bigger ecosystem around MariaDB which will help all of us.

Now we have had a couple of internal drafts (with heavy input from our community advocates) and we have released our first public draft.

Please give us feedback about this either on my blog or the knowledge base so that we can take your thoughts into account for our final version!

A quick look at MySQL 5.5 GA

Congratulations to Oracle and the MySQL team for getting 5.5 out the door!

I first have to acknowledge that I have not been able to follow the MySQL 5.5 development as closely as I would have wanted during the last 2 years as most of the planning of MySQL 6.0, 5.4 and 5.5 has happened behind closed doors, without insight for the community. The commits have been open, until recently, but it's not easy to follow what is happening just based on the commits. I am sure that I am missing below some of the important features in 5.5 and forgetting to acknowledge some of the people that have done great work on 5.5.

That said, the 5.5 release contains some things that a lot of heavy users of MySQL have waited a long time for. Here follows some of the main things that I know about.

InnoDB improvements:

• The new version 1.1 of the already improved InnoDB plugin is now the default (and only) InnoDB (in 5.1 the InnoDB plugin version 1.0.x was an optional engine).
• Multiple buffer pools and rollback segments, splitting of log_sys and flush_list mutex, and other improvements that together give a major contribution to the improved scalability and performance of 5.5.
• Thanks to the InnoDB team for continuing to do impressive work!
  

Adding DDL locking.

• In earlier MySQL versions LOCK_open, which protected table opening (among other things), was one of the main things that stopped MySQL from scaling on many CPU's, especially when many tables were used. By introducing proper DDL locking this issue is now mostly fixed.
• A big thanks to Konstantin Osipov and Dimitry Lenev for the hard work they have done in getting this to work.
  

Reducing scalability bottlenecks in the core server, such as LOCK_open, LOCK_alarm, LOCK_thread_count, Query cache and binlog mutexes and so on.

• Thanks to Mikael Ronström for driving this!

A big speed improvement for multi-cpu on windows.

• This was actually the result of a minor bug fix for how mutex and condition variables were used on Windows. By using native Vista primitives instead of homegrown ones, a big speedup was achieved.
  
• Thanks to Vladislav Vaintroub for getting this fixed!

Optional Semi-synchronous replication (the server will wait for at least one slave to catch up before continuing) and replication heart beat are good improvements for those that use replication.

• Thanks to Google and Mark Callaghan's team for providing the initial patches for this.
  

The performance schema gives a lot of more insight in which mutex and IO calls are the bottleneck for your queries.

• Thanks to Marc Alff for driving and developing this.
  

Pluggable authentication

• Done by MIT student R.J.Silk and Sergei Golubchik for MariaDB 5.2. Donated to Oracle by Monty Program Ab.
  

And of course a big thanks to everyone else who has been involved in this and been fixing bugs in 5.1 and later.

Some good links for more information about the improvements in 5.5:
- New In MySQL 5.5 Performance_scale Unleashed Presentation
- Introduction to MySQL 5.5
- MySQL 5.5: What's New in Replication

It's now almost exactly 2 years since the MySQL 5.1 GA release, so what's the verdict of 5.5?

The two things that impress me are the work on the InnoDB storage engine and the DLL locking. Most of the other new features are minor (technically) adjustments even if the results, like with the windows speed improvement, can be impressive. As a technical person I have a hard time getting impressed by something that could be done in a few hours and could easily have been done in 5.1.

What is worrying me the most is the things that are not in 5.5. Some if the things that were supposed to be in the next MySQL release are:

Pool of threads (instead of one thread per connection like it is in MySQL 5.1).

• As far as I know, this feature has been moved to appear in 5.6 enterprise and will not be part of the community server.

An open source, free backup utility and backup API that would work for all storage engines. This has probably been the most awaited feature of MySQL ever!

• This project was discontinued by Oracle the same day the feature was deemed ready to be pushed into 5.5.

In 5.5 some internal subsystems, like safemalloc, a portable memory checker, were removed.

Another worry I have is how long Oracle will be able to continue development of MySQL while important core developers continue to leave. Of the above, Konstantin Osipov and Vladislav Vaintroub have already left Oracle and there are not many old-timers left.

When it comes to MariaDB we are continuing to work making MariaDB 5.3 stable. MariaDB 5.1, 5.2 and 5.3 use the InnoDB-plugin compatible XtraDB storage engine, which already has many of the speed improvements of the new InnoDB plugin, but not yet the windows speed improvement.

We have already merged MariaDB 5.3 with MySQL 5.5 and we will release the MariaDB 5.5 tree shortly. It has taken some time as there are a lot of things done in MariaDB 5.3 that conflict with new code in MySQL 5.5 and we also have to add back some of the features removed in MySQL 5.5.

The next big decision is to decide which version we should release next as a stable release; MariaDB 5.3 or MariaDB 5.5.

The problem is that it's very hard to evaluate how stable 5.5 really is as that 5.5 has not been tested by the public enough and some of the bugs in the public bug system, like the critical Bug #33082, can't be accessed. The bug database shows 294 open bugs for MySQL 5.1 and 150 for MySQL 5.5. I haven't however had time to evaluate all public bugs, so I don't know how severe they really are. A lot depends also how many bugs will be filed during the next few weeks after the GA release.

Note: I corrected the above sentence after getting confirmation that the MySQL team has not yet switched to Oracle's internal bug system. This is scheduled to happen first in the spring. Thanks for helping me get the facts right!

The good news is that many of the bugs that I originally reported for the 5.1 GA release have been fixed. However there are still things that could have been better:

• regression in parser compared to 5.1.x. This was reported just after 5.5 was released, but something like this should have been noticed during testing.
• regression compared to 5.1.x.
• cannot select from innodb_trx when trx_query contains blobs that aren't strings
• Object names not accessible via case sensitive reference on lctn=1 systems
• Change in execution plan for count_distinct_group_on_key causes 400 % performance drop
  

And there are still bugs that were supposed to be fixed before the 5.1 GA which are still not fixed:

• cost calculations (in optimizer) are off
• Slowdown (related to logging) in 5.1.21 vs. 5.1.20
• Stored routines do not detect changes in meta-data.
• crash when federated table loses connection during insert ... select
• comparisons with Information schema tables don't honor collation
• Partitioning performance drops drastically with hundreds of partitions
• Event set to DISABLED and ON COMPLETION NOT PRESERVE is deleted at server start
  

So our plan is to continue to fix the last issues in MariaDB 5.3 and at the same time keep MariaDB 5.5 up to date. Early next year when it's time to decide which one to release as stable, we will have more information about the stability of MySQL 5.5. Let's all hope for the best!

In search of a BSD/LGPL/Apache licensed client library for MySQL and MariaDB

We have received many requests for a client library for MariaDB (and MySQL) under a license other than the GPL, and decided that it's now time to do something about it.

Some time ago we released the LGPL client library used in MySQL 3.23. This can be used by many applications to connect to MariaDB, MySQL, and Drizzle, but doesn't satisfy all the requirements one would like to see for a true replacement for the current GPL client library.

Minimum requirements are:

• It should be binary link level compatible with the current MySQL 5.1 client library. To use it, one would just have to re-link the application with the new library. No re-compilation of the source should be necessary.
• It should have all functionality of the MySQL 5.1 client library (including prepared statements).
• It should be able to connect to all current versions of MariaDB, MySQL and Drizzle.
• It should support all the new functionality in MariaDB's client library like new client functions and pluggable authentication.
• The license should be one of BSD/LGPL/Apache.
  

We have now started to actively plan for getting this done. The plans can be found in our Worklog.

We have already found some companies that are willing to sponsor this project, but not yet enough to get it all done.

If you are willing to donate some time and effort on this, please contact us at
'maria-developers at lists.launchpad.com' or 'community at askmonty.org' (we strongly prefer the public maria developers list).

If you are willing to sponsor this effort, please contact us at 'sales at askmonty.org' or use our contact form.

We will put all monetary sponsorship into the Worklog. As with all worklog items, if someone from the community delivers a working solution that is accepted into the MariaDB source they will get 60% of the so far committed money.