jcole’s weblog

This morning, Barack Obama was in Reno, at UNR for one of his Rally for Change events. Since I haven’t seen him in person before, I figured I would go up there and see him speak, take some pictures, etc. It was pretty awesome, and he gave a really good speech. I took a few hundred pictures, so here are a few select shots… I’ll get the rest posted soon:

 
 

Obama speaks to the crowd gathered at the Rally for Change event in Reno, Nevada on October 25, 2008.

 
 

The crowd at the Obama rally in Reno, Nevada.

 
 

Obama frustrated during a power failure in the middle of his speech.

As of today, Proven Scaling has launched an official blog, which we’ll all be writing entries on. Check it out and subscribe!

All of this talk of Drizzle (a fork of the MySQL server by Brian and Monty) has reminded me of a topic I have wanted to discuss for quite some time…

One of the things that sets MySQL apart (in, IMHO, a very bad way) from other Open Source database projects/products such as PostgreSQL (license) and Firebird (license) is that the MySQL documentation is NOT Open Source. The MySQL documentation is and always has been copyright MySQL AB, and “… use of this documentation, in whole or in part, in another publication, requires the prior written consent from an authorized representative of MySQL AB”. This presents a major impediment to forking the server: who wants to re-write many hundreds of pages of documentation on things that haven’t even changed? Even if you’re OK with all of your new code being GPL (and anyone forking ought to be), and never being able to dual license or re-license your MySQL fork (oh well), you will have to start with no documentation, or publish an errata against the official MySQL documentation (which will only go so far).

Does this mean that MySQL is not really Open Source? I would say not exactly, although I could probably be convinced either way. Others may say yes. But it does go a long way to making the point that some things may not be quite as “open” as they initially appear. What do you think? Can a piece of software really be Open Source while its primary/only documentation is not? Did you know that the MySQL documentation was not Open Source?

Kaj just announced that as of 5.1.25, MySQL Cluster will no longer be included in the normal MySQL packages. Instead, a new branch is being created for MySQL Cluster releases, where the first release is informally called “6.2.15″ but the releases are really named “mysql-5.1.23-ndb-6.2.15″. This new branch is based on the MySQL Cluster Carrier Grade Edition of MySQL.

Overall, this seems like a good idea — the needs for releases, schedules, pressures, etc., are at odds between MySQL Cluster and MySQL’s core. I am, however, baffled by the decision of how to release the new product: coupled with MySQL as a single monolithic package with compiled-in storage engine. After all, 5.1 has long been touted to have this amazing new pluggable storage engine architecture. Why not use it?

With Oracle/InnoDB recently announcing that they are decoupling from MySQL and releasing their storage engine as a plugin, this would make so much sense. The only thing I can think of is that MySQL and/or the MySQL Cluster team must think that the pluggable storage engine concept is not workable in its current state or easy enough to use… which I would agree¹ with absolutely. However, having a team within MySQL really pushing a product and using the pluggable interface to make their releases would help dramatically to make the interface really usable for the rest of the world.

Why not do it? Let’s hear it…

¹ My short-list of gripes, by no way complete: The error message interface sucks, no way to compile a plugin outside of the MySQL source (or even symlink it into the source tree), plugins are tied to an exact version of MySQL server, and no reasonable way to manage plugins in an OS context (RPMs, .debs, etc).

Yum is an extremely popular system to download, install, and update RPM-based packages from multiple repositories. Proven Scaling has launched a set of repositories to augment the existing central distributions’ repositories with packages our customers need for deploying MySQL-based systems. We’ve been working on it for a while, and have had many people making use of it. We are providing:

• RPMs of community and enterprise releases of MySQL for RHEL/CentOS, as built by MySQL and distributed on MySQL.com
• RPMs of community tools such as maatkit and innotop and their dependencies.
• Proven Scaling-created tools such as mysql_snapshot (an LVM snapshot-based backup utility).
• Difficult to find RPMs of Perl libraries (dependencies for other scripts, such as innotop).

Here are the yum repositories we are providing:

• community releases of MySQL 5.0 for RHEL 3/4/5
• enterprise releases of MySQL 5.0 for RHEL 3/4/5
• extra tools for MySQL systems for most Linuxes

To install these repositories, grab the .repo file and place it in /etc/yum.repos.d/. You should then be able to install packages using e.g. yum install maatkit. Here are the .repo files:

• provenscaling-mysql-community-5.0.repo
• provenscaling-mysql-enterprise-5.0.repo
• provenscaling-extras.repo

We hope you like them and find them useful! Let me know if there are any additional packages you think we should add.

My slides have just been uploaded for the talk I just gave at the MySQL Conference and Expo 2008 titled “Scaling out MySQL: Hardware today and tomorrow“. You can download them now as PPT and PDF.

Thanks for coming to my talk!

Yesterday, Oracle‘s Ken Jacobs and Heikki Tuuri, creator of InnoDB, have announced the immediate release of InnoDB Plugin 1.0 for MySQL 5.1. I’ve already downloaded it and played around with it a bit. I haven’t had time to do any performance benchmarks or similar just yet. Those will come in due time.

This release is the beginning of two exciting things: InnoDB is now officially decoupled from MySQL release-wise, and lots of new features have been added to this new release. I will come to what the decoupling means in a moment, but first, the major new features in this release of InnoDB (from my perspective, and with my commentary):

• Fast in-place index management — The ability to add and drop indexes without rebuilding the entire table in place. This isn’t a complete implementation of the long-awaited “online ALTER TABLE“, as that is mostly a MySQL problem rather than an InnoDB problem.
• Compression of data and indexes — This should allow data size on disk to be reduced substantially.
• Storage of entire BLOB, TEXT, and VARCHAR data off-page — This can allow more efficient PRIMARY KEY indexes (where the data is stored in InnoDB because of the index clustering) on tables with BLOB, TEXT, or large VARCHAR columns
• New information_schema tables — InnoDB is now providing a lot more visibility into what is going on internally. I’m hopeful to extend this even further.
• InnoDB-specific “strict mode” — Don’t allow InnoDB to fudge things internally, forcing it to error in circumstances it can’t handle, rather than just give a warning or silently continue.

All of the above features look excellent, but one of the more interesting aspects of this announcement is the fact that MySQL and InnoDB are now decoupled for release. That is, Oracle can make a release of InnoDB without having to wait for MySQL to make their own release. While this will make it slightly more difficult to describe what version of MySQL/InnoDB you’re using (especially without a way to find that out from MySQL), it has the potential to make the release process much quicker and more efficient.

I’m quite hopeful, and from what I have seen in the recent past with the collaboration between Heikki, Ken, and Yasufumi (an outside contributor) on fixing InnoDB performance bugs, quite positive, that Oracle will be a lot more accepting of outside patches and code contributions to the InnoDB codebase than MySQL has been recently. I’ve got a lot of ideas for new features targeted at manageability that I’d like to get implemented. I’d love to hear Oracle’s comments on how they will accept patches to InnoDB now, and what they see the community interaction looking like in the future. Ken, any comment?

The only negative aspects I can see with this announcement are:

• Oracle spent a couple of years working on this in silence, away from the community. Most everyone was surprised by this release, as we haven’t seen anything from Oracle about InnoDB in a long time, and to some extent we were sitting in the corner hopeful that Oracle really doesn’t plan on killing the project. While I understand that perfectly well from a business standpoint, I am hopeful that working for long periods away from the community can be minimized in the future, so that we can all stay more involved.
• It is now a bit harder to get a new MySQL/InnoDB installation up and running, as the newest InnoDB is not part of the MySQL packages anymore. I’m sure this can be cleaned up with some smart RPM (etc.) packaging.

Overall, I am excited about this announcement, and quite happy that Oracle is making some serious contributions and commitments to maintaining InnoDB. Thanks for all your hard work, Ken and Heikki and the rest of the InnoDB team! Let me know if there’s anything I can do for you!

Just announced at the MySQL Partner meeting as part of the MySQL Conference and Expo in Santa Clara, CA:

MySQL will start offering some features (specifically ones related to online backups) only in MySQL Enterprise. This represents a substantive change to their development model — previously they have been developing features in both MySQL Community and MySQL Enterprise. However, with a shift to offering some features only in MySQL Enterprise, this means a shift to development of those features occurring (and thus code being tested) only in MySQL Enterprise.

As I’ve discussed before, the size of the user base for MySQL Enterprise is much smaller than for MySQL Community. That means these critical features will be tested by only a few of their customers. So, in effect, they will be giving their paying customers real, true, untested code. How is this supposed to work? In addition, this means that they are changing their internal development model, splitting the relationship between the two trees, and overall going even further down the path of getting the RHEL/Fedora model backwards.

What do you think about this? Leave a comment, I’m really curious as to everyone’s feeling on this.

UPDATE: Marten Mickos has just acknowledge that I understood the slide quite correctly, and they will indeed develop new features in MySQL Enterprise (in 6.0), without making them available in MySQL Community. Hmm!

We’re hosting the Silicon Valley MySQL Meetup tonight at the MySQL Conference and Expo venue in Ballroom D in Santa Clara, CA. We start at 7pm and run about 2 hours. Come on down!

If you follow the MySQL world at all, or you just have your eyes open, you have probably noticed that an agreement has been reached for Sun to acquire MySQL AB for about one billion dollars. Quite a few people have asked for my thoughts on the matter, so I will provide them publicly here for all. Overall, I see this as a mostly good thing.

I think that Sun has a very good chance of leading MySQL better than MySQL. At the same time, it’s always disconcerting to see a project managed within a very large company. Having been through the large company picture once already, I know how wrongly things can go when too many people (especially management types) are involved in a project.

At the same time, though, I’ve always liked Sun, and have high hopes for Sun’s management of the MySQL project and the people. There are a number of things I would like to see Sun do with MySQL following the acquisition:

Fix the MySQL Enterprise development model

The MySQL Enterprise (and by proxy MySQL Community) development model has been broken for a long time now. Too long. Take a cue from Sun itself and from RedHat and fix it right. I have a lot of ideas as to how the development model should work, and although our efforts have been time-constrained, we’ve made some effort to actually implement those ideas in DorsalSource.

Fix the product

There are a lot of areas where MySQL has been lacking for a long time, and the power users have been either crying in their beer (most users), or doing the work themselves (us, Google). I have shared some of these ideas with various people over the years, but here are some of the areas/ideas we have:

• Replication works fairly well (usually), but its model is completely broken and deficient to go forward with it with all the new features of MySQL. It lacks any real solution for multiple masters, synchronous or semi-synchronous replication, safety (checksums, binlog index and master info sync problems), and conflict resolution or automatic detection (transaction ids).
• Fix the internal memory allocation model so that it’s possible to constrain the memory usage of MySQL. The current situations sucks.
• Remove some of the outdated cruft littered all over MySQL: MERGE tables
• Clean up the logging (general, slow, [future] custom logs) code to be completely configurable and sane.
• Fix auto_increment. We have suffered with it too long. The storage engine isn’t the place to generate sequences, and InnoDB and the replication model suffer greatly for it.
• Fix the optimizer so that it makes more sane choices and can be more easily extended.
• This is partly a product problem and partly a people problem, but stop creating 100 different experimental storage engines, and pushing them as truth. Yes, archive, federated, blackhole, I’m looking at you.

Get some muzzles on the sales and marketing team

As Proven Scaling well knows, MySQL basically sells itself. It would be great if we didn’t have to hear from any more customers that MySQL’s sales team has or is trying to screw them over. No consulting without support? Stupid rule. Insane point of view on licensing? Get rid of it. Fluff? Don’t do it.

Bonus?

Sun should buy Innobase Oy back from the clutches of Oracle, pull InnoDB into MySQL proper, and relicense the full set of code under LGPL, BSD, or another similar license. Personally, I think the GPL is fine for MySQL, but the MySQL sales team has done so much damage to people’s ideas about the GPL—when they do and do not need licenses for MySQL—that it’s hard to continue under the GPL now.

Conclusion

I’m hopeful. I’m hesitant to shout out in glee. If anyone has any questions or comments regarding the above ideas, comments, or thoughts: let me know! I’d be happy to discuss any of them.