jcole’s weblog

For a few years now Proven Scaling has been hosting the MySQL mirror/archive at mirror.provenscaling.com, which provides mainly older MySQL releases (which have been removed from MySQL’s mirror long ago) and MySQL Enterprise releases for the last few years.

Since Proven Scaling has been winding down (and we’ve started paying the bills personally), it doesn’t really make sense for us to maintain as large of a colocation footprint as we have. We’re looking to shift things around, and the mirror is something that’s fairly painful for us to host with a small footprint. We’re hoping another MySQL community member with an already-large datacenter/bandwidth footprint will want to pick it up, where it won’t affect their bottom line so much.

I think there’s still some value in having these files be available, so I’m reluctant to just shut it down (but if no one is interested, that may happen).

Some basic stats about the mirror (for July, 2010):

• Data size: ~305G
• Hits: 232,242
• Transfer/mo: ~703G (this is pretty variable, and was ~1.23TB in May)
• Transfer/day: ~22G avg, ~71G max

I’d be happy to provide file listings and Webalizer stats to interested parties. I will also note that a lot of the data transfer seems to be junk traffic from Asia, although I’m not sure what can be done to minimize this. Ideally we would arrange to rsync the full data set to a server somewhere (or provide ssh access for you to do the same), and you’d maintain the mirror as necessary going forward. We’re happy also to point mirror.provenscaling.com at the new home.

If you’re interested, email me at jeremy@jcole.us.

I am speaking in a few days at OSCON 2010 in Portland, Oregon. My talk, “MySQL Bottleneck Hunters – How Schooner Increased MySQL Performance by 8x” is about how Schooner optimized MySQL to run on modern hardware with flash memory. Come on by if you’re at OSCON!

Here’s the abstract:

MySQL Bottleneck Hunters – How Schooner Increased MySQL Performance by 8x — Thursday, July 22, 2010 at 11:30am

MySQL users have an insatiable need for speed, capacity, and availability, all at a reasonable cost. This session will provide technical overview of the approach that Schooner engineering took to optimize MySQL Enterprise and InnoDB with flash memory, multi-core processors, and DRAM to achieve an 8x improvement in performance relative to existing systems.

This session will provide a crash course in the performance tuning tools and techniques that Schooner used to radically improve MySQL performance on commodity hardware. It will also offer an overview of the results that were achieved, and compare these results to other commonly deployed MySQL server architectures, including stock MySQL on SSDs and Fusion-io, using the industry-standard DBT2 benchmark.

I am by now a few days late, but I meant to point out once the milestone had been reached:

MySQL Bug #11660, “Expose either SQLState, mysql_error() or other diagnostics in stored procedures” was opened on June 30, 2005. It has now been open for more than five years with no fix in the server, no fix in sight, and no real workaround. This makes it very difficult to recommend the use of MySQL stored procedures for most serious applications, as it is somewhere between very cumbersome and impossible to handle errors properly. How can something so important go so long without a fix?

It seems like everyone in the MySQL community has been chiming in on the so-called “open core” versus Open Source debate. It seems like at least one potential opinion has been unsaid so far, and it’s the one I share: It doesn’t matter if you’re closed source, “open core”, or Open Source, as long as you are completely honest with your customers and/or your users about which of those you are, and that you communicate any changes, particularly in a more closed direction.

I think this is the problem that MySQL (regardless of owners) have suffered in the past — they faced pressure to make money (which is fine), and they decided to used closed source approaches to do so (which is fine), and they picked some features, which they’d promised their users for a long time, to do it with (which is fine, although arguably not that nice), and they failed to communicate it well to either users or customers (which is tragic). The end result is, as could be expected, a bad taste in both users’ and customers’ (and in many cases, even employees’) mouths. In addition to this failure to communicate, they’ve also fallen over pretty terribly when others attempt to do the communication for them, as demonstrated when they made the decision (as far as I know, overturned) two years ago to offer some features only on the Enterprise side of the Community-Enterprise split. If you’re being completely honest and open with your communications to your customers and users, a post like mine should spark a “so what, we already heard that” instead of a huge storm of debate.

From my perspective, MySQL has never been “true” Open Source, as the documentation (which I helped to write a large part of as an early employee of MySQL) is not free, and never has been.

All of this is just fine, but don’t be surprised when the public is critical of you, and isn’t clamoring for the mountain tops to sing your “open source” praises.

If you participate at all in the MySQL Meetup circuit, by now it’s likely you’ve heard that the agreement currently in place between Meetup.com and MySQL/Sun is expiring quite soon. Because of that, MySQL Meetups which used to be free for organizers are reverting to paid status very soon (costing the organizer at least $12 per month). MySQL Meetup organizers (like myself) have received an email from Meetup.com giving them 7 days warning. MySQL/Sun have suggested that all MySQL Meetups move to Facebook.

I don’t know all of the details of MySQL and Meetup.com’s prior arrangement for this sponsorship, but from what I gather it did not involve MySQL paying $12 per month for each Meetup—I suspect it was a free/barter agreement. I specifically don’t know Meetup.com’s side of the story; but it doesn’t matter. I feel that Meetup.com has always provided a useful, functional, and fairly priced service. It was nice that it used to be free for MySQL Meetups, but nonetheless it’s a service I’m willing to pay for (and I did, for a few months, before I found out that they could be free).

In fact, I’m willing to do more. I think, given the pittance of a cost involved, it is entirely unnecessarily disruptive to the MySQL community to expect all MySQL Meetups to move to a new service… and I’m going to continue to use Meetup.com for the Silicon Valley MySQL Meetup. Furthermore, I am willing to personally sponsor about 10 other MySQL Meetups around the world.

So, to that end: If you are a MySQL Meetup organizer, and you would prefer to continue using Meetup.com to organize your meetings, please contact me. We can work out the details, and I will pay for a handful of you. Please send me at least the URL to your Meetup, so I can check it out. I’m looking to sponsor interesting and active Meetups, not necessarily idle Meetups that have never actually had a meeting. However, you can be anywhere in the world, and you don’t have to be a particularly large group.

I have been discussing this offer (before publication) with Peter Zaitsev of Percona, and he is willing to sponsor a few MySQL Meetups as well.

Thanks, and have fun meeting up!

I have been silent on the topic of this year’s MySQL conference, and really, silent in nearly all ways anyway. Today, as the MySQL Users Conference Conference and Expo 2009 starts up, some people will be wondering where I am, so I ought to at least answer that: I am skipping the conference this year.

As my wife can tell you, this is not a decision I’ve taken lightly, and I’ve gone back and forth on it for months and then down to the final days leading up to now. I’ve decided after much internal and external debate just to skip it all this year, including the side conferences and other stuff. Why? The reasons are basically:

• It’s my opinion that the “state of the art” with MySQL has not changed, so I am not really missing much technical content to further myself. There’s a lot of interesting stuff going on with Percona and Google’s work, but I largely follow that through blogs and mailing lists and personal contacts already.
• I strongly disagree with the speaker selection process used in this and the previous couple of years’ conferences. I feel that they actively discourage any speakers who refuse to stay on script—those willing to tell users the reality of the situation. I can’t and/or won’t do that, so I didn’t submit anything this year—my initial effort, months ago, to disengage from the conference. I watched from the sidelines this year and listened to everyone else’s complaints about not being selected. I don’t care to hear about the process, the details, the counter-arguments, etc., but really, any selection process which ends up leaving out Percona, for any reason, is just broken. I know they got some consolation slots in the end, but that doesn’t change my opinion.
• The previous years’ conferences, while they have been fun in various ways, have been declining in actual technical content and increasing in politics, marketing hype, PR stuff, and whatnot. This does not have value to me (see above).
• I am tired of explaining MySQL Enterprise vs. MySQL Community, InnoDB/Oracle vs. MySQL Inc., where the patches have gone, why the foreign key implementation in MySQL sucks, why the triggers implementation sucks, the problems with replication, etc., ad infinitum. Nothing has changed, it’s all basically as broken as it was last year, the year before that, etc.

OK, many people know the above about me already. So why not go to the Percona Performance Conference or hang out and chat with people, or…

• The most important reason: I am making a concerted effort to stay out of the politics of the whole conference this year. While I feel that I add nominally value to the conference by analyzing and reporting on the happenings, and asking uncomfortable questions, I just can’t do it this year. I need a break.

I debated potentially taking this entire week as vacation and really disappearing for the entire week, disconnected, but I really value the friendship of all of my friends that come out to California from the far reaches of the Earth for the conference. I will be in the Bay Area (as usual every week), and I would very much enjoy having a politics-free drama-free dinner if you’re up for it. Contact me!

Administrative Note: I have disabled comments on this entry, as I don’t want the debate and politics to shift over here. I apologize in advance for my anti-blog-like behavior in that regard. I welcome any and all notes and comments at jeremy@jcole.us.

As of today, Proven Scaling has launched an official blog, which we’ll all be writing entries on. Check it out and subscribe!

All of this talk of Drizzle (a fork of the MySQL server by Brian and Monty) has reminded me of a topic I have wanted to discuss for quite some time…

One of the things that sets MySQL apart (in, IMHO, a very bad way) from other Open Source database projects/products such as PostgreSQL (license) and Firebird (license) is that the MySQL documentation is NOT Open Source. The MySQL documentation is and always has been copyright MySQL AB, and “… use of this documentation, in whole or in part, in another publication, requires the prior written consent from an authorized representative of MySQL AB”. This presents a major impediment to forking the server: who wants to re-write many hundreds of pages of documentation on things that haven’t even changed? Even if you’re OK with all of your new code being GPL (and anyone forking ought to be), and never being able to dual license or re-license your MySQL fork (oh well), you will have to start with no documentation, or publish an errata against the official MySQL documentation (which will only go so far).

Does this mean that MySQL is not really Open Source? I would say not exactly, although I could probably be convinced either way. Others may say yes. But it does go a long way to making the point that some things may not be quite as “open” as they initially appear. What do you think? Can a piece of software really be Open Source while its primary/only documentation is not? Did you know that the MySQL documentation was not Open Source?

Kaj just announced that as of 5.1.25, MySQL Cluster will no longer be included in the normal MySQL packages. Instead, a new branch is being created for MySQL Cluster releases, where the first release is informally called “6.2.15″ but the releases are really named “mysql-5.1.23-ndb-6.2.15″. This new branch is based on the MySQL Cluster Carrier Grade Edition of MySQL.

Overall, this seems like a good idea — the needs for releases, schedules, pressures, etc., are at odds between MySQL Cluster and MySQL’s core. I am, however, baffled by the decision of how to release the new product: coupled with MySQL as a single monolithic package with compiled-in storage engine. After all, 5.1 has long been touted to have this amazing new pluggable storage engine architecture. Why not use it?

With Oracle/InnoDB recently announcing that they are decoupling from MySQL and releasing their storage engine as a plugin, this would make so much sense. The only thing I can think of is that MySQL and/or the MySQL Cluster team must think that the pluggable storage engine concept is not workable in its current state or easy enough to use… which I would agree¹ with absolutely. However, having a team within MySQL really pushing a product and using the pluggable interface to make their releases would help dramatically to make the interface really usable for the rest of the world.

Why not do it? Let’s hear it…

¹ My short-list of gripes, by no way complete: The error message interface sucks, no way to compile a plugin outside of the MySQL source (or even symlink it into the source tree), plugins are tied to an exact version of MySQL server, and no reasonable way to manage plugins in an OS context (RPMs, .debs, etc).

Yum is an extremely popular system to download, install, and update RPM-based packages from multiple repositories. Proven Scaling has launched a set of repositories to augment the existing central distributions’ repositories with packages our customers need for deploying MySQL-based systems. We’ve been working on it for a while, and have had many people making use of it. We are providing:

• RPMs of community and enterprise releases of MySQL for RHEL/CentOS, as built by MySQL and distributed on MySQL.com
• RPMs of community tools such as maatkit and innotop and their dependencies.
• Proven Scaling-created tools such as mysql_snapshot (an LVM snapshot-based backup utility).
• Difficult to find RPMs of Perl libraries (dependencies for other scripts, such as innotop).

Here are the yum repositories we are providing:

• community releases of MySQL 5.0 for RHEL 3/4/5
• enterprise releases of MySQL 5.0 for RHEL 3/4/5
• extra tools for MySQL systems for most Linuxes

To install these repositories, grab the .repo file and place it in /etc/yum.repos.d/. You should then be able to install packages using e.g. yum install maatkit. Here are the .repo files:

• provenscaling-mysql-community-5.0.repo
• provenscaling-mysql-enterprise-5.0.repo
• provenscaling-extras.repo

We hope you like them and find them useful! Let me know if there are any additional packages you think we should add.