Reflections on return to MySQL Community and Ecosystem

After a four year hiatus, my participation in last week’s Percona Live MySQL Users conference marked my official return to the MySQL Community and Ecosystem. As with earlier renditions this year’s “UC” was very well attended with a healthy mix of familiar faces and new blood, all coming together to discuss, present and explore the boundaries of the most popular and widely used open source database on the planet.  There were many good, informative keynote and technical sessions, BoFs and the exhibit hall was packed most of the operating hours with those interested in what the MySQL ecosystem is up to.  I also found it very refreshing that Oracle was among the most active in presenting useful technical content around their current and future MySQL open source product releases. All in all, this year’s UC was very well done and as a Percona employee my hope is that we continue to host the annual event for many years to come.

With all that said, I feel compelled to share some observations from the week and offer some ideas on how we (Percona + the MySQL Community) might improve on similar events going forward.  So, my observations, in no particular order:

The MySQL community is alive and well.   By a wide margin MYSQL remains the #1 open source database in use today due largely in part to how it is embraced, developed and evangelized by the MySQL Community and those who lead it.  For an example, simply follow how the engineering teams from Facebook, Twitter, Google and LinkedIn have chosen to build WebScaleSQL on MySQL 5.6 Community edition.  That decision speaks volumes to the vitality and relevance of not only the MySQL Community, but also to the code branch chosen for this work. A huge win for all involved.

The MySQL Ecosystem is mature and continues to innovate.  I spent the last year working in the NoSQL database space and it felt very much like my early days with MySQL; many problems to solve, but few production-ready solutions available for immediate help.  In contrast, walking through the UC expo hall this week I was encouraged to see many vendors that cover the gamut when it comes to solving problems around MySQL; high availability, backup/recovery, advanced search, replication and clustering deployment and management, storage, performance monitoring and tuning, consulting, training, tools, etc.  The cool thing is that most of the solutions also provide innovative support for MySQL as it is used in the Cloud or in conjunction with other emerging technologies such as OpenStack, NoSQL, and Hadoop.

Speaking of OpenStack, it is evolving in the true spirit of community driven open source development and those with ambitions of providing or using DaaS based on MySQL should bone up accordingly.  It is quickly becoming the defacto standard for Cloud deployments built on MySQL.

MySQL users have many choices.  One of the advantages of the open source development model is it encourages healthy competition in the development of a specific product.  In this case, MySQL users have their choice of Oracle’s MySQL, Percona Server, MariaDB, TokuDB and relatedCluster/HA options from each.  The competition among these and other branches of the product is to the advantage of the overall MySQL Community.

And lastly, Oracle continues its investment in MySQL.  Oracle has remained true to its word by continuing to invest in the development of upstream MySQL.  5.6 is the best release of the product to date and 5.7 is looking very strong.  Oracle’s presence and participation at the UC again this year is validation that MySQL matters to them and they see Percona and those behind WebScaleSQL as partners in the ongoing development of the Community facing product. It was great to see Tomas and the Oracle/MySQL engineering team so actively engaged with the MySQL Community and overall ecosystem at this year’s event.

Like the MySQL Community, I think it is important for the UC dynamic to continue to evolve to draw in fresh ideas, innovation and a realistic depiction of how MySQL is used within hybrid applications and architectures.  So to close, here are some non-exhaustive thoughts along those lines:

Align sessions into role specific tracks.  This may be as simple as creating Developer, DBA, SysAdmin and Business centric tracks that align with attendee demographics and areas of interest.  CFPs could then be channeled into this template so topics are balanced and scheduled by area of interest. Keynotes could also be scheduled that speak to each of the tracks, setting the tone for each.

Encourage Sessions that present MySQL hybrid use cases.  Most NoSQL solutions are being used as a complement to vs. a replacement of MySQL.  Developers, DBAs, SysAdmins alike need to learn how to extend their current MySQL applications to leverage these new platforms without building new systems from scratch.  

Add a C-level track that explains MySQL, the MySQL ecosystem and the business value of mature, open source software.  This could be in the form of a 1-2 day subset of the UC that includes sessions that cover high-level technical topics that tie the lower-level track sessions with topics like TCO, support, services, etc. This is now a common component of events hosted by the new NoSQL providers and would bridge an understanding gap between MySQL techies and the decision-makers who depend on them.  

As always, interested in other opinions, so please let me know your thoughts.

Great to be back in the MySQL fold.  Thanks for holding my place!  

MongoDB 2.5.0 Development Release now available

In the spirit of releasing "early and often", 10gen Engineering has released MongoDB 2.5.0 for development testing and feedback.  You can download the bits and view the change log and release notes here.

And while 2.5.0 is not considered production-ready we very much appreciate early adoption and feedback on the quality on our work to this point.  So, also in the spirit "early and often", please let us know the good, bad and ugly by filing Jira tickets against 2.5.0.  Your helping with our engineering and development efforts is greatly and humbly appreciated!

So what's in 2.5.0 and why should you care?  While not exhaustive here's a quick summary of two enhancements of interest (details on these things are in the release notes and change log):

LDAP Support for Authentication
MongoDB 2.4 and higher provides support for proxy-based authentication of users.  2.5.0 allows administrators to configure a MongoDB cluster to authenticate users via Linux PAM or by proxying authentication requests to a specified LDAP service.  This specific feature is foundational to our enabling MongoDB to be tightly integrated into SSO environments, so the more eyes we can get looking at it early on the better.   Please see the release notes for the details on getting set up.  Please, please, please help us prove this out!  Did I say please?

SASL Library Change
To ensure SASL authentication works consistently across all Linux distros, specifically in conjunction with MongoDB Enterprise Kerberos authentication (a very good thing that many care about as reflected in JIRA), MongoDB 2.5.0 now uses Cyrus SASL instead of GNU SASL (libgsasl).  See the release notes for compatibility and SASL2 and Cyrus SASL library plugin dependencies across Linux platforms.

You might also find Eliot's blog on Mongo's new Matcher interesting as it lays the foundation for more complex, advanced query handling in the future.   

Happy testing! and as always, thanks for your support of MongoDB!

Thoughts on move to 10gen and MongoDB as NoSQL market leader

It has been a little over a month since I joined 10gen and I want to share some of my impressions now that some of the newness has worn off and I am settling into my role.  For context, in March I left Oracle and my position as Director of Product Management for the MySQL database to assume a similar role at 10gen working with MongoDB.  I still have many good friends and respected colleagues on the MySQL team and truly believe them and the MySQL product to be in good hands under Oracle.  That said, here are my thoughts on my move and on the database market as a whole (note the emphasis here is on use case vs proprietary and open source):

10gen reminds me very much of MySQL AB prior to the Sun acquisition. 
I joined MySQL in 2005, a full 3 years before the Sun acquisition. Back then, there was little distinction across the titles and roles within the MySQL team. Basically, everything that everybody did everyday made every bit of a difference. This meant that there was a cohesion across the entire team and that most tasks were met with a “can-do”, positive attitude by the owner or owners regardless of where they fell on the org chart. I see this same trait carried out in the 10geners I have been in contact with and hope it carries on even as the company continues to grow.

10gen places a huge importance on all employees having a working technical knowledge of its products. 
I joined 10gen with years of application and database development experience, mostly using relational models. While most concepts apply to a document-oriented model there are enough technical differences that leave me as a complete novice when it comes to specific MongoDB details and its practical use cases. I have spent my first month in a series of self-guided and instructor led technical training sessions and a practical, real-world bootcamp that have proven to be a welcome quickstart to my understanding and working with the technology. This will pay great dividends as I get more overwhelmed with my true PM duties.

MongoDB is winning the NoSQL database market. 
More importantly, in winning this market it is also winning many general use deployments and projects that have traditionally been implemented on other open source or proprietary rdbms solutions. How is this happening? It really boils down to a few simple, but important factors:

• MongoDB is winning the hearts and minds of the developer.  By providing flexible, direct access to schema and data definition via JSON, there is little/no developer learning curve when moving between application development and data definition and management. 

• MongoDB is a true “hero” maker.  Replication and cluster based sharding are designed as the default deployments and are comparatively simple to implement. Developers can add HA scalability to their upfront deployment plan without adding pain to a DBA or Sysadmin’s life, which is a huge advantage over other databases’ modus operandi. 

• MongoDB users leverage > 90% of its functionality.  Without the complexity and overhead of unneeded features. On the flipside most Oracle, SQL Server users/applications leverage < 20% of features while paying for them all. These things, along with the tremendous momentum around downloads, user events, big name community and customer success stories are good indicators that MongoDB is poised to not only win the NoSQL database market, but the overall database market in due time.  

• It took Oracle 30 years to build an empire; I believe 10gen + MongoDB can do better for both community users and paying customers in a much shorter timeframe if we remain focused on and true to the points noted above. 

One final thought.  I have gained a true appreciation and respect for Eliot, Dwight, Max and the other executive level leaders of the 10gen business. They are clear in their ambitions for MongoDB and any supporting products that come out of 10gen in the days, months and years to come. They are on public record as saying that 10gen’s goal is to build a sustainable investment and business around the best and most widely used general use database in world. That long term mentality, with many small sprints factored in along the way, is what excites me most about my move to the 10gen team.

Cheers, from 10gen employee/partner #237! Very excited to be here.

5.1 + new InnoDB Plugin

If you're wondering what MySQL 5.1 with the new InnoDB Plugin is all about you'll want to tune into the webinar I am hosting next Tuesday. I'll cover the why, what and how behind the immediate performance and scale improvements that can be had, especially on modern (> 4 CPU) servers, by enabling the new plugin in MySQL 5.1. Learn more and get registered, it should be good time!

You can also take it to a deeper level by joining Brian Miezejewski from our PS team on 3/9 for a practical guide to using and tuning the new plugin features to improve performance and scale. Hope you can join us!

One reason a no indexes approach is nice

Like a lot of you, I’ve been following with interest Percona’s testing of the open source column databases. One thing I think is pretty cool about some column databases that work with MySQL is that they don’t require you to create indexes. The reason is, in general, the column is the index. Not having to create indexes is nice because lots of indexes can really bog down a database if you’ve got a lot of load or DML activities because the indexes have to be maintained for all data input and alterations.

In Percona’s test, they showed the load time for all the different databases, but I noticed that the times didn’t include the index creation for LucidDB or MonetDB. I decided to follow Vadim’s link on the LucidDB index creation and totaled up the time it took to create the indexes. For the index and statistics times, it was 384,314 seconds and when you add that to the 140,736 seconds for the table load, you get 6 days just to create the database. That’s quite a difference from the 6 hours for InfiniDB and 14 hours for InfoBright, both of which don’t need or use indexes.

I’m sure indexes supply a benefit for some column DB’s in various use cases, but if the database was real dynamic and required a lot of new objects with indexes be added, continuous heavy loads, or DML, it would seem that indexes could really put a ding in things. In that case, it would seem column DB’s thatdon’t require indexes could have an edge there.

Less time finding, more time fixing! Enterprise Monitor 2.1, Updated Query Analyzer Now GA!

I just wanted to tip my hat to the MySQL Enterprise Tools Engineering team for another great release of the Enterprise Monitor. Not to name names, but I want to give a special thanks to a team that always over delivers on a collective commitment to producing quality software. So, a mega thanks to:

Andy Bang, Sloan Childers, Darren Oldag, Eric Herman, Jan Kneschke, Kay Roepke, Mark Matthews, Bill Weber, Diego Medina, Marcos Palacios, Carsten "Pino" Segieth, Josh Sled, Keith Russell, Mark Leith, Heidi Bergh-Hoff, and Gary Whizin (and also welcome Michael Schuster!)

Yet another great job guys!

The new version, 2.1, was posted as GA early on Tuesday and it is quite possibly the best release of the Enterprise Monitor to date.

For those not familiar with the Enterprise Monitor, it is included in a MySQL Enterprise subscription and is a distributed web-based app that users deploy in their environment to monitor and tune the security, performance and availability of their dev, QA and production MySQL servers. It is comprised of :

• An agent, written in C, that is installed on each monitored data source, which collects MySQL and OS metrics, SQL code and exec results
• A central server (aka "service manager"), written in Java, that collects, monitors and alerts on the data collected by each agent. The service manager uses MySQL Best Practice Advisors to measure the collected data against user-defined thresholds and to proactively notify DBAs of problems or tuning opportunities. Alerts are sent to a central console or via SMTP or SNMP notifications.
• A repository that holds the data collected by each agent. The service manager monitors the data stored in repository vs maintaining a persistent connection to each MySQL server.

The Enterprise Monitor also provides MySQL specific monitoring features. There is a Replication Monitor that proactively monitors replication topologies for synch and performance issues and a new Query Analyzer that made its debut in release 2.0 in late 2008. The Query Analyzer is designed to save DBAs/Devs time in finding the most expensive queries (by total exec time, exec count, amount of data returned, etc) running across all dev, QA and prod MySQL servers without any dependence on the MySQL logs or things like SHOW PROCESSLIST. You can learn more about the Query Analyzer here.

Based on customer interviews and years spent in the field, we understand the pain associated with finding bottlenecks rooted in poorly written or inefficient SQL code (this is consistently the #1 problem we hear when talking with MySQL DBAs and devs.) This new release helps a DBA/Dev spend less time "finding" and more time "fixing" poorly performing queries. The key new features in the Query Analyzer include:

• Clickable MySQL and OS graphs that visual correlation system and query activity. Mouse-over a spike in any graph and drill into the queries that were running at the same time. Big time saver.
• Drill down capability for query specific executions – drill into any query and see execution specific graphs for exec time, count, data returned. Helps you see the "normal" exec pattern for a query and identify outliers that may occur during specific windows or time or for specific variable combinations.
• Counts for SQL errors/warnings - help you quickly identify queries that may have never finished or that finished in error. These routinely go undetected by the MySQL logs, etc.
• UI support for EXPLAIN generation threshold - gone are the days of hacking the quan.lua script to set this value!

We have also added new Advisor Rules and Graphs around connections, stale table statistics, tables without indexes, PKS and locked and long running processes.

The new release also includes an updated "What's New" page that allows you to optionally subscribe to live feeds for your open support issues (nice, especially for the issues with a status of "waiting on customer") and for MySQL product updates and alerts. Nice time saver, especially for getting updates on new releases of the Enterprise Server and for Monitor Advisors and graphs.

So how do you get it? Glad you asked...if you are an Enterprise subscriber you can grab the new release and all of the updated docs from the Enterprise Customer download page. If you are interested in learning more or want to try the new Monitor and Query Analyzer for yourself, you can register for a 30 day trial subscription which includes a fully functioning version of the Monitor and Query Analyzer. You also learn everything about MySQL Enterprise by visiting the MySQL web site.

Look forward to hearing about your experience with the new release!

Update on MySQL Enterprise Monitor and "Quan"

Just a quick update that the new MySQL Enterprise Monitor has reached RC readiness and is set for official launch in the next few weeks. This is version 2.1 and features:

• Enhanced Query Analyzer with correlation graphs so you can highlight and drill into spikes in key system resources to see the queries that were running at the same time
• Query specific execution graphs so you can track the "normal" behavior of your queries over time
• GUI support for Query Analyzer EXPLAIN generation
  
• Live feeds for MySQL product updates and alerts
• Live feeds for your open support issues
• New Advisor Rules and Graphs
• and some other things...

Here's a screenshot of the Query Analyzer and correlation graphs...


Enterprise subscribers can grab the RC here. If you aren't a subscriber but want to try it for yourself, you can get it here.

Look for the official announcement on GA availability early next week. Another great job by the MySQL Enterprise Tools Engineering team!