Thursday, June 28, 2007

PBXT: Top 5 wishes of a Storage Engine

In response to Ronald's challenge in Top 5 wishes for MySQL, here is my top 5 wish list. However, it make sense for me to put a slightly different spin on the top 5 series, and write from a storage engine developers perspective.

1. A generic engine test suite

A set of mysql-test-run test scripts and results that are intended to be run by all engines. The tests will verify basic functionality and compatibility, and form the basis for an engine certification process.

2. Internal APIs

PBXT already has to call into MySQL to open .frm files, and transform path and file names. The BLOB Streaming engine will need to access user privilege information. Other engines use the cross-platform functionality provided by mysys. What we need is a number of official, well-defined APIs to various MySQL internal functionality.

3. Customizable table and column attributes

Specialized engines require specialized information. Right now, this information is being packed into table and column comments (hack, hack, ...).

4. Push-down restrict and join conditions

This is a big one for engines in general. Many engines are being created that can do certain searches better than the MySQL query processor. However, for the optimizer to know whether to push down a condition or not will probably require a better performance metric.

5. Custom data types

SQL-92 has the concept of a domain, which is basically a named data type. This could be used as the basis for custom data types provided by a storage engine, made available in the form of a new domain.

And without numbering them, let me slip in a few more wishes. How about MySQL community project development hosted on MySQLForge, complete with integration into the MySQL bug tracking system?! And I have heard that this may also be possible: PBXT and other GPL community engines on the MySQL Community distribution :)


Unknown said...

Hi Paul!

Great ideas. Re: #2, that goes hand in hand with my #1 wish as well (a modular core kernel with defined APIs).

Also, community project hosting might be nearer than you think... the Forge is going to be in a constant state of development this next quarter as I get more time to focus on it...



rpbouman said...

Hi Paul,

yeah great ideas!

I'd like to see proper domains too. About the engine condition pushdown, I was thinking a while ago that there should be hints to have more fine-grained pushdown control. I mean, session scope is too wide, it should be there for each WHERE condition, possible even parital condition expressions (although I can imagine a problem there for the optimizer)

Anonymous said...

I may be out of my league here, so I apologize for my greenness. I for one, would love custom data types in MySQL. Probably not so much for the purposes of specific engine extensions, but rather like to explain things by example:

Why should we need to use INET_ATON() and INET_NTOA() when we could just have an IP address datatype and be done with it? It seems to me that the INET functions are prone to programmer error by simply forgetting that you need to run one of those functions.

To be fair, though, I am not a developer, or at least not on the level that would be required to even think about to implement such a thing, so I suppose one ought to take my comments with a grain of salt.


Unknown said...

I'm seeking people that can actually help make constructive changes to the MySQL Storage Engine like you suggest. Anybody want to work for Sun and show them how it's done?

- rob beck
rbeck at prism group dot com