"One size fits all" Relational Database is dead. What say u?

OK, this is my first blog.
I will mostly blog on my professional life and it has a lot to do with high performance distributed computing. Especially, main-memory based distributed data management systems.

Let me begin by taking a look at the traditional relational database.

For the last two decades or so, major Database vendors have taken a "kitchen sink" approach trying to solve any problem associated with data management by bundling this as part of the database engine. Don't take my word on this. Here is how Jim gray's put this "We live in a time of extreme change, much of it precipitated by an avalanche of information that otherwise threatens to swallow us whole. Under the mounting onslaught, our traditional relational database constructs—always cumbersome at best—are now clearly at risk of collapsing altogether" Checkout this article . Adam Bosworth also has some interesting comments in his blog "Where have all the good databases gone"

Alright! What specifically I am talking about?

Consider this for starters:

Take a portal - today you want to build scalable systems using a clustered architecture where you can keep adding processing nodes in the middle tier so you can linearly scale as the number of users keeps going up.

Now, I don't have to tell you how important availability is for folks like these. Always up, always predictable in terms of performance and responsiveness. Enormous attention has been paid to make the middle tier highly available. But, alas, when it comes to the backend data sources, it is left upto the DB vendor.

The traditional database is built to do one thing very well - do a darn a good job in managing data securely, correctly and persistently on DISK. It is centralized by design. Everything is ACID. Ensuring high availability when you are constrained by the strong consistency rules (everything to disk) is very tough to manage. Replicate a database so you can failover to the replica during failure conditions, you are all of sudden left with random inconsistencies to deal with. Try to replicate synchronously, you pay a huge price in terms of performance. You want to provide dramatic scalability, through many replicated databases, you better be ready to live with a very compromised data consistency model.

Ahh! so, something like Oracle RAC is the answer, one might argue? Yes, for a number of use cases. But, here is what one has to consider:
1) You are still dealing with a disk centric architecture where all disks are shared and each DB process instance has equal access to all disks used to manage the tablespaces.
Here are a few important points worth noting:

• The unit of data movement in the shared buffer cache across the cluster happens to be a logical data block, which is typically a multiple of 8KB. The design is primarily optimized to make disk IO very efficient. AFAIK, even when requesting or updating a single row the complete block has to be transferred. Compare this to an alternative offering (a distributed main memory object management system) where the typical unit of data transfer is an object entry. In a update heavy and latency sensitive environment, the unit could actually be just the "delta" - the exact change.

• In RAC, If accessing a database block of any class does not locate a buffered copy in the local cache, a global cache operation is initiated. Before reading a block from disk, an attempt is made to find the block in the buffer cache of another instance. If the block is in another instance, a version of the block may be shipped. Again, there is no notion of a node being responsible for a block; there could be many copies of the block depending on how the query engine parallelized the data processing. This will be a problem if the update rate is high - imagine distributed locks on coarse grained data blocks in a cluster of 100 nodes; for every single update?

• At the end of the day, RAC is designed for everything to be durable to disk and requires careful thought and planning around a high speed private interconnect for buffer cache management and a very efficient cluster file system

Yes, there have been tremendous improvements to the relational database, but, it may not the answer for all our data management needs.

Consider this - How many enterprise class apps are being built that just depend on one database. Amazon's CTO, werner wogel talks about how 100 different backend services are engaged to just construct a single page that you and me see. Talk to a investment bank, events streaming in at mind boggling speeds have to be acted on not by one process, but, by potentially 100's of processes, instantenously. Everything needs to talk to one another in real-time. Consistent, correct data has to be shared across many processes built using heterogenous languages, and connected to heterogenous data sources all in real-time. Ahh! what do call this SOA, ESB, etc.

You can come up with a massive loosely coupled architecture for the connecting everything with everything, but, you have to pay extra attention to how you will manage data efficiently in this massively distributed environment.

Isn't it time for a middle-tier high performance data management layer that can capitalize on the abundant main-memory available on cheap commodity hardware today ? - a piece of middleware technology that can draw data from a multitude of data sources, move data between different applications with tight control on correctness and availability allowing all real-time operations to occur with a high and predictable Quality of Service.

Is this a distributed cache? A messaging solution?

OK, let me sell you what we do ....