New Era for OLTP databases

Coming out of hibernation to blog. Made exciting with our new launch of the SQL data fabric. No wonder, when you start talking about a "first class" SQL interface to our distributed data grid/fabric platform it is compared with the RDB and modern day clustering extensions to the relational database.

Here is a zooming presentation that attempts to position the product as a horizontal partitioned memory oriented database (unlike the typical parallel DBMS that is vertically partitioned for OLAP class applications). I think you will find the use of prezi fascinating. Would love to get some feedback on the presentation. All this is still very preliminary and we will be sharing a lot more details in the days ahead.

(Click MORE --> full screen to get a better view)



here is a related webcast from our principal architect, Dave Brown

Challenges in throughput scaling in a partitioned design

Here is an interesting article from Billy Newport posted on highscalability.com

The gist of this article is a point that hits home for me - you cannot assume higher scaling (especially throughput) just by adding more partitions for managing your data.

The article correctly points out that one slow partition can throttle the system throughput for scatter-gather type queries that are parallelized on each partition. But, this assumes that all nodes are equal (or remain equal) from a load standpoint. I think, this ability to detect and handle "hotspots" is something that deserves a lot more attention than it gets. I suspect, there is this general myth that increase in the number of partitions will always result in higher throughput.

For instance, we at GemStone have seen the following interesting cases (+ more):
(1) 80-20 rule: Often, it is the 20% of the data that is lot more popular at any given time. So, you have a situation where too many clients are converging onto a few partitions creating imbalance. A whole bunch of your partitions are just idle and heavily underutilized. You can mitigate this to some extent by creating more replicas for the "hot" data, but, it can be difficult to predict when you need the extra copy and for how long.

(2) GC pauses: Consider a case where a large number of clients are updating partitioned data deployed in a JVM based data grid/fabric. Sooner or later, you are exposed to the dreaded "full gc" cycle causing the incoming client to be paused. Say, your normal update request takes fraction of a millisecond and let us also assume that the client requests are uniformly balanced across all the partitions. Essentially, it is more or less guaranteed that all clients will vector to each partition atleast once every second. Now, a full GC in any one partition causes every single client to be now pause. Have a GC that takes a whole minute, and you have all clients now waiting. Worse, the moment this paused partition gets out of it, the next partition gets into the same situation and so on.

So, what do you do?

We at GemStone Systems think some of the problems can be mitigated by providing two key capabilities:
1) dynamic rebalancing: This is the ability for the system as a whole to adjust itself by relocating buckets (subset of data within a partition) to less loaded nodes. And, doing this in such a fashion so that there are no pauses are introduced.

2) Enough instrumentation within the system to proactively avoid hotspots: GemFire captures a wide range of statistics on the query, update rate on any given partition, the average response times, CPU utilization, GC pause times, overall heap utilization with respect to a configurable threshold to reduce the probability of full GCs, etc. Applications get access to these statistics through a simple API and can then use a API to trigger rebalancing to offload some of the data from "hot" partitions to less loaded ones. Of course, there is no guarantee that the past throughput characterictics observed on any given partition to be representative of te future but the explicit control provides a way for the application developers and adminstrators to dictate what happens. Our sense is that they know best. We even provide a way for the application to simulate a rebalance to see the effect it will have rather than actually doing it.

Explore more on GemFire data fabric

High performance data sharing between C#, Java and C++

SOAP and High Performance: an oxymoron?

XML messaging using the SOAP protocol has become the lingua franca for interoperability. Though loose coupling and simplicity through a standard text based protocol has its appeal, it is no secret that SOAP isn't suitable for high performance messaging. The simplicity and extensibility of XML/SOAP has generated great deal of interest and resulted in pervasive support across many languages and scripting environments.

This paper evaluates the performance of SOAP for real-time trading and compares it to native binary protocol such as FIX (Financial Information eXchange).

Here is a simple example comparing two text based protocols, one that uses simple 'tag=value' pairs and the other a SOAP envelope. The price you pay for XML and SOAP is obvious.

A FIX message
8=FIX.4.3 9=00000098 35=X 49=ABC 56=XYZ 34=1
52=20021116-10:15:28 262=MYREQ 268=1 279=1
278=FOO.last270=13.42 271=1200 10=185

Equivalent message using FIXML (FIX Markup Language)

The benchmark conducted summaries that SOAP messages are 3.5-4.5 larger than FIX, latency is 2-3 times worse, and encoding/decoding costs are increased by up to nearly 9 times.

Web services based on REST offers an efficient alternative to SOAP based web services. It is a simple HTTP based protocol that allows access to resources through CRUD (Create, Read, Update and Delete) operations. However, this is mostly geared towards simple request-response type synchronous messaging between services. Products that offer reliable, asynchronous messaging semantics on top of REST could be something to watch out for.



Traditional messaging might be popular, but it has its limitations

High performance applications commonly use message oriented middleware products for asynchronous sharing of events and data across heterogeneous applications such as IBM MQ or Tibco Rendezvous.

However, messaging solutions, as the name implies, package, move and deliver messages, one at a time. The receiving applications often need sufficient context to act on the incoming message. Typically the publisher provides this contextual information through message headers increasing the overhead for each message sent. So, for instance, an incoming Order may have to contain sufficient information to identify the associated customer. With the related contextual data arriving in encoded form, such as the Customer ID, the application can only process the Order after it can fetch all the related data (the entire customer record with credit information, for instance). This might require a round trip to the database. Therefore, if all the related data required for processing an incoming message requires external data source access, the processing speed can only go as fast as the weakest link - the database in most circumstances.

Application environments with a high sustained message rate are much more exposed to this "weakest link" problem because enqueing in the messaging system doesn't really help. At some point, either messages have to be discarded or the publisher rate throttled.

To support heterogeneous applications most messaging solutions require the application to construct text based self-describing messages. It becomes the responsibility of the application to use an appropriate encoding format for data such that it can be decoded by the receiving application. Should you use XML for the data format you will experience performance problems similar to those outlined above.

In addition, a highly concurrent application environment with multiple publishers will have an increased probability for race conditions causing data integrity issues. Messaging systems have no inherent capabilities to deal with relationships between messages or ordering of messages across multiple messaging destinations (queues or topics). So, for instance, if Orders (parent) and corresponding LineItems (children) are delivered on two different queues, it is possible for the child data item to be delivered before the parent. Applications would normally be aware of these constraints and route related messages on the same messaging endpoint. It would be advantageous though, if the infrastructure used for transferring messages was aware of this relationship.

Most messaging solutions are really designed to support diverse platforms, multiple language bindings, multiple protocols, flexibility in terms of message reliability, and more. But when it comes to object oriented applications such as C++, Java or .NET applications that want to share data, it can be cumbersome constructing or interpreting message headers, encoding/decoding to/from text based payloads, configuring message delivery options, correlating messages, looking up related data from backend databases before taking action, etc.

Take this MQ example:

qMgr = new MQQueueManager();

/***********************************************************/
/* Open the queue, build the message, and put the message. */
/***********************************************************/
int openOptions = MQC.MQOO_OUTPUT;
MQQueue myQueue = qMgr.accessQueue(args[0], openOptions,
null, null, null);

MQMessage myMessage = new MQMessage();
myMessage.writeString(<>);
myMessage.format = MQC.MQFMT_STRING;

MQPutMessageOptions pmo = new MQPutMessageOptions();
myQueue.put(myMessage, pmo);

/**********************************************************/
/* Close the queue and disconnect from the queue manager. */
/**********************************************************/
myQueue.close();
qMgr.disconnect();

The application developer has to create a QueueManager, fetch queues, define the options to write into the queue, encode their application objects into some text based format, construct Message objects, configure message format, publish into the queue, close the queue and disconnect from the QueueManager. Quite cumbersome, right?


Introducing a Data Fabric

Shared objects/events across heterogeneous applications through main memory caching

The basic idea is as follows:

Instead of requiring application developers to think about messages being the mechanism to share information, why not let them use the same paradigm they use when communicating between various components within the application; simply share domain objects in common data structures like a Map and use native thread based notification services. Extend this concept such that these data structures are distributed and visible to disparate applications. Essentially, applications share data and events with each other through a shared object database that is distributed in nature. Applications perform CRUD operations on a database and receive notifications when changes to data they are interested in occur.

There are some important characteristics that differentiate this database compared to a regular disk-based relational database:

1). It is distributed in nature - applications publish data objects and the database copies/moves these to multiple nodes. Sounds like database replication, except the location of data, the number of copies made, how the data is transported are aspects that are different
2). It is primarily memory based and hence fast
3). It is active in nature and pays attention to complex expressions of interest from subscribing applications and when they change, instantaneously pushing the change to the application.

This kind of data management system is referred to as a data fabric or data grid. A Google search on “Enterprise Data Fabric” will provide an idea of various vendor offerings.

A data fabric or data grid combines the important features and semantics seen in database technology and messaging for high performance applications.

A true data fabric includes the following comprehensive capabilities:

Ø Designed to offer in-memory data access speeds: Data is managed in concurrent data structures (link to concurrent java stuff), primarily in memory with minimum contention issues.
Ø Flexibility in data storage: Data can be stored locally in process, replicated to multiple nodes, partitioned across multiple nodes, maintained both in-memory and in-disk or simply fetched lazily from a back-end data store. Where the data is located or how many copies are maintained becomes a configuration issue and is based on the requirements around performance, high availability and volume of data being managed, yet completely abstracted away from the developer. Data locality is virtualized in a data fabric.

Ø Simple development model: A data fabric offers a simple Map like interface to applications. Applications can simply fetch by key or put domain objects directly into the cache. There is no need to worry about headers, encoding, decoding to some intermediate format, etc.

Ø Scalable: By distributing the data the data fabric uses resources across multiple nodes. Deployments can simply add additional capacity on the fly and automatically get the data rebalanced and handle increasing load (concurrent activity or data volume).

Ø Transactional: The fabric can automatically participate in any ongoing transactions and ensure consistency of data across all the applications sharing the data. For instance, if two applications concurrently decide to create the same customer order and publish this to others, the conflict will be detected and handled appropriately. This is a big difference compared to messaging - with no notion of identity that can be associated with data inherent within the messaging system, it is non trivial to detect conflicts when two applications decide to make the same change at the same time.

Ø Reliable Publish-Subscribe semantics: Applications perform CRUD operations on a local cache and the corresponding event is routed to nodes that subscribe to the data. Data objects can either be synchronously or asynchronously pushed to subscribing applications. Events are pushed to subscribers that contain the new, changed or deleted object(s). The data fabric is intelligent enough to only propagate changes to data objects or its relationships, keeping the underlying network traffic to a minimum.

Ø Querying: Similar to a regular database, the data in-memory can be indexed and queried using SQL like syntax.

Ø Continuous querying: Applications register complex queries, which are queries with complex predicates, joins, etc and, unlike a regular query, are not just executed once. They become resident in the database and give the impression that the query is continuously running. As the data changes the continuous query engine calculates how the result set has changed, pushes the "delta" to the application and merges this with a cache result set on the application node.

Ø Heterogeneous language support: Application objects are automatically serialized in a neutral format within the fabric such that the same object can be de-serialized into an instance of a class in another application written using a different language (with a similar class structure).

For additional information, read more on the architecture of the GemFire Data Fabric

Understanding the data fabric through a use case

Let us look at a financial Trade order processing system that can route orders to an exchange offering the best liquidity.

Here is how an order flows through the system:

- Orders arrive as FIX messages from trading partners and clients to an Order processing application
- The incoming FIX message is validated (authentication and authorization), normalized (combined with other related information such as Trader information, etc) and written into the data fabric as an Order object
- If the incoming order rate is very high, the validation and normalization process itself can be load balanced across multiple nodes
- Orders are then routed to a Trading Strategy engine that uses different algorithms to determine the best time, quantity and exchange to route the order. Execution of the algorithms requires current market data from multiple exchanges (these are the prices for the different securities traded in the exchanges) and also uses various risk metrics in its calculations. The incoming market data is being pushed typically at a very high rate (20,000 or more ticks per second)
- Orders are batched and then routed to an Exchange router, the application responsible for reliably getting the trade order executed on a market exchange
- Finally, there is the entire process of post trade information exchange that we will ignore as it is not relevant for this discussion.


Let us assume that the order processing application and the exchange router application are written in Java and the computation intensive strategy engine is written in C++. These three applications are sharing data and events in real-time.

While it is beyond the scope of this article to explain all the aspects of the data fabric, we will focus on how the fabric enables objects and event sharing.


Shared Object model: With the data fabric, the domain model classes are designed such that they are more or less equivalent for each language in use. One might use an Object modeling tool and generate these classes for both Java and C++.

Serialization framework: Developers implement callbacks similar to Java Externalizable where object fields are written to a stream managed by the serialization framework provided by the data fabric. The framework serializes data into a language neutral wire format that can be consumed by any application that uses the serialization framework at the consuming end.

In this case, the order processing application knows that an order is a complex object comprised of several fields and sub fields. Using the serialization framework, the application writes out the relevant fields out to the stream and puts it out on the wire.

The framework handles translation of primitives across languages and across processor architectures, removing that burden from the end user application. It also preserves complex object relationships across the serialization boundary.
At the receiving end, the incoming bytes are identified as an order object (because the type Id for the payload would be the same across all languages).
Once the object type is identified, the rest of the payload is streamed into the order object on the receiving end (let's say that this is the strategy engine)

Another advantage of this mechanism is that the de-serialization of the payload from the wire (the other half of the serialization framework) results in a ready to use object in the language of the consuming application, which receives a notification about the change and can act on it.

For our strategy engine, the data fabric then fires a notification and hands off the order object (now represented in C++) to the application which then acts on it.

Much of this work can be automated using tools that make it almost trivial to define a data model that is inherently faster and more efficient than traditional serialization mechanisms using XML marshalling, Java or C# serialization.

“Delta” Propagation: When a new incoming FIX message is a “change request” to a pending order, the application merely fetches the Order from the fabric and applies the change to one or more fields – for example, the customer wants to change the “buy” or “sell” volume of the trade, which would be updating a single field in the object. Given that the fabric maintains identity for all objects in the distributed system much like a database, it is now able to calculate the exact “delta” and just transmit that to the strategy engine or any number of connected applications that are listening for the incoming Order requests. By dramatically cutting down on the network and serialization overhead, “Delta” propagation allows the system to scale much better and push much more data through the system than would otherwise be possible.


Conclusion
XML based interoperability works well for a wide class of applications and very well suited to loosely couple applications, but, isn’t well suited for eXtreme Transaction processing applications. Messaging system lack enough context and hence prone to data integrity and consistency issues.

Distributed main memory based architecture such as distributed caching or data fabric (grid) technologies that combines the functions and semantics of database technology and reliable messaging technology may be a better fit.

Heterogeneous applications share a common object domain model - objects published or altered by one application is shared with other applications at memory speeds. The key to fast object sharing is the use of domain classes across these heterogeneous applications that are more or less similar, a native serialization protocol that can detect and dispatch object change "deltas" and an optimized neutral object wire format.

--------

Reliable pub Sub is an integral part of a Data Fabric

Can the data fabric (or a distributed cache with strong reliability semantics in message distribution) be used as a replacement for traditional messaging? This is a question we frequently get asked and this has been a sweet spot for us at GemStone. Here is some rant on the rationale:

Instead of a distributed main memory based data management solution that merely manages key-value pairs, imagine a solution that can manage objects along with relationships. In other words, ensures data integrity through knowledge about the entire data model. Ahh! like a database.
Now, combine this attribute with the ability to distribute data (replicate, partition, whatever) to many nodes, reliably and provide notifications to subsribing applications. What you get is a ACTIVE, distributed data management system - a system that inherently provides reliable pub-sub along with key semantics of the traditional data management system. Applications simply update a database (objects and relationships), express interest through complex query expressions on the data model and get delivered notifications based on their interest.

In a traditional messaging solution:

1. Publishing application has to explicitly construct messages, add header information for message identification and to allow subscribers to make sense of the message add enough contexual information. Often, this contextual information takes the form of identifiers (keys that point to the real data in some database).
2. Most messaging solutions use hub-spoke mechanisms to queue, relay messages to subscribers. i.e. messages hop from application process to some server process managed by the messaging provider and then to the receiving application. Look at many practical applications and you will find that the cost of messaging is not necessarily in the network, but, rather in the CPU costs associated with (de)serialization and all the copying of byte[] that goes on each process space and in the kernel layers.
3. Receiving application again has to do the reverse - parse headers, deserialize message body and then make sense of the message. To make any decision, often the application has to look up related data from other enterprise repositories such as a relational database, often slowing down the rate at which the entire flow can operate to the slowest link - often the relational database query.
   

Now, with a data fabric, all applications are sharing a single data model and express interest on the data model through simple, intuitive queries. The underlying fabric is constantly detecting what and how data and relationships are changing and simply sends notifications of the changes to the consuming application.

Note the following advantages with a data fabric:

1. Application processes are connected to each other in a p2p fashion with direct connections between each. This allows the fabric to avoid unnecessary network hops, dramatically reducing latency and CPU costs associated with message transfer.
2. As the data is typically held in multiple locations and often replicated to the process space of the consuming application, the publisher doesn't have to send obese messages - applications merely change the data fabric and underneath the hood the right "delta" event gets propagated to the consuming applications.
3. The receiving application when notified can take immediate business decisions as the contextual information they need is cached right there.

More later ....

Continuous Querying Article on August JDJ issue

Myself and gideon puttogether a rather simple use case to illustrae the power of the continuous querying technology and reached out to Java Developer Journal. They apparantly found this to be powerful enough - it became the front page feature articel for their August issue.

Java Feature — Building Real-Time Applications with Continuous Query Technology
— The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.
......

SOA n GRID synergistic?

A colleague of mine brought this SOA vs GRID aricle to my attention .... ' provides some interesting data on this topic and debunks certain popularly held beliefs..'



---------
Here is my take on the SOA-GRID synergy or lack thereof in practice followed by a discussion on where and if a distributed cache can fit in such architectures …..

Clearly the whole business of service orientation is based on the fact that one discovers a service based on desired operation and other QoS considerations, before binding. Service clients are loosely coupled to service providers by definition.
In the case where service clients come and go, say, like with a portal service that aggregates information from 10 other services, each request could be routed through a intermediary that isolates client from directly connecting to the server. This could be the UDDI registry for instance.

You agree with this, ya? Now, the question is, in what percentage of such SOA architectures will one use a dynamic provisioning service (loosely a GRID)? i.e. every time I use the discovery service it may point me to a different server to get the request fulfilled. In many cases, the expected load might be very predictable or the service provider has to run on specific platform that is my legacy and cannot be deployed into a utility computing center. Folks talk about high availability and hence the need for the Grid, but, I think, that is just bull. All SOA architectures will use a JEE or .Net façade and inherently be HA. So, basically, there is no case for a Grid in this situation.

But, more you look into the future, more likely I want my services to deployed using low cost commodity hardware, be bound by contracts on the QoS (availability and performance), forcing me to think about how my services can constantly be monitored and dynamically reprovisioned on the fly. The power of the Grid - well, actually a sophisticated provisioning and virtualization engine would become a necessity. I am not talking about a typical Grid solution, aka a compute grid - a scheduling engine.

In any case, like the who's who within the OGSA community would say, there is natural synergy between SOA and GRID. SOA is about how to connect services together to realize higher value (integration) and GRID is merely a deployment strategy for such services.

I thought, IBM was already on track to deliver on this promise within their WebSphere platform, circumventing the need for DS or EGO. Ahh! Who am I kidding?
Checkout Oracle's fusion strategy and how automatic provisioning and virtualization is integrated into 10G. Quite impressive.

Now, let me shift gear and see if and where a distributed caching solution fits in a SOA architecture .....

Where do databases and messaging solutions fit in? Just SOA architectures? Only in Compute Grid apps? Duh! everywhere. And, that is my position.
OK, here is the hiccup - again, SOA, by definition is all about getting apps (what do call this now … yeah! Services) talking to one another in a loosely coupled fashion. The service provider can change the underlying technology and behaviour any way it want and remains isolated from other services. I just have to make sure my contract is still intact. So, this cache thing breaks this, one could argue. The cynic would say 'I want a ESB bus that uses XML messaging not a bloody cache'.
Well, gentlemen, I got news for you. I think, all this loosely coupling stuff is baloney. Well, it holds water to a point, but no more.

The fact of the matter is, in real life, application models and yes, data models change and undergo significant enhancements. XML or no XML, your apps, if they need to talk to one another have to change.

You want to solve real problems, create a highly scalable, performant SOA architecture - use a memory based data fabric - An ESB designed for data intensive environments. You still want to use XML - shove data into the Enterprise Service Fabric (ESF) as XML. You don't like using APIs, but rather prefer the slippery SOAP (JAXM) route, use the web services interface.

And, now, fire storms have erupted in california ….

Introducing Distributed Data Fabric for the middle tier

You should read the prior post first, tell me what you agree with and where you disagree .....

What is the GemFire Enterprise Data Fabric?

GemFire Enterprise Data Fabric is a high performance, distributed operational data management infrastructure that sits between your clustered application processes and back-end data sources to provide very low latency, predictable, high throughput data sharing and event distribution.

It is about operational data management – Unlike a Data warehousing system where terabytes (or petabytes) of data is consolidated from multiple databases for offline data analysis, the EDF is a real-time data sharing facility specifically optimized for working with operational data needed by real-time applications – it is the “now” data, the fast moving data shared across many processes and applications. It is a layer of abstraction in the middle tier that collocates frequently used data with the application and works with backend databases behind the scenes.

Distributed Data Caching – the most important characteristic of the GemFire Data Fabric is that it is fast – many times faster than the traditional disk based database management system, because it is primarily main-memory based. Its engine harnesses the memory and disk across many clustered machines for unprecedented data access rates and scalability. It utilizes highly concurrent main-memory data structures to avoid lock contention and a data distribution layer that avoids redundant message copying, native serialization and smart buffering to ensure messages move from node to node faster than what traditional messaging would provide.

It does this without compromising the availability or consistency of data – a configurable policy dictate the number of redundant memory copies for various data types, storing data synchronously or asynchronously on disk and uses a variety of failure detection models built into the distribution system to ensure data correctness.

Key Database semantics are retained – simple distributed caching solutions provide caching of serialized objects – simple key-value pairs managed in Hashmaps that can be replicated to your cluster nodes. GemFire, provides support for multiple data models across multiple popular languages – data can be managed as Java or C++ objects natively, native XML documents or in SQL tables.

Similar to a Database management system, distributed data in GemFire can be managed in transactions, queried upon, persistently stored and recovered from disk.

Unlike a relational database management system, where all updates are persisted and transactional in nature (ACID), GemFire relaxes the constraints allowing applications to control when and for what kind of data you need total ACID (provide link) characteristics.

For instance, a very high performance financial services application trying to get price updates distributed what is most important is the distribution latency – there is no need for transactional isolation.

The end result is a data management system that spends fewer CPU cycles for managing data and offering higher performance.

Continuous Analytics

With data in the fabric changing rapidly as it is updated by many processes and external data sources it is important for real-time applications to be notified when events of interest are being generated in the fabric. Something a messaging platform is quite suited to do. GemFire data fabric takes this to the next level – applications can now register complex patterns of interest, expressed through SQL queries; Queries that are continuously running. Unlike a database system where queries have to be executed on resident data, here data (or events) is continuously evaluated by a query engine that is aware of the interest expressed by hundreds of distributed client processes.

Reliable messaging and routing

When using a messaging platform, application developers expect reliable and guaranteed Publish-Subscribe semantics. The system has knowledge about active or durable subscribers and provides different levels of message delivery guarantees to subscribers. GemFire EDF incorporates these messaging features on top of what looks like a database to the developer.

Unlike traditional messaging where applications have to deal with piecemeal messages, message construction, incorporating contextual information in messages, managing data consistency across publishers and subscribers, GemFire enables a more intuitive approach - one where applications simply deal with a data model (Object or SQL), subscribe to portions of the data model and publishers make updates to the business objects or relationships. Subscribers are simply notified on the changes to the underlying distributed database.

What makes GemFire EDF unique ?

For the last two decades or so, relational database management systems have taken a "kitchen sink" approach trying to solve any problem associated with data management by bundling this as part of the database engine.

Relational databases are centralized and passive in nature. It does a good job in managing data securely, correctly and persistently, but does not actively push the data to applications that might be interested, now. Second, databases are designed to optimize access to disk and to guarantee the transactional properties at all times. This limits the speed and scalability of a database engine in a highly distributed environment.

Compare this to a data environment where data storage structures are highly optimized for management in memory and concurrent access. To notify applications instantaneously, GemFire immediately routes data to the right node through a data distribution layer that is designed to reduce contention points and avoid unnecessary copies of messages before being transported.

Messaging solutions are most suited for very loosely coupled applications. Though this has its benefits, applications are left with the tough job of managing contextual information to make decisions, often requiring round trips to a database. This eliminates any performance advantages that applications can derive from messaging.

Besides this, often, the asynchronous nature of messages can also result in inconsistencies – the contextual information in the database may not reflect the correct state when the message is received.

GemFire provides an operational data infrastructure that brings data and events into one distributed platform – applications can focus on what matters most – operate on business objects and relationships. Interested applications are immediately notified as and when the data model changes. Data is co-located and accessible at memory speeds and data correctness is always ensured.

Modern day Event Driven Architectures require applications to react to events being pushed at very high rates from multiple streaming data sources, aggregate this data with other slow moving data managed in databases and distribute data and events to many application processes.

Traditional centralized databases simply are not designed to handle this mounting onslaught – what you need is a distributed memory based architecture that can analyze the incoming stream data, combine this with related information and present a consistent and correct data model to the application.

What makes GemFire unique, is this ability to not just analyze fast moving data, but the ability to present a data model (like a database) and route data/events to applications with guaranteed reliability (the semantics of reliable messaging).

Bottomline: Time has come for a middle tier data management layer to manage your operational data and events to enable a new generation of real-time applications with QoS guarantees on performance, continuous availability and scalability. You want to be able to do this, while retaining your investments in existing databases.

"One size fits all" Relational Database is dead. What say u?

OK, this is my first blog.
I will mostly blog on my professional life and it has a lot to do with high performance distributed computing. Especially, main-memory based distributed data management systems.

Let me begin by taking a look at the traditional relational database.

For the last two decades or so, major Database vendors have taken a "kitchen sink" approach trying to solve any problem associated with data management by bundling this as part of the database engine. Don't take my word on this. Here is how Jim gray's put this "We live in a time of extreme change, much of it precipitated by an avalanche of information that otherwise threatens to swallow us whole. Under the mounting onslaught, our traditional relational database constructs—always cumbersome at best—are now clearly at risk of collapsing altogether" Checkout this article . Adam Bosworth also has some interesting comments in his blog "Where have all the good databases gone"

Alright! What specifically I am talking about?

Consider this for starters:

Take a portal - today you want to build scalable systems using a clustered architecture where you can keep adding processing nodes in the middle tier so you can linearly scale as the number of users keeps going up.

Now, I don't have to tell you how important availability is for folks like these. Always up, always predictable in terms of performance and responsiveness. Enormous attention has been paid to make the middle tier highly available. But, alas, when it comes to the backend data sources, it is left upto the DB vendor.

The traditional database is built to do one thing very well - do a darn a good job in managing data securely, correctly and persistently on DISK. It is centralized by design. Everything is ACID. Ensuring high availability when you are constrained by the strong consistency rules (everything to disk) is very tough to manage. Replicate a database so you can failover to the replica during failure conditions, you are all of sudden left with random inconsistencies to deal with. Try to replicate synchronously, you pay a huge price in terms of performance. You want to provide dramatic scalability, through many replicated databases, you better be ready to live with a very compromised data consistency model.

Ahh! so, something like Oracle RAC is the answer, one might argue? Yes, for a number of use cases. But, here is what one has to consider:
1) You are still dealing with a disk centric architecture where all disks are shared and each DB process instance has equal access to all disks used to manage the tablespaces.
Here are a few important points worth noting:

• The unit of data movement in the shared buffer cache across the cluster happens to be a logical data block, which is typically a multiple of 8KB. The design is primarily optimized to make disk IO very efficient. AFAIK, even when requesting or updating a single row the complete block has to be transferred. Compare this to an alternative offering (a distributed main memory object management system) where the typical unit of data transfer is an object entry. In a update heavy and latency sensitive environment, the unit could actually be just the "delta" - the exact change.

• In RAC, If accessing a database block of any class does not locate a buffered copy in the local cache, a global cache operation is initiated. Before reading a block from disk, an attempt is made to find the block in the buffer cache of another instance. If the block is in another instance, a version of the block may be shipped. Again, there is no notion of a node being responsible for a block; there could be many copies of the block depending on how the query engine parallelized the data processing. This will be a problem if the update rate is high - imagine distributed locks on coarse grained data blocks in a cluster of 100 nodes; for every single update?

• At the end of the day, RAC is designed for everything to be durable to disk and requires careful thought and planning around a high speed private interconnect for buffer cache management and a very efficient cluster file system

Yes, there have been tremendous improvements to the relational database, but, it may not the answer for all our data management needs.

Consider this - How many enterprise class apps are being built that just depend on one database. Amazon's CTO, werner wogel talks about how 100 different backend services are engaged to just construct a single page that you and me see. Talk to a investment bank, events streaming in at mind boggling speeds have to be acted on not by one process, but, by potentially 100's of processes, instantenously. Everything needs to talk to one another in real-time. Consistent, correct data has to be shared across many processes built using heterogenous languages, and connected to heterogenous data sources all in real-time. Ahh! what do call this SOA, ESB, etc.

You can come up with a massive loosely coupled architecture for the connecting everything with everything, but, you have to pay extra attention to how you will manage data efficiently in this massively distributed environment.

Isn't it time for a middle-tier high performance data management layer that can capitalize on the abundant main-memory available on cheap commodity hardware today ? - a piece of middleware technology that can draw data from a multitude of data sources, move data between different applications with tight control on correctness and availability allowing all real-time operations to occur with a high and predictable Quality of Service.

Is this a distributed cache? A messaging solution?

OK, let me sell you what we do ....

Finally, I am blogging

Boy! this blog thing is new to me. Honestly, I have never blogged nor have I closely watched any blog. This nearly middle aged boy is "old school". Well, about me .... I do a lot of architecture for GemStone on paper :-) as their Chief Architect (www.gemstone.com). But, to my credit, I have been doing distributed data management stuff for, well... too long. I guess, googling me might give you a better idea ...

I guess, I am laggard when it comes to blogging. Better now, than never. It is all about the mindshare.

Serious bloggers, any advise to share ?