tag:blogger.com,1999:blog-314296852024-03-14T02:20:58.360-07:00JagsLogTechnical rants on distributed computing, high performance data management, etc. You are warned! A lot will be shameless promotion for VMWare productsJags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-31429685.post-60266757879820256622011-12-13T19:03:00.000-08:002011-12-13T22:48:31.484-08:00SQLFire 1.0 - the Data Fabric Sequel<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">This week we finally reached GA status for <a href="http://www.vmware.com/go/sqlfire">VMWare vFabric SQLFire</a> - </span><span class="Apple-style-span" style="font-family: Calibri, sans-serif; font-size: 15px;">a memory-optimized distributed SQL database delivering
dynamic scalability and high performance for data-intensive modern applications. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">In this post, I will highlight some important elements in our design and draw out some of our core values.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">The current breed of popular NoSQL stores promote different approaches to data modelling, storage architectures and consistency models to solve the scalability and performance problems in relational databases. The overarching messages in all of them seems to suggest that the core of the problem with traditional relational databases is SQL. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">But, ironically, the core of the scalability problem has little to do with SQL itself - it is the manner in which the traditional DB manages disk buffers, manages its locks and latches through a centralized architecture to preserve strict ACID properties that represents a challenge. Here is a slide from <a href="http://www.cs.brown.edu/~pavlo/presentations/hstore-hpts-oct2009.pdf">research at MIT and Brown</a> university on where the time is spent in OLTP databases. </span><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-jQ8H3oWv9oM/TueZpPmZ0VI/AAAAAAAAC6A/mSl2WnmaOQE/s1600/WhereDoesTheTimeGo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="297" src="http://4.bp.blogspot.com/-jQ8H3oWv9oM/TueZpPmZ0VI/AAAAAAAAC6A/mSl2WnmaOQE/s400/WhereDoesTheTimeGo.png" width="400" /></a></div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif; font-size: large;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif; font-size: large;">Design center</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">With </span><a href="http://www.vmware.com/go/sqlfire" style="font-family: Verdana, sans-serif;">SQLFire</a><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"> we change the design center in a few interesting ways:</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><b>1) Optimize for main memory</b>: we assume memory is abundant across a cluster of servers and optimize the design through highly concurrent data structures all resident in memory. The design is not concerned with buffering contiguous disk blocks in memory but rather manages application rows in memory hashmaps in a form so it can be directly consumed by clients. Changes are synchronously propagated to redundants in the cluster for HA. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><b>2) Rethink ACID transactions</b>: There is no support for strict serializable transactions but assume that most applications can get by with simpler "read committed" and "repeatable read" semantics. Instead of worrying about "read ahead" transaction logs on disk, all transactional state resides in distributed memory and uses a non-2PC commit algorithm optimized for small duration, non-overlapping transactions. The central theme is to avoid any single points of contentions like with a distribtued lock service. See some details <a href="http://pubs.vmware.com/vfabric5/topic/com.vmware.vfabric.sqlfire.1.0/developers_guide/topics/queries/transactions.html?resultof=%22%64%69%73%74%72%69%62%75%74%65%64%22%20%22%64%69%73%74%72%69%62%75%74%22%20%22%74%72%61%6e%73%61%63%74%69%6f%6e%22%20%22%74%72%61%6e%73%61%63%74%22%20">here</a>.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<b style="font-family: Verdana, sans-serif;">3) "Partition aware DB design"</b><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">: Almost every single high scale DB solution offers a way to linearly scale by hashing keys to a set of partitions. But, how do you make SQL queries and DML scale when they involve joins or complex conditions? Given that distributed joins inherently don't scale we promote the idea that the designer should think about common data access patterns and choose the partitioning strategy accordingly. To make things relatively simple for the designer, we extended the DDL (Data definition language in SQL) so the designer can specify how related data should be colocated ( for instance '</span><i><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">create table Orders (...) colocate with Customer</span></i><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">' tells us that the order records for a customer should always be colocated onto the same partition). The colocation now makes join processing and query optimization a local partition problem (avoids large transfers of intermediate data sets). The design assumes classic OLTP workload patterns where vast majority of individual requests can be pruned to a few nodes and that the concurrent workload from all users is spread across the entire data set (and, hence across all the partitions). Look here for some<span id="goog_1449061737"></span></span> <a href="http://pubs.vmware.com/vfabric5/topic/com.vmware.vfabric.sqlfire.1.0/data_management/database_design_chapter.html?resultof=%22%64%65%73%69%67%6e%69%6e%67%22%20%22%64%65%73%69%67%6e%22%20%22%56%46%61%62%72%69%63%22%20%22%76%66%61%62%72%69%63%22%20%22%53%51%4c%66%69%72%65%22%20%22%73%71%6c%66%69%72%65%22%20" style="font-family: Verdana, sans-serif;">details</a><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span id="goog_1449061738"></span>.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><b>4) Shared nothing logs on disk</b>: Disk stores are merely "append only" logs and designed so that application writes are never exposed to the disk seek latencies. Writes are synchronously streamed to disk on all replicas. A lot of the disk store design looks similar to other NoSQL systems - rolling logs, background/offline compression, memory tables pointing to disk offsets, etc. But, the one aspect that represents core IP is all around managing consistent copies on disk in the face of failures. Given that distributed members can come and go, how do we make sure that the disk state a member is working with is the one I should be working with? I cover our "shared nothing disk architecture" in lot more detail <a href="http://jagslog.blogspot.com/2010/10/what-is-new-in-vmware-vfabric-gemfire.html">here</a>.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><b>5) Parallelize data access and application behavior</b>: We extend the classic stored procedure model by allowing applications to parallelize the procedure across the cluster or just a subset of nodes by hinting the data the procedure is dependent on. This applicaton hinting is done by supplying a "where clause" that is used to determine where to route and parallelize the execution. Unlike traditional databases, procedures can be arbitrary application Java code (you can infact embed the cluster members in your Spring container) and run collocated with the data. Yes, literally in the same process space where the data is stored. Controversial, yes, but, now your application code can do a scan as efficiently as the database engine.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><b>6) Dynamic rebalancing of data and behavior</b>: This is the act of figuring out what data buckets should be migrated when new capacity (cluster size grows) is allocated (or removed) and how to do this without causing consistency issues or introducing contention points for concurrent readers and writes. Here is the <a href="http://brucesch.blogspot.com/2011/09/patent-granted.html">patent</a> that describes some aspects of the design. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif; font-size: large;">Embedded or a client-server topology</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">SQLFire supports switching from the classic client-server (your DB runs in its own processes) topology to embedded mode where the DB cluster and the application cluster is one and the same (for Java apps). </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">We believe the emdedded model will be </span><span style="text-indent: -24px;"><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">very useful in
scenarios where the data sets are relatively small. It simplifies deployment concerns and at the same
time provides significant boost in performance when replicated tables are in
use.</span></span><br />
<div class="MsoListParagraph" style="mso-list: l0 level1 lfo1; text-indent: -.25in;">
<o:p></o:p></div>
<br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">All you do is change the DB URL from</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">'jdbc:sqlfire://server_Host:port' to 'jdbc:sqlfire:;mcast-port=portNum'</span><span class="Apple-style-span" style="font-family: Verdana, sans-serif;"> and now all your application processes that use the same DB URL will become part of a single distributed system. Essentially, the mcast-port port identifies a broadcast channel for membership gossiping. New servers will automatically join the cluster once authenticated. Any replicated tables will automatically get hosted in the new process and partitioned tables could get rebalanced and share some of the data with the new process. All this is abstracted away from the developer. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">As far as the application is concerned, you just create connections and execute SQL like with any other DB.</span><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-FhYbmax3vt8/TufCVtx6RMI/AAAAAAAAC6I/1NNX866Qy_g/s1600/topology1-whitebg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="263" src="http://2.bp.blogspot.com/-FhYbmax3vt8/TufCVtx6RMI/AAAAAAAAC6I/1NNX866Qy_g/s400/topology1-whitebg.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-SH_f1m5JXjM/TufCWRjB8HI/AAAAAAAAC6Q/gq7eFOuP40c/s1600/topology2-whitebg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="236" src="http://1.bp.blogspot.com/-SH_f1m5JXjM/TufCWRjB8HI/AAAAAAAAC6Q/gq7eFOuP40c/s400/topology2-whitebg.png" width="400" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif; font-size: large;">How well does it perform and scale? </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Here are the results of a simple benchmark done internally using commodity (2 CPU) machines showcasing linear scaling with concurrent user load. I will soon augment this with more interesting workload characterization. The details are </span><a href="http://communities.vmware.com/docs/DOC-15958" style="font-family: Verdana, sans-serif;">here</a><span class="Apple-style-span" style="font-family: Verdana, sans-serif;">.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://communities.vmware.com/servlet/JiveServlet/showImage/102-15958-4-15560/throughput.GIF" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="http://communities.vmware.com/servlet/JiveServlet/showImage/102-15958-4-15560/throughput.GIF" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://communities.vmware.com/servlet/JiveServlet/showImage/102-15958-4-15562/response.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="http://communities.vmware.com/servlet/JiveServlet/showImage/102-15958-4-15562/response.gif" width="400" /></a></div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><span class="Apple-style-span" style="font-size: large;">Comparing SQLFire and GemFire</span></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Here is a high level view into how the two products compare. I hope to add a blog post that provides specific details on the differences and use cases where one might apply better than the other.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-BOssomduhDc/TugRuXEIzBI/AAAAAAAAC6Y/1nmZ3_mpGng/s1600/gemfire_sqlfire_positioning.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-BOssomduhDc/TugRuXEIzBI/AAAAAAAAC6Y/1nmZ3_mpGng/s400/gemfire_sqlfire_positioning.JPG" width="400" /></a></div>
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="color: #1f497d; font-family: Calibri, sans-serif; font-size: 15px;">SQLFire
benefits from the years of commercially deployed production code found in
GemFire</span><span class="Apple-style-span" style="color: #1f497d; font-family: Calibri, sans-serif; font-size: 15px;">. SQLFire adds a rich SQL engine with the idea that now folks can manage operational data primarily in memory, partitioned across any number of nodes and with a disk architecture that avoids disk seeks. Note the two offerings, SQLFire and GemFire, are completely unique products and deployed separately</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">As always, I would love to get your candid feedback (<a href="http://communities.vmware.com/community/vmtn/appplatform/vfabric_sqlfire?view=discussions">link to our forum</a>). I assure you that trying it out is very simple - just like using Apache Derby or H2. </span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;">Get to the download, docs and quickstart all from <a href="http://communities.vmware.com/community/vmtn/appplatform/vfabric_sqlfire">here</a>. The developer license is perpetual and works on upto 3 server nodes.</span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana, sans-serif;"><br /></span><br />
<br /></div>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com1tag:blogger.com,1999:blog-31429685.post-53870467107140855942011-11-20T21:13:00.001-08:002011-12-03T11:37:47.116-08:00HPTS 2011 talk on 'Flexible OLTP in the future'<div dir="ltr" style="text-align: left;" trbidi="on">
I recently spoke at HPTS 2011 (<a href="http://hpts.ws/agenda.html">High Performance Transaction Systems</a>). If you haven't already you should check out some of the very interesting content on NoSQL ecosystem, future in core density, big data experiences and scars, etc.<br />
<br />
Here is the abstract:<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Flexible OLTP data models in the future</span><br />
=================================<br />
<br />
There has been a flurry of highly scalable data stores and a dramatic spike in the interest level. The solutions with the most mindshare seem to be inspired by <a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCsQFjAA&url=http%3A%2F%2Fwww.allthingsdistributed.com%2Ffiles%2Famazon-dynamo-sosp2007.pdf&ei=L97JTueNJqnYiQLG9ay_Dw&usg=AFQjCNHhJccl0_0I9x7tkWizMx6NjcuUkQ">Dynamo's</a> (Amazon) eventually consistency model or a data model that promotes nested, self-describing data structures like<a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB4QFjAA&url=http%3A%2F%2Flabs.google.com%2Fpapers%2Fbigtable-osdi06.pdf&ei=Xd7JTsXWKcqWiAKS_czWDw&usg=AFQjCNHN6UZKpTHsquOhtbcjMa06GqDPQQ"> BigTable</a> from Google. At the same time you see projects within these corporations evolving to architectures like <a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&ved=0CDIQFjAD&url=http%3A%2F%2Fresearch.google.com%2Fpubs%2Farchive%2F36971.pdf&ei=dt7JTrn6Ga7OiAKJ7I0H&usg=AFQjCNE4SCcFM_Uh2bnyyWU6sm8Kbp7uvQ">MegaStore </a>and Dremel (Google) where features from the column-oriented data model is blended together with the relational model.<br />
<br />
The shift from just highly structured data to unstructured and semistructured content is evident. New applications are being developed or existing applications are being modified at break neck speed. Developers want the data model evolution to be extremely simple and want support for nested structures so they can map to representations like JSON with ease so there is little impedance between the application programming model and the database. Next generation enterprise applications will increasingly work with structured and semi-structured data from a multitude of data sources. A pure relational model is too rigid and a pure BigTable like model has too many shortcomings and cannot be integrated with existing relational databases systems.<br />
<br />
In this talk, I present an alternative. We prefer the familiar "row oriented" over "column oriented" approach but still tilt the relational model - mostly the schema definition to support partitioning and colocation, redundancy level and support for dynamic and nested columns.<br />
Each of these extensions will support different desired attributes - partitioning and colocation primitives cover horizontal scaling, availability primitives allow explicit support for replication model and the placement policies (local vs across data centers), dynamic columns will address flexibility for schema evolution (different rows have different columns and added with no DDL requirements) and nested columns that support organizing data in a hierarchy.<br />
<br />
We draw inspiration for the data model from <a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CBwQFjAA&url=http%3A%2F%2Fwww.ics.uci.edu%2F~cs223%2Fpapers%2Fcidr07p15.pdf&ei=pt7JTum_IMqhiQKYreTcDw&usg=AFQjCNFDghTdIhZ-z8bPCYa3CAs0IBGZqw">Pat helland's 'Life beyond distributed transactions'</a> by adopting entity groups as a first class artifact designers start with, and define relationships between entities within the group (associations based on reference as well as containment). Rationalizing the design around entity groups will force the designer to think about data access patterns and how the data will be colocated in partitions. We then cover why ACID properties and sophiticated querying becomes significantly less challenging to accomplish. There are many ideas around partitioning policies, tradeoffs in supporting transactions and joins across entity groups that are worth discussion.<br />
<br />
The idea is to present a model and generate discussion on how to achieve the best of both worlds. Flexible schemas without losing referential integrity, support for associations and the power of SQL. It is ironic that NoSQL databases like Mongodb are getting to be more popular as they begin to add SQL like querying capabilities.<br />
<div>
<br /></div>
<div>
<div>
<div id="__ss_10248211" style="width: 425px;">
<strong style="display: block; margin: 12px 0 4px;"><a href="http://www.slideshare.net/jagsr123/hpts-2011-flexibleoltp" title="Hpts 2011 flexible_oltp">Hpts 2011 flexible_oltp</a></strong><object height="355" id="__sse10248211" width="425"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=hpts2011flexibleoltp-111120230610-phpapp02&stripped_title=hpts-2011-flexibleoltp&userName=jagsr123" />
<param name="allowFullScreen" value="true"/>
<param name="allowScriptAccess" value="always"/>
<param name="wmode" value="transparent"/>
<embed name="__sse10248211" src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=hpts2011flexibleoltp-111120230610-phpapp02&stripped_title=hpts-2011-flexibleoltp&userName=jagsr123" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" wmode="transparent" width="425" height="355"></embed></object><br />
<div style="padding: 5px 0 12px;">
View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/jagsr123">Jags Ramnarayan</a>.</div>
</div>
</div>
</div>
<div>
<br />
Finally, <a href="http://hpts.ws/summaries.html">this summarizes</a> all the different views shared at HPTS.</div>
</div>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-79747835003139828052011-09-19T15:53:00.000-07:002011-09-20T12:44:57.041-07:00What is new in vFabric GemFire 6.6?<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<a href="https://www.vmware.com/support/pubs/vfabric-gemfire.html">GemFire 6.6</a> was released (Sept 2011) as part of the new <a href="http://blog.springsource.com/2011/06/14/announcing-vfabric-5/">vFabric 5.0</a> product suite and, it represents a big step along the following important dimensions:<br />
<ol style="text-align: left;">
<li><b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">developer productivity</span></b></li>
<li><b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">more DBMS like features </span></b></li>
<li><b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">better scaling features</span></b></li>
</ol>
<br />
Here are some highlights on each dimension:<br />
<br />
<b><span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;">Developer productivity</span></b>: Introduced a new <a href="http://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.gemfire.6.6/developing/data_serialization/gemfire_pdx_serialization.html">serialization framework called PDX</a> (stands for Portable Data eXchange and not my favorite airport).<br />
PDX is a framework that provides a portable, compact, language neutral and versionable format for representing object data in GemFire. It is proprietary but designed for high efficiency. It is comparable to other serialization frameworks like <a href="http://avro.apache.org/">apache Avro</a>, <a href="http://code.google.com/p/protobuf/">Google protobuf</a> ,etc. <br />
Alright. I realize the above definition is a mouth full :-)<br />
<br />
Simply put, the framework supports versioning allowing apps using older class versions to work with apps with newer versions of the domain classes and vice versa, provides a format and type system for interop between the various languages and an API so server side application code can operate on objects without requiring the domain classes (i.e. no deserialization).<br />
The type evolution has to be incremental - this is the only way to avoid data loss or exceptions.<br />
The raw serialization performance is comparable to Avro, protobuf but is much more optimized for distribution and operating in a GemFire cluster. The chart below is the result of a open source benchmark on popular serialization frameworks. The details are available <a href="https://github.com/eishay/jvm-serializers/wiki">here</a>. 'Total' represents the total time required to create, serialize and then deserialize. See the benchmark description for details.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-c5SyjfkwUvQ/TnfBn4hBHhI/AAAAAAAAC5U/lfwGaL3lvwM/s1600/serializationBenchmark.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="131" src="http://1.bp.blogspot.com/-c5SyjfkwUvQ/TnfBn4hBHhI/AAAAAAAAC5U/lfwGaL3lvwM/s400/serializationBenchmark.JPG" width="400" /></a></div>
<br />
You can either implement serialization callbacks (for optimal performance) or simply use the <a href="http://pubs.vmware.com/vfabric5/topic/com.vmware.vfabric.gemfire.6.6/developing/data_serialization/auto_serialization.html">built in PDXSerializer</a> (reflection based today). Arguably, the best part of the framework is its support for object access in server side functions or callbacks like listeners without requiring the application classes. You can dynamically discover the fields and nested objects and operate on these using the <a href="http://www.vmware.com/support/developer/vfabric-gemfire/66-api/com/gemstone/gemfire/pdx/package-frame.html">PDX API</a>. On the application client that has the domain classes the same PDXInstance is automatically turned into the domain object.<br />
<br />
We introduced a new <a href="http://pubs.vmware.com/vfabric5/topic/com.vmware.vfabric.gemfire.6.6/deploying/gfsh/chapter_overview.html?resultof=%22%67%66%73%68%22%20">command shell called gfsh</a> (pronounced "gee - fish" ) - a command line tool for browsing and editing data stored in GemFire. Its rich set of Unix-flavored commands allows you to easily access data, monitor peers, redirect outputs to files, and run batch scripts. This is an initial step towards a more complete tool that can provision, monitor, debug, tune and administer a cluster as a whole. Ultimately, we hope to advance the gfsh scripting language making integration of GemFire deployments into cloud like virtualized environments a "breeze".<br />
<br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>More DBMS like: </b></span><br />
<b>Querying and Indexing</b><br />
we added several features to our query engine - query/index on hashmaps, bind parameters from edge clients, OrderBy support for partitioned data regions, full support for LIKE predicates and being able to index regions that overflow to disk.<br />
<br />
Increasingly we see developers wanting to decouple the data model in GemFire from the class schema used within their applicatons. Even though PDX offers an excellent option, we also see developers mapping their data into "self describing" hashmaps in GemFire. The data store is basically "<a href="http://stackoverflow.com/questions/2117372/what-are-the-advantages-of-using-a-schema-free-database-like-mongodb-compared-to">schema free</a>" and allows many application teams to change the object model without impacting each other. Given a simple KV storage model in GemFire this has never been an issue except for querying. Now, not only can you store maps, you can index keys within these Hashmaps and execute highly performant queries.<br />
Do take note that the query engine now natively understands the PDX data structures with no need for application classes on servers.<br />
<br />
We expanded distributed transactions by allowing edge clients to initiate or terminate transactions. No need to invoke a server side function for transactions. We also added a new JCA resource adapter that supports participation in externally coordinated transactions as a <a href="http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jta/llr.html">"Last resource"</a>.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Trebuchet MS', sans-serif;"><b>Finally, on the scaling dimension:</b></span><br />
You are probably aware that <a href="http://jagslog.blogspot.com/2010/10/what-is-new-in-vmware-vfabric-gemfire.html">GemFire's shared nothing persistence</a> relies on append-only operation logs to provide very high write throughput. There are no additional Btree data files to maintain like in a traditional database system. The tradeoff with this design is cluster recovery speed. One has to walk through the logs to recover the data back into memory and the time for the entire cluster to bootstrap from disk is proportional to the volume of data (and inversely proportional to the cluster count). And, this can be long (put mildly) with large data volumes even though you can parallelize the recovery across the cluster. To minimize this recovery delay, the 6.6 persistence layer now also manages "key files" on disk. We simply recover the keys back into memory and lazily recover the data giving recovery in general a significant performance boost.<br />
<br />
Prior to 6.6, GemFire randomly picked a different host to manage redundant copies for partitioned data regions. Often, customers provision multiple racks and want their redundant copy to always be stored on a different physical rack. Occasionally, we also see customers wanting to store their redundant data on a different site. We added support for "redundancy zone" in partitioned region configuration allowing users to identify one or more redundancy zones (could be racks, sites, etc). GemFire will automatically enforce managing redundants in different zones.<br />
<br />
Everything mentioned happens to be more of a prelude. The list of enhancements is much longer and is documented <a href="http://www.vmware.com.pr/support/vfabric-gemfire/doc/vfabric-gemfire-rn-6.6.0.html">here</a>.<br />
<br />
The product documentation is available <a href="http://www.vmware.com/support/pubs/vfabric-gemfire.html">here</a>.<br />
<div>
You can start discussions <a href="http://forums.gemstone.com/viewforum.php?f=3&sid=53ef625e9124d5ba27ceeaa18687effa">here</a>.</div>
<div>
<br /></div>
<div>
Would love to hear your thoughts. </div>
<div>
<br /></div>
</div>
Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com2tag:blogger.com,1999:blog-31429685.post-37678759153548076222010-11-14T21:42:00.000-08:002010-12-06T18:53:34.348-08:00Announcing vFabric GemFire HTTP session management module for Tomcat/TCServer<a href="http://en.wikipedia.org/wiki/Session_management">Clustering of HTTP Sessions</a> has been around for a while. I suspect many of you are already wondering how is this better or unique compared to the built-in clustering mechanisms in application servers or third party solutions like EHCache, Oracle Coherence, etc. Several of our customers already manage sessions in <a href="http://community.gemstone.com/display/gemfire/GemFire+Enterprise">GemFire</a> explicitly using our API. The need to effectively manage large quantities of session state is too great not to use a distributed data grid solution such as GemFire, which can partition session state in memory across the cluster for scale and maintain synchronous copies of session state in memory on multiple nodes to provide high availability (HA).<br />
<br />
<b>So, what is new</b>? This announcement brings clean, pluggable integration of GemFire within Tomcat and <a href="http://www.springsource.com/products/tcserver">SpringSource tc Server</a> environments. HTTP session management is enabled through very simple configuration steps. Unlike other competitive products, there is no vendor-specificonfiguration of "cache" XML in most cases. If you understand your application volume and scale requirements then it might just be a matter of configuring one of the pre-packaged templates when you start up tc Server instances.<br />
<br />
You can read about the features, download and give it a try from our <a href="http://community.gemstone.com/display/gemfire/HTTP+Session+Management+Module">community site</a>.<br />
<br />
But, there are several interesting patterns that go beyond this basic value proposition and I will attempt to cover these below ...<br />
<br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Support for multi site clustering</b></span>: Today, HA for sessions implies copies maintained on 2 or more servers that are part of the same cluster. If the entire cluster were to fail or becomes unreachable (say a network split occurs) your applications is likely going to fail over to some DR cluster but without access to the sessions. Wouldn't it be nice if your session state survives such failure conditions? With Gemfire, you can configure the session state to be replicated over what we call "<a href="http://community.gemstone.com/display/gemfire/Setting+Up+GemFire+HTTP+Session+Management+for+tc+Server">WAN gateways</a>"(See section "multi site setup") to your DR cluster. The replication is asynchronous with support for batching and conflation.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/_o_zMfh3Veig/TODFVx5G7eI/AAAAAAAACz0/RrxfX8Mht-Y/s1600/gateway.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="210" src="http://1.bp.blogspot.com/_o_zMfh3Veig/TODFVx5G7eI/AAAAAAAACz0/RrxfX8Mht-Y/s320/gateway.png" width="320" /></a></div><br />
<br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Sessions are getting increasingly obese and may never die</b></span>: Session state often reference personalization information - things like preferences, past transactions and even things like past chat sessions with customer support. Traditionally, all this data was only managed in the RDB and fetched every single time the user logs in - one of the main causes of database overload. Increasingly, I see sessions that range in size from a few KB to several MB each. These are represented by convoluted object graphs and when changes occur, they represent only a minuscule fraction of the entire session state. Managing complex, constantly changing and ever growing session state requires special consideration:<br />
<ol><li>when sessions change the session replication layer needs to be smart about only replicating the changes and not the entire session every single time. </li>
<li>User Sessions may come and go but the associated state may last forever. This implies you cannot maintain everything in memory. You need to offload to disk as well as a super efficient mechanism to persist the state to disk across the cluster. This persistence mechanism cannot be expensive and shouldn't require administration overhead like with common RDBs.</li>
</ol>With GemFire, when sessions are updated only updated attributes are sent over the wire - to servers and replicas. Session state can overflow to disks across the cluster and can even be made persistent. The state will recover in parallel even if the entire cluster were to be restarted.<br />
<br />
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Burst into the cloud</b></span>: There is increasing interest in the ability to go beyond what the cluster can handle when the load threshold is too great. Ideally, applications can burst into the "cloud" – most likely a private cloud that supports on-demand provisioning of resources using virtualization. Bursting, basically, means new Tomcat instances would get sparked on demand in some remote data center. This may also mean migrating users to the new cluster. Now, wouldn't it be nice if the session state were to magically appear or be accessible from the primary cluster?<br />
Alright! in reality, magic is just an illusion. Nothing happens magically in GemFire. Let me spare you a long narration. With GemFire, one can achieve this by configuring WAN gateways to potential remote clusters. If and when a remote "cloud" cluster were to be launched, the sessions will automatically land up getting replicated.<br />
<br />
<b><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Sessions span heterogeneous apps</span></b>: Session state published by one clustered application may need to be accessible by other distributed applications. These applications won't necessarily be deployed as part of the same application server cluster. <span class="Apple-style-span" style="font-family: Times, 'Times New Roman', serif;">You need session state that can outlast your application or your cluster, with a storage format such that it is accessible from other languages. </span>Though we don't support pluggable session management for other languages like .NET(yet), the application developer can still use GemFire native serialization and access the session state from other environments like C# and C++. We intend to support seamless session access across many environments in the future.<br />
<br />
So, again, You can read about the features, download and give it a try from our <a href="http://community.gemstone.com/display/gemfire/HTTP+Session+Management+Module">community site</a>.Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com4tag:blogger.com,1999:blog-31429685.post-57228618303197493962010-10-07T12:49:00.000-07:002010-10-07T23:12:07.171-07:00What is new in VMWare's vFabric GemFire 6.5?Given the breadth of new capabilities, <a href="http://community.gemstone.com/display/gemfire/GemFire+Enterprise"><span class="Apple-style-span" style="color: orange;">GemFire 6.5</span></a> might as well have been called 7.0. One of the important themes for us has been to make sure all stateful services can be partitioned for near linear scaling. With this release we go beyond partitioning in-memory data and application behavior. We can now manage data on disk in a highly partitioned manner and even process subscriptions with linear scaling. Our pursuit is simple - no matter which features of the product are being used, there is a level of assurance that the application will scale with increasing capacity.<br />
<br />
<br />
In a nutshell(it is a big nut), here are some of the capabilities we introduced:<br />
<br />
<ul><li><b>Database class reliability through Shared-nothing parallel persistence</b>: A unique high performance design to pool disks across the cluster for storing and recovering data. GemFire always supported disk persistence and recovery for replicated data regions but now this capability has been extended for partitioned data also. The design principles adopted are fundamentally different than ones in typical clustered databases. For one, disk storage is shared nothing - each cache process owns its disk store eliminating process level contentions. Second, the design is tilted to favor memory i.e. there are no complex B-Tree data structures on disk; instead we assume complex query navigations will always be done through in-memory indexes. Third, the design uses rolling append-only log files to avoid disk seeks completely. Finally, the design preserves the rebalancing model in GemFire when capacity is increased or decreased - the disk data also relocates itself. Well, there is more to the story which I will cover once we walk through what else is new in 6.5..</li>
<li><b>Simplified and Intuitive programming model</b>: First, we simplified the programming model by reducing some of the boiler plate bootstrapping code that was required in the past and introduced pre-packaged templates for common configurations and topologies. Second, we launched the new <a href="http://www.springsource.org/spring-gemfire"><span class="Apple-style-span" style="color: orange;">Spring-Gemfire</span></a> project to introduce a clean programming model for the Spring developer. Note that Spring-GemFire is not bundled with GemFire 6.5. If you are already familiar with the GemFire APIs or just getting started, I would recommend going through the new tutorial that walks through some of the new <a href="https://community.gemstone.com/display/gemfire/GemFire+Tutorialhttps://community.gemstone.com/display/gemfire/GemFire+Tutorial"><span class="Apple-style-span" style="color: orange;">simplified programming API</span>s</a>. And, not to worry - your existing application will continue to just run fine. The old APIs are fully supported.</li>
<li><b>Improved scale-out capabilities</b>: Application deployments using the tiered model (client process embedding a local cache talking to a server farm) could see a 2X or more performance gain when accessing partitioned data. With 6.5, client processes gain knowledge about server side partitions and use it on the fly to direct traffic directly to the server with the required data set. Application clients subscribing to events using key based register interest or "continuous queries" now have their subscriptions registered on all the partitioned nodes. This allows each partition to process the subcription filters on the nodes where the data changes are applied dramatically reducing messaging traffic between peers unlike in the previous releases. The net effect is that more subscriptions can be processed and the event delivery latency to clients is also reduced.</li>
<li><b>Colocated transactions</b>: If each logical partition were to own the entire transactional working set then highly applications can scale linearly if the concurrent transaction load is uniformly spread across the data set and hence across all the partitions. Each partition can coordinate its transaction without any impact to other partitions with no locking requirements across partitions. GemFire 6.5 introduces a change to the distributed transaction design to detect cases where the data is colocated and avoids engaging the built-in distribued lock service.</li>
<li><b>C++, C# client improvements</b>: The clients can now receive and send object change "deltas", CQ processing on the client has improved, Parallel Function execution on the Grid is automatially HA, SSL and user level security support has been added and the client cache sizes can be significantly larger with support for 64-bit architectures.</li>
</ul><br />
<div><div>For a complete list of features added in 6.5, click <a href="http://community.gemstone.com/pages/viewpage.action?pageId=6032141"><span class="Apple-style-span" style="color: orange;">here</span></a>. </div><div><br />
</div><div>Next, I will rant and rave our disk persistence story.</div><div><br />
</div><div><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Factors contributing to very high disk throughput:</b></span></div><div><br />
</div><div><b><i>Pooling</i></b>: Like mentioned before, each cache instance manages its own disk store and there is no disk contention between processes. Each partition can locally manage its data on local disk(s). Assuming application "write" load can be uniformly balanced across the cluster, the aggregate disk throughput will be (Disk transfer rate * NumOfPartitions), assuming a single disk per partition. Disk transfer rates can be upto 100MB/sec on commodity machines today compared to just 2MB/sec in the 1980s. </div><div><br />
</div><div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/_o_zMfh3Veig/TK1i58lzD4I/AAAAAAAACs4/x_Pbdyuo_RI/s1600/diskRates.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="81" src="http://1.bp.blogspot.com/_o_zMfh3Veig/TK1i58lzD4I/AAAAAAAACs4/x_Pbdyuo_RI/s400/diskRates.gif" width="400" /></a> </div><div class="separator" style="clear: both; text-align: center;"><span class="Apple-style-span" style="font-size: small;">reference : </span><a href="http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf"><span class="Apple-style-span" style="font-size: small;">http://www.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf</span></a></div><div><br />
</div><div><b><i>Avoid seeks</i></b>: By managing most (or all) of the data in cluster memory all reads are served without navigating through BTree based indexes and data files on disk, which as you know will result in continuous seeking on disk. The average disk seek times today are still 2ms or higher. </div></div><div><br />
</div><div><br />
</div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/_o_zMfh3Veig/TK1kSuQnycI/AAAAAAAACs8/xPqI0p8MImY/s1600/diskPersist1.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="404" src="http://4.bp.blogspot.com/_o_zMfh3Veig/TK1kSuQnycI/AAAAAAAACs8/xPqI0p8MImY/s640/diskPersist1.gif" width="640" /></a></div><div><br />
</div><div><b><i>Buffered logging</i></b>: When writes do occur, these operations are simply logged to disk in "append-only" log/data files (See figure above). Appending implies we can continuously write to consecutive sectors on disk without requiring disk head movement. Probably the most controversial decision we had to make was to allow all writes to only be flushed to the OS buffer rather than 'fsync' all the way to disk. The writes are buffered by the IO subsystem in the kernel allowing the IO scheduler to merge and sort disk writes to achieve the highest possible disk throughput. The write requests need not initiate any disk I/O until some time in the future. Thus, from the perspective of a user application, write requests stream at much higher speeds, unencumbered by the performance of the disk. Any data loss risks due to sudden failures at a hardware level is mitigated by having multiple nodes write in parallel to disk. In fact, it is assumed that hardware will fail, especially in large clusters and data centers and software needs to take into account such failures. The system is designed to recover in parallel from disk and to guarantee data consistency when data copies on disk don't agree with each other. Each member of the distributed system logs membership changes to its persistent files and uses this during recovery to figure out the replica that has the latest changes and automatically synchronizes these changes at startup.</div><div><br />
</div><div><br />
</div><div><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"><b>Motivations for native GemFire persistence instead of an RDB</b></span></div><div><br />
</div><div>Most data grid deployments today use a RDB as the backing store. A synchronous design where every change is reflected in the database first has obvious challenges for "write heavy" applications. You are only as fast as the RDB and the database could become the single point of failure (SPOF). Transaction execution complexity also increases involving the data grid and the database in a 2-phase commit protocol. </div><div>So, one remedy is to execute all writes on the data grid and asynchronously propagate the changes to the RDB. This pattern also has the same SPOF challenges and not well suited for cases with sustained high write rates. </div><div>Designs that go with Database "shards" - each cache instance writing to its own independent DB instance for scale is interesting but tough to actually implement with good HA characteristics. </div><div><br />
</div><div>We believe that the GemFire parallel persistence model gets rid of all these limitations. Now, I am not advocating that you shouldn't move the data in the grid back to your classic RDB. You probably need to do this for all sorts of reasons as you pipe the information to upstream and downstream applications. But, think twice if your RDB is merely meant to be a backing store.</div><div>Even with customers that remain apprehensive with GemFire acting as the backing store, some want to still manage data on Gemfire disk stores just so that the cluster recovery can be fast. </div><div><br />
</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Handling failure conditions</span></b></div><div><br />
</div><div>The choice to use buffered logs meant we had to make sure the data is written to disk on multiple nodes for reliability. When partitions failed their disk stores can become stale. If later the entire cluster were to be bounced, the recovery logic has to make sure the state of the data in-memory and on disk reflected the latest state. We spent a lot of our energy in making sure the design for recovery always gaurantees consistency and freshness of data. </div><div><br />
</div><div><b><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Summary</span></b></div><div>In summary, the design is fundamentally very different from the traditional DB approach - data is stored on disk and memory is used to optimize disk IO (manage raw disk blocks instead of app data objects). Instead, in our thinking, data is primarily managed in memory and disk is primarily used for recovery and to address safety concerns. <i>Disk capacity has increased more than 10000-fold over the last 25 years and seems likely to continue increasing in the future. Unfortunately, though, the access rate to information on disk has improved much more slowly: the transfer rate for large blocks has improved “only” 50-fold, and seek time and rotational latency have only improved by a factor of two</i>.</div><div><br />
</div></div><div>You can read the product documentation on persistence<span class="Apple-style-span" style="color: orange;"> </span><a href="http://www.gemstone.com/docs/6.5.0/product/docs/html/Manuals/SystemAdministratorsGuide/wwhelp/wwhimpl/common/html/wwhelp.htm#context=SystemAdministratorsGuide&file=diskStores.7.2.html"><span class="Apple-style-span" style="color: orange;">here</span></a>.</div><div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><a href="http://www.springsource.org/spring-gemfire"><span class="Apple-style-span" style="font-size: x-large;"><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;">Spring-GemFire</span></span></a><span class="Apple-style-span" style="font-size: x-large;"><span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', serif;"> integration</span></span></div><div><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 3px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">We outlined a high level technology integration strategy to our customers and the press when we merged with SpringSource/VMWare on how GemFire will integrate with the spring framework (Java and .NET). The core Spring engineers have already delivered on part of the promise through the first milestone release making it natural for the Spring developer to access GemFire.</span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">Among other things, the integration provides:</span></div><span class="Apple-style-span" style="font-family: inherit;"></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><ol><li>Simpler ways to configure a cache and data regions. You can then inject the region into your app POJOs just like any spring dependency.</li>
<li>Allow the Spring developer to use Spring transaction model - declarative configuration and doing transactions consistently across a variety of providers. So, basically, the application doesn't have to explicitly invoke the GFE transaction APIs. Of course, our txn caveats do apply.</li>
<li>Wiring dependencies for callbacks easily: So, for instance, if you are using a CacheWriter or loader that needs DB URL, connection properties, etc you can now use the conventional Spring way of configuring DataSources and can be auto injected into your callback.</li>
<li>ETC</li>
</ol></span><br />
<div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">I encourage you to read </span><a href="http://blog.springsource.com/2010/08/03/spring-gemfire-1-0-0-m1-released-for-java-and-net/" rel="nofollow" style="color: #ff6600;"><span class="Apple-style-span" style="font-family: inherit;">Costin Leau's blog </span></a><span class="Apple-style-span" style="font-family: inherit;">for specific details, download and give it a try. Your feedback will be very valuable and much appreciated.</span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">I hope to amend this blog post with further details on the various other "scale out" features in 6.5 soon. </span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">If there is enough interest, go through our community site on 6.5, download and try out the new tutorial. </span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://community.gemstone.com/display/gemfire/GemFire+Enterprise"><span class="Apple-style-span" style="font-family: inherit;">http://community.gemstone.com/display/gemfire/GemFire+Enterprise</span></a></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://community.gemstone.com/display/gemfire/Getting+Started"><span class="Apple-style-span" style="font-family: inherit;">http://community.gemstone.com/display/gemfire/Getting+Started</span></a></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">----</span></div><div style="color: black; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><span class="Apple-style-span" style="font-family: inherit;">Cheers!</span></div><div style="color: black; font-family: 'Gill Sans', Futura, sans-serif; font-size: 14px; font-weight: normal; line-height: 1.4em; margin-bottom: 10px; margin-left: 15px; margin-right: 1px; margin-top: 1px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><br />
</div></div>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com2tag:blogger.com,1999:blog-31429685.post-4702669858177993922010-05-06T13:20:00.000-07:002010-05-06T23:01:43.692-07:00SpringSource/VMWare acquires GemStoneHere is some specific details on the potential technology synergies ....<br />
<br />
<span style="font-size: large;">Synergy with SpringSource</span><br />
<span style="font-size: large;"></span><br />
<br />
What makes the integration very synergistic and complementary are the areas of focus for the two companies : GemStone/GemFire has primarily focused on getting the infrastructure for clustering and managing data in the middle tier to be extremely reliable and offer tremendous flexibility; SpringSource focus has been to offer a ubiquitous programming model (through the Spring framework) that is simple to adopt and quite open where a myriad of products and technologies can be seamlessly integrated and a light weight runtime environment through the enterprise class TC server and a management environment enabled by HypericHQ. In a sense, the GemFire integration brings first class data management and clustering to the run time - a key component enabling "extreme scale" application deployment. <br />
<br />
<br />
GemStone, as well as the merged entity is totally committed to support heterogeneous access to the GemFire data fabric with continued support for Java, C++ and .NET. In fact, we are exploring opportunities to integrate with Spring.NET and continue on our commitments to simplify the programming models for non-Java environments. <br />
<br />
GemStone management and the SpringSource/VMWare management remains committed to make sure we deliver on the roadmap and future extensions to the platform that have already been discussed with customers. In fact, we will go well beyond our current commitments by leveraging the Spring framework and integration with a multitude of Spring modules to provide a much simpler configuration and development model for our customers.<br />
<br />
<br />
<span style="font-size: large;">How might we leverage the Spring framework?</span><br />
<br />
The spring framework is all about choice for enterprise Java applications. We now enable more choice by offering a clustered data management solution that can be used in multiple ways:<br />
<br />
<br />
1) <strong>As a transparent L2 Cache</strong>: Spring application using Hibernate will be able to plugin a L2 Cache that can intercept traffic to/from a backend database for scaling and performance reasons. Note that GemFire already supports Hibernate based L2 Cache plugins but now this effort will take increased focus. On the .NET side of things, we will accelerate support for Spring.NET applications using nHibernate.<br />
<br />
2) <strong>AOP cache</strong>: We will be able to offer sophisticated AOP caching interceptors that can transparently use a highly scalable cache. <br />
<br />
3) <strong>Parallel Data aware method invocations</strong>: Spring bean service invocations could transparently make use of the GemFire data-aware parallel function execution capabilities where behavior execution can happen in parallel, operate on localized data sets and go through a "aggregation" phase to produce a final result. <br />
<br />
4) <strong>Session state management</strong>: We already provide an abstraction layer for session state management on top of GemFire today. The intent would be to include this capability as an integral part of the platform. The highlights of this add-on include the ability to handle very large object graphs efficiently, dynamic partitioning and load balancing, visibility of session state across multiple clusters (WAN), HA and the ability to maintain a "edge" cache that maximizes memory utilization through a heap LRU based eviction algorithm. <br />
<br />
We are exploring options to enable quick integration of the data fabric into existing Spring applications through enhancements to the Eclipse plugins - for instance, the configuration of the cache regions and deployment topologies will be simplified. We will also investigate how we can leverage Spring Roo (next generation rapid application development tool) where a developer will be able to build a Java application with integrated caching in minutes. <br />
<br />
<span style="font-size: large;">Integration with Spring Modules</span> <br />
<br />
Unlike some of the other distributed caching options available in the market, GemFire natively supports memory based, reliable event notifications. Applications can subscribe to data in the fabric through "continuous queries" and can receive in-order, reliable notifications when the data of interest changes. Spring Integration extends the Spring framework into the messaging domain and provides a higher level of abstraction so business components are further isolated from the infrastructure and developers are relieved of complex integration responsibilities. We are exploring techniques to leverage this simple and intuitive API so applications are abstracted away from dealing with GemFire specific notification APIs. <br />
<br />
<br />
Spring Batch enables extremely high-volume and high performance batch jobs though optimization and partitioning techniques. Simple as well as complex, high-volume batch jobs can leverage the spring framework in a highly scalable manner to process significant volumes of information. Integration with the GemFire function service will now mean developers can develop Spring batch applications but behind the scenes leverage the parallelism and "data aware" routing capabilities built into GemFire. <br />
<br />
There are many integration possibilities all of which will make the job of the developer integrating with a data fabric/grid significantly simpler. <br />
<br />
<br />
<span style="font-size: large;">GemFire in the cloud</span><br />
<br />
Our offering will expand the VMWare strategy to deliver 'Platform as a Service' solutions over time. The already powerful arsenal includes VMWare VSphere as the cloud operating system, the Spring framework as the enterprise application development model and SpringSource TCServer as a lightweight application platform. Now, GemStone extends the capabilities with GemStone's GemFire as the scalable, elastic data platform. <br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/_o_zMfh3Veig/S-MetBiIV5I/AAAAAAAACVs/--tAjC48v_g/s1600/SpringSourceGemFire.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/_o_zMfh3Veig/S-MetBiIV5I/AAAAAAAACVs/--tAjC48v_g/s320/SpringSourceGemFire.gif" tt="true" /></a></div><br />
<br />
While a good case is being offered by VMWare on the current VMWare products positioned for the cloud, I will focus on the rationale for GemFire as a data platform for the cloud. <br />
<br />
One of the big advantages with applications deployed in a cloud environment is to tap into additional capacity 'just in time' based on demand changes. The underlying platform needs to detect and respond by exhibiting elastic characteristics. The additional capacity could be provisioned through virtual or physical machines spread across subnets or perhaps even across data centers. This means application behavior as well the associate state (data) needs to horizontal scale through migration. The behavior migration is well taken care of by modern day clustered application servers, but it is lot more challenging with data. The traditional relational database is rooted in a design where the focus is optimizing disk IO and maintaining ACID properties. The frame of reference was quite different - data was centralized and extensive locking/latching techniques used to preserve the ACID transaction properties. The design did not account for data to be managed across a large cluster of heterogeneous machines. Several clustered databases today offer a shared nothing architecture at the database engine layer but fall short at the storage layer (where it is shared everything). For elasticity, we believe the design has to change along the following two important dimensions: <br />
<br />
1) <strong>distribution orientation </strong>- efficient distribution methods to move data around a large network efficiently without loss of consistency <br />
<br />
2) <strong>memory orientation </strong>- when demand spikes it is lot faster and easier to move data between memory segments across machines. GemFire manages data by horizontally partitioning the data across any number of machines, and initiates automatic data rebalancing when additional capacity is added to its cluster. Unlike common database architectures, GemFire primarily manages data in memory and leaves the persistence to disk as a choice for the developer. Besides being more efficient to simply manage ephemeral state such as session state only in memory, regulations in some environments may mandate that nothing be stored on disk. <br />
<br />
We also know that applications deployed on the cloud will be subject to higher levels of SLAs - continuous availability and very predictable response times. Data stored in GemFire is typically always synchronously copied to one or more machines often also redundantly stored on disk and even asynchronously copied across data centers in case the entire data center goes down. Again, by allowing the data to be primarily managed in memory, the cost of maintaining data redundantly is considerably reduced. GemFire's built in mechanism to carry out continuous instrumentation and potential integration with cloud provisioning environments (vCloud API) permits apriori detection of changing load patterns to initiate data rebalancing without any operator intervention. <br />
<br />
Highly parallelizable, computationally intensive applications such as large scale web applications, or analytic applications (such as risk analysis in finance) are ideal candidates for cloud deployments. A lot of these computational intensive tasks tend to be data intensive also. Spring enabled services can now be parallelized through the use of GemFire's data-aware parallel function service where the underlying application behavior can now be parallelized and executed on the nodes carrying the data sets required by each parallel activity. This data localization will dramatically increase the throughput and reduce costs with little out-of-process data access. Instead of developing applications using the custom GemFire APIs, application development could rely on the open spring community driven modules such as Spring Batch simplifying the development task. <br />
<br />
We also believe the combination of SpringSource/VMWare infrastructure and GemFire will enable customers to implement "cloud bursting" strategies where on-premises stateful services can much more easily expand to a in-house private or even a public cloud. <br />
<br />
<br />
<span style="font-size: large;">Vision of a first class middle tier data management platform</span><br />
<br />
GemFire started off as a pure main memory distributed cache where transactional updates were immediately persisted to the database of record. In recent years, with the addition of capabilities such as reliable asynchronous writes to the database, data partitioning with dynamic rebalancing and wide area network data management the product now is often used as the high performance store with data written to the database either in batches, at the end of the business day or a batch run. The database is being relegated as the archival store for integration purposes. <br />
<br />
We are now adding support for parallel shared nothing disk storage layer in the product. Unlike other cluster database management system designs where the disks are shared across many distributed processes, each member of the cluster can persist in parallel to their local disks. Transactions do not result in disk flushes but rather allow the IO scheduler to time the block writes to disk dramatically increased disk write throughput. Any data loss risks due to sudden failures at a hardware level are mitigated by having multiple nodes write in parallel to disk. In fact, it is assumed that hardware will fail, especially in large clusters and data centers and software needs to take into account such failures. The system is designed to recover in parallel from disk and to guarantee data consistency when data copies on disk don't agree with each other.<br />
<br />
It is our strong belief that with the proliferation of highly distributed middle tier architectures, a data tier that is colocated with the application tier, is primarily clustered and is used to manage ephemeral data will substantially change the architectures for all enterprise applications that demand scaling and predictability.<br />
<br />
It is with this belief that we built the new <a href="http://community.gemstone.com/display/sqlfabric/SQLFabric">SQLFabric product </a>- same underpinnings as GemFire but uses SQL as the interface. We took this step in spite of the recent push by the "no sql" database alternatives for the cloud. Our decades of database implementation and research experience does not implicate SQL as a query language when it comes to the desired characteristics of elasticity and performance. This is simply a matter of design choices made by SQL database vendors based on a frame of reference that is no longer valid. In other words, there is nothing wrong with SQL as a language but rather it is the design of disk oriented, centralized databases that is sub-optimal. The premise behind SQLFabric is to capitalize on the power of SQL as an expressive, flexible, very well understood query language, but alter the design underpinnings in common databases for scalability and high performance. By providing ubiquitous interfaces like JDBC and ADO.NET the product becomes significantly easier to adopt and integrates well into existing eco-systems.Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com7tag:blogger.com,1999:blog-31429685.post-71001021488678535262009-11-22T10:08:00.000-08:002009-11-22T10:44:19.651-08:00New Era for OLTP databasesComing out of hibernation to blog. Made exciting with our new launch of the <a href="http://community.gemstone.com/display/sqlfabric/SQLFabric">SQL data fabric</a>. No wonder, when you start talking about a "first class" SQL interface to our distributed data grid/fabric platform it is compared with the RDB and modern day clustering extensions to the relational database.<br />
<br />
Here is a zooming presentation that attempts to position the product as a horizontal partitioned memory oriented database (unlike the typical parallel DBMS that is vertically partitioned for OLAP class applications). I think you will find the use of <a href="http://prezi.com/">prezi</a> fascinating. Would love to get some feedback on the presentation. All this is still very preliminary and we will be sharing a lot more details in the days ahead.<br />
<br />
(Click MORE --> full screen to get a better view) <br />
<br />
<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" height="400" id="prezi_gkuneyar15kv" name="prezi_gkuneyar15kv" width="550"> <param name="movie" value="http://prezi.com/bin/preziloader.swf"><param name="allowfullscreen" value="true"><param name="allowscriptaccess" value="always"><param name="bgcolor" value="#ffffff"><param name="flashvars" value="prezi_id=gkuneyar15kv&lock_to_path=1&color=ffffff&autoplay=no"><embed id="preziEmbed_gkuneyar15kv" name="preziEmbed_gkuneyar15kv" src="http://prezi.com/bin/preziloader.swf" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="550" height="400" bgcolor="#ffffff" flashvars="prezi_id=gkuneyar15kv&lock_to_path=1&color=ffffff&autoplay=no"></embed> </object><br />
<br />
here is a <a href="http://community.gemstone.com/display/community/2009/10/28/Entering+a+New+Architectural+Era+-+The+Elastic%2C+SQL%2C+OLTP+Database">related webcast from our principal architect, Dave Brown</a><br />
<br />
<br />
<br />
<input id="gwProxy" type="hidden" /><input id="jsProxy" onclick="jsCall();" type="hidden" /><br />
<div id="refHTML"></div><input id="gwProxy" type="hidden" /><!--Session data--><input id="jsProxy" onclick="jsCall();" type="hidden" /><div id="refHTML"></div>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-36229866158511302952009-06-11T14:12:00.000-07:002009-06-11T14:23:26.592-07:00Challenges in throughput scaling in a partitioned designHere is an interesting article from <a href="http://highscalability.com/dealing-multi-partition-transactions-distributed-kv-solution">Billy Newport posted on highscalability.com<br /><br /></a>The gist of this article is a point that hits home for me - <span style="font-style: italic;">you cannot assume higher scaling (especially throughput) just by adding more partitions for managing your data.</span><br /><br />The article correctly points out that one slow partition can throttle the system throughput for scatter-gather type queries that are parallelized on each partition. But, this assumes that all nodes are equal (or remain equal) from a load standpoint. I think, this ability to detect and handle "hotspots" is something that deserves a lot more attention than it gets. I suspect, there is this general myth that increase in the number of partitions will always result in higher throughput.<br /><br />For instance, we at GemStone have seen the following interesting cases (+ more):<br /><span style="font-weight: bold;">(1) 80-20 rule: </span>Often, it is the 20% of the data that is lot more popular at any given time. So, you have a situation where too many clients are converging onto a few partitions creating imbalance. A whole bunch of your partitions are just idle and heavily underutilized. You can mitigate this to some extent by creating more replicas for the "hot" data, but, it can be difficult to predict when you need the extra copy and for how long.<br /><span style="font-weight: bold;"><br />(2) GC pauses: </span>Consider a case where a large number of clients are updating partitioned data deployed in a JVM based data grid/fabric. Sooner or later, you are exposed to the dreaded "full gc" cycle causing the incoming client to be paused. Say, your normal update request takes fraction of a millisecond and let us also assume that the client requests are uniformly balanced across all the partitions. Essentially, it is more or less guaranteed that all clients will vector to each partition atleast once every second. Now, a full GC in any one partition causes every single client to be now pause. Have a GC that takes a whole minute, and you have all clients now waiting. Worse, the moment this paused partition gets out of it, the next partition gets into the same situation and so on.<br /><br />So, what do you do?<br /><br />We at <a href="https://community.gemstone.com/display/gemfire/GemFire+Enterprise">GemStone Systems</a> think some of the problems can be mitigated by providing two key capabilities:<br /><span style="font-weight: bold;">1) dynamic rebalancing: </span>This is the ability for the system as a whole to adjust itself by relocating buckets (subset of data within a partition) to less loaded nodes. And, doing this in such a fashion so that there are no pauses are introduced.<br /><span style="font-weight: bold;"><br />2) Enough instrumentation within the system to proactively avoid hotspots: </span>GemFire captures a wide range of statistics on the query, update rate on any given partition, the average response times, CPU utilization, GC pause times, overall heap utilization with respect to a configurable threshold to reduce the probability of full GCs, etc. Applications get access to these statistics through a simple API and can then use a API to trigger rebalancing to offload some of the data from "hot" partitions to less loaded ones. Of course, there is no guarantee that the past throughput characterictics observed on any given partition to be representative of te future but the explicit control provides a way for the application developers and adminstrators to dictate what happens. Our sense is that they know best. We even provide a way for the application to simulate a rebalance to see the effect it will have rather than actually doing it.<br /><br />Explore more on <a href="https://community.gemstone.com/display/gemfire/GemFire+Enterprise">GemFire data fabric</a>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-67456413369196587582007-07-19T23:21:00.000-07:002007-07-24T23:54:02.402-07:00High performance data sharing between C#, Java and C++<span style="font-size:130%;"><strong>SOAP and High Performance: an oxymoron? </strong></span><div><div><div><br /><p>XML messaging using the SOAP protocol has become the lingua franca for interoperability. Though loose coupling and simplicity through a standard text based protocol has its appeal, it is no secret that SOAP isn't suitable for high performance messaging. The simplicity and extensibility of XML/SOAP has generated great deal of interest and resulted in pervasive support across many languages and scripting environments.<br /><br />This <a href="http://www2003.org/cdrom/papers/alternate/P872/p872-kohlhoff.html">paper </a>evaluates the performance of SOAP for real-time trading and compares it to native binary protocol such as <a href="http://www.fixprotocol.org/what-is-fix.shtml">FIX </a>(Financial Information eXchange).<br /><br />Here is a simple example comparing two text based protocols, one that uses simple 'tag=value' pairs and the other a SOAP envelope. The price you pay for XML and SOAP is obvious.<br /><br /><span style="color:#336666;">A FIX message<br /></span><span style="font-family:courier new;font-size:85%;"> 8=FIX.4.3 9=00000098 35=X 49=ABC 56=XYZ 34=1 </span><br /><span style="font-family:courier new;font-size:85%;"> 52=20021116-10:15:28 262=MYREQ 268=1 279=1 </span><br /><span style="font-family:courier new;font-size:85%;"> 278=FOO.last270=13.42 271=1200 10=185</span><br /><br /><span style="color:#336666;">Equivalent message using FIXML (FIX Markup Language) </span><span style="color:#336666;"><br /></span><span style="font-size:85%;"><span style="font-family:courier new;"></span></span></p><br /><a href="http://bp0.blogger.com/_o_zMfh3Veig/Rqbxu0pttwI/AAAAAAAAAAk/EjRJWkuqxbQ/s1600-h/XMLexample.gif"><img id="BLOGGER_PHOTO_ID_5091022215498807042" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 586px; CURSOR: hand; HEIGHT: 452px" height="280" alt="" src="http://bp0.blogger.com/_o_zMfh3Veig/Rqbxu0pttwI/AAAAAAAAAAk/EjRJWkuqxbQ/s320/XMLexample.gif" width="509" border="0" /></a><br /><p><br /><br /><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><em><strong></strong></em> </p><p><strong><em></em></strong> </p><p><strong><em></em></strong> </p><p><strong><em></em></strong> </p><p><em><strong>The benchmark conducted summaries that SOAP messages are 3.5-4.5 larger than FIX, latency is 2-3 times worse, and encoding/decoding costs are increased by up to nearly 9 times.<br /></strong></em><br />Web services based on <a href="http://developers.sun.com/docs/web/swdp/r2/tutorial/doc/p34.html">REST </a>offers an efficient alternative to SOAP based web services. It is a simple HTTP based protocol that allows access to resources through CRUD (Create, Read, Update and Delete) operations. However, this is mostly geared towards simple request-response type synchronous messaging between services. Products that offer reliable, asynchronous messaging semantics on top of REST could be something to watch out for.<br /><br /><br /><br /><span style="font-size:130%;"><strong>Traditional messaging might be popular, but it has its limitations<br /></strong></span><br />High performance applications commonly use message oriented middleware products for asynchronous sharing of events and data across heterogeneous applications such as IBM MQ or Tibco Rendezvous.<br /><br />However, messaging solutions, as the name implies, package, move and deliver messages, one at a time. The receiving applications often need sufficient context to act on the incoming message. Typically the publisher provides this contextual information through message headers increasing the overhead for each message sent. So, for instance, an incoming Order may have to contain sufficient information to identify the associated customer. With the related contextual data arriving in encoded form, such as the Customer ID, the application can only process the Order after it can fetch all the related data (the entire customer record with credit information, for instance). This might require a round trip to the database. Therefore, if all the related data required for processing an incoming message requires external data source access, the processing speed can only go as fast as the weakest link - the database in most circumstances.<br /><br />Application environments with a high sustained message rate are much more exposed to this "weakest link" problem because enqueing in the messaging system doesn't really help. At some point, either messages have to be discarded or the publisher rate throttled.<br /><br />To support heterogeneous applications most messaging solutions require the application to <a style="mso-comment-reference: at_1; mso-comment-date: 20070724T1435">construct text based self-describing messages</a>. It becomes the responsibility of the application to use an appropriate encoding format for data such that it can be decoded by the receiving application. Should you use XML for the data format you will experience performance problems similar to those outlined above.<br /><br />In addition, a highly concurrent application environment with multiple publishers will have an increased probability for race conditions causing data integrity issues. Messaging systems have no inherent capabilities to deal with relationships between messages or ordering of messages across multiple messaging destinations (queues or topics). So, for instance, if Orders (parent) and corresponding LineItems (children) are delivered on two different queues, it is possible for the child data item to be delivered before the parent. Applications would normally be aware of these constraints and route related messages on the same messaging endpoint. It would be advantageous though, if the infrastructure used for transferring messages was aware of this relationship.<br /><br />Most messaging solutions are really designed to support diverse platforms, multiple language bindings, multiple protocols, flexibility in terms of message reliability, and more. But when it comes to object oriented applications such as C++, Java or .NET applications that want to share data, it can be cumbersome constructing or interpreting message headers, encoding/decoding to/from text based payloads, configuring message delivery options, correlating messages, looking up related data from backend databases before taking action, etc.<br /><br />Take this MQ example:<br /><br /><span style="font-size:85%;"><span style="font-family:courier new;">qMgr = new MQQueueManager(<arguments>);<br /><br />/***********************************************************/<br />/* Open the queue, build the message, and put the message. */<br />/***********************************************************/<br />int openOptions = MQC.MQOO_OUTPUT;<br />MQQueue myQueue = qMgr.accessQueue(args[0], openOptions,<br />null, null, null);<br /><br />MQMessage myMessage = new MQMessage();<br />myMessage.writeString(<>);<br />myMessage.format = MQC.MQFMT_STRING;<br /><br />MQPutMessageOptions pmo = new MQPutMessageOptions();<br />myQueue.put(myMessage, pmo);<br /><br />/**********************************************************/<br />/* Close the queue and disconnect from the queue manager. */<br />/**********************************************************/<br />myQueue.close();<br />qMgr.disconnect();</span><br /></span><br /><em>The application developer has to create a QueueManager, fetch queues, define the options to write into the queue, encode their application objects into some text based format, construct Message objects, configure message format, publish into the queue, close the queue and disconnect from the QueueManager. Quite cumbersome, right?<br /></em><a name="_msocom_1"></a><br /><br /><span style="font-size:130%;"><strong>Introducing a Data Fabric<br /></strong></span><br /><strong><em>Shared objects/events across heterogeneous applications through main memory caching<br /></em></strong><br />The basic idea is as follows:<br /><br />Instead of requiring application developers to think about messages being the mechanism to share information, why not let them use the same paradigm they use when communicating between various components within the application; simply share domain objects in common data structures like a Map and use native thread based notification services. Extend this concept such that these data structures are distributed and visible to disparate applications. Essentially, applications share data and events with each other through a shared object database that is distributed in nature. Applications perform CRUD operations on a database and receive notifications when changes to data they are interested in occur.<br /><br />There are some important characteristics that differentiate this database compared to a regular disk-based relational database:<br /><br />1). It is distributed in nature - applications publish data objects and the database copies/moves these to multiple nodes. Sounds like database replication, except the location of data, the number of copies made, how the data is transported are aspects that are different<br />2). It is primarily memory based and hence fast<br />3). It is active in nature and pays attention to complex expressions of interest from subscribing applications and when they change, instantaneously pushing the change to the application.<br /><br />This kind of data management system is referred to as a data fabric or data grid. A Google search on “Enterprise Data Fabric” will provide an idea of various vendor offerings.<br /><br />A data fabric or data grid combines the important features and semantics seen in database technology and messaging for high performance applications.</p><p>A true data fabric includes the following comprehensive capabilities:<br /><br />Ø <strong><em>Designed to offer in-memory data access speeds</em></strong>: Data is managed in concurrent data structures (link to concurrent java stuff), primarily in memory with minimum contention issues.<br />Ø <strong><em>Flexibility in data storage</em></strong>: Data can be stored locally in process, replicated to multiple nodes, partitioned across multiple nodes, maintained both in-memory and in-disk or simply fetched lazily from a back-end data store. Where the data is located or how many copies are maintained becomes a configuration issue and is based on the requirements around performance, high availability and volume of data being managed, yet completely abstracted away from the developer. Data locality is virtualized in a data fabric.<br /><br />Ø <strong><em>Simple development model</em></strong>: A data fabric offers a simple Map like interface to applications. Applications can simply fetch by key or put domain objects directly into the cache. There is no need to worry about headers, encoding, decoding to some intermediate format, etc.<br /><br />Ø <strong><em>Scalable</em></strong>: By distributing the data the data fabric uses resources across multiple nodes. Deployments can simply add additional capacity on the fly and automatically get the data rebalanced and handle increasing load (concurrent activity or data volume).<br /><br />Ø <strong><em>Transactional</em></strong>: The fabric can automatically participate in any ongoing transactions and ensure consistency of data across all the applications sharing the data. For instance, if two applications concurrently decide to create the same customer order and publish this to others, the conflict will be detected and handled appropriately. This is a big difference compared to messaging - with no notion of identity that can be associated with data inherent within the messaging system, it is non trivial to detect conflicts when two applications decide to make the same change at the same time.<br /><br />Ø <strong><em>Reliable Publish-Subscribe semantics</em></strong>: Applications perform CRUD operations on a local cache and the corresponding event is routed to nodes that subscribe to the data. Data objects can either be synchronously or asynchronously pushed to subscribing applications. Events are pushed to subscribers that contain the new, changed or deleted object(s). The data fabric is intelligent enough to only propagate changes to data objects or its relationships, keeping the underlying network traffic to a minimum.<br /><br />Ø <strong><em>Querying</em></strong>: Similar to a regular database, the data in-memory can be indexed and queried using SQL like syntax.<br /><br />Ø <a href="http://java.sys-con.com/read/260054.htm"><strong><em>Continuous querying</em></strong></a>: Applications register complex queries, which are queries with complex predicates, joins, etc and, unlike a regular query, are not just executed once. They become resident in the database and give the impression that the query is continuously running. As the data changes the continuous query engine calculates how the result set has changed, pushes the "delta" to the application and merges this with a cache result set on the application node.<br /><br />Ø <strong><em>Heterogeneous language support</em></strong>: Application objects are automatically serialized in a neutral format within the fabric such that the same object can be de-serialized into an instance of a class in another application written using a different language (with a similar class structure).<br /><br /><a href="http://bp3.blogger.com/_o_zMfh3Veig/RqbwskpttuI/AAAAAAAAAAU/hvLARJwHfmI/s1600-h/GemFire.gif"><img id="BLOGGER_PHOTO_ID_5091021077332473570" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 479px; CURSOR: hand; HEIGHT: 276px" height="169" alt="" src="http://bp3.blogger.com/_o_zMfh3Veig/RqbwskpttuI/AAAAAAAAAAU/hvLARJwHfmI/s320/GemFire.gif" width="371" border="0" /></a><br /><br /><br /></p><p><picture></p><br /><br /><br /><br /><br /><p>For additional information, read more on the <a href="http://www.gemstone.com/pdf/GemFire_Architecture.pdf">architecture of the GemFire Data Fabric</a><br /><br /></p><br /><br /><p><br /><span style="font-size:130%;"><strong></strong></span></p><br /><p><span style="font-size:130%;"><strong></strong></span></p><p><span style="font-size:130%;"><strong>Understanding the data fabric through a use case<br /></strong></span><br />Let us look at a financial Trade order processing system that can route orders to an exchange offering the best liquidity.<br /><br />Here is how an order flows through the system:<br /><br />- Orders arrive as FIX messages from trading partners and clients to an Order processing application<br />- The incoming FIX message is validated (authentication and authorization), normalized (combined with other related information such as Trader information, etc) and written into the data fabric as an Order object<br />- If the incoming order rate is very high, the validation and normalization process itself can be load balanced across multiple nodes<br />- Orders are then routed to a Trading Strategy engine that uses different algorithms to determine the best time, quantity and exchange to route the order. Execution of the algorithms requires current market data from multiple exchanges (these are the prices for the different securities traded in the exchanges) and also uses various risk metrics in its calculations. The incoming market data is being pushed typically at a very high rate (20,000 or more ticks per second)<br />- Orders are batched and then routed to an Exchange router, the application responsible for reliably getting the trade order executed on a market exchange<br />- Finally, there is the entire process of post trade information exchange that we will ignore as it is not relevant for this discussion.<br /><br /><br />Let us assume that the order processing application and the exchange router application are written in Java and the computation intensive strategy engine is written in C++. These three applications are sharing data and events in real-time.<br /><br />While it is beyond the scope of this article to explain all the aspects of the data fabric, we will focus on how the fabric enables objects and event sharing.<br /><br /><br /><strong><em>Shared Object model</em></strong>: With the data fabric, the domain model classes are designed such that they are more or less equivalent for each language in use. One might use an Object modeling tool and generate these classes for both Java and C++.<br /><br /><strong><em>Serialization framework</em></strong>: Developers implement callbacks similar to Java Externalizable where object fields are written to a stream managed by the serialization framework provided by the data fabric. The framework serializes data into a language neutral wire format that can be consumed by any application that uses the serialization framework at the consuming end.<br /><br />In this case, the order processing application knows that an order is a complex object comprised of several fields and sub fields. Using the serialization framework, the application writes out the relevant fields out to the stream and puts it out on the wire.<br /><br />The framework handles translation of primitives across languages and across processor architectures, removing that burden from the end user application. It also preserves complex object relationships across the serialization boundary.<br />At the receiving end, the incoming bytes are identified as an order object (because the type Id for the payload would be the same across all languages).<br />Once the object type is identified, the rest of the payload is streamed into the order object on the receiving end (let's say that this is the strategy engine)<br /><br />Another advantage of this mechanism is that the de-serialization of the payload from the wire (the other half of the serialization framework) results in a ready to use object in the language of the consuming application, which receives a notification about the change and can act on it.<br /><br />For our strategy engine, the data fabric then fires a notification and hands off the order object (now represented in C++) to the application which then acts on it.<br /><br />Much of this work can be automated using tools that make it almost trivial to define a data model that is inherently faster and more efficient than traditional serialization mechanisms using XML marshalling, Java or C# serialization.<br /><br /><strong><em>“Delta” Propagation</em></strong>: When a new incoming FIX message is a “change request” to a pending order, the application merely fetches the Order from the fabric and applies the change to one or more fields – for example, the customer wants to change the “buy” or “sell” volume of the trade, which would be updating a single field in the object. Given that the fabric maintains identity for all objects in the distributed system much like a database, it is now able to calculate the exact “delta” and just transmit that to the strategy engine or any number of connected applications that are listening for the incoming Order requests. By dramatically cutting down on the network and serialization overhead, “Delta” propagation allows the system to scale much better and push much more data through the system than would otherwise be possible.<br /><br /><br /><a style="mso-comment-reference: at_1; mso-comment-date: 20070724T1512"><span style="font-size:130%;"><strong>Conclusion</strong></span></a><span style="font-size:130%;"><strong><br /></strong></span>XML based interoperability works well for a wide class of applications and very well suited to loosely couple applications, but, isn’t well suited for <a href="http://www.gartner.com/DisplayDocument?doc_cd=131036">eXtreme Transaction processing </a>applications. Messaging system lack enough context and hence prone to data integrity and consistency issues.<br /><br />Distributed main memory based architecture such as distributed caching or data fabric (grid) technologies that combines the functions and semantics of database technology and reliable messaging technology may be a better fit.<br /><br />Heterogeneous applications share a common object domain model - objects published or altered by one application is shared with other applications at memory speeds. The key to fast object sharing is the use of domain classes across these heterogeneous applications that are more or less similar, a native serialization protocol that can detect and dispatch object change "deltas" and an optimized neutral object wire format.<br /><br /><a name="_msocom_1"></a>--------</p><p><br /> </p></div></div></div>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-19448142159357130732007-04-09T22:15:00.000-07:002007-04-09T22:23:27.760-07:00Reliable pub Sub is an integral part of a Data FabricCan the data fabric (or a distributed cache with strong reliability semantics in message distribution) be used as a replacement for traditional messaging? This is a question we frequently get asked and this has been a sweet spot for us at GemStone. Here is some rant on the rationale:<br /><br /><p>Instead of a distributed main memory based data management solution that merely manages key-value pairs, imagine a solution that can manage objects along with relationships. In other words, ensures data integrity through knowledge about the entire data model. Ahh! like a database.<br />Now, combine this attribute with the ability to distribute data (replicate, partition, whatever) to many nodes, reliably and provide notifications to subsribing applications. What you get is a ACTIVE, distributed data management system - a system that inherently provides reliable pub-sub along with key semantics of the traditional data management system. Applications simply update a database (objects and relationships), express interest through complex query expressions on the data model and get delivered notifications based on their interest.<br /></p><p>In a traditional messaging solution:<br /></p><ol><li>Publishing application has to explicitly construct messages, add header information for message identification and to allow subscribers to make sense of the message add enough contexual information. Often, this contextual information takes the form of identifiers (keys that point to the real data in some database).</li><li>Most messaging solutions use hub-spoke mechanisms to queue, relay messages to subscribers. i.e. messages hop from application process to some server process managed by the messaging provider and then to the receiving application. Look at many practical applications and you will find that the cost of messaging is not necessarily in the network, but, rather in the CPU costs associated with (de)serialization and all the copying of byte[] that goes on each process space and in the kernel layers.</li><li>Receiving application again has to do the reverse - parse headers, deserialize message body and then make sense of the message. To make any decision, often the application has to look up related data from other enterprise repositories such as a relational database, often slowing down the rate at which the entire flow can operate to the slowest link - often the relational database query.<br /></li></ol>Now, with a data fabric, all applications are sharing a single data model and express interest on the data model through simple, intuitive queries. The underlying fabric is constantly detecting what and how data and relationships are changing and simply sends notifications of the changes to the consuming application.<br /><br />Note the following advantages with a data fabric:<br /><br /><ol><li>Application processes are connected to each other in a p2p fashion with direct connections between each. This allows the fabric to avoid unnecessary network hops, dramatically reducing latency and CPU costs associated with message transfer.</li><li>As the data is typically held in multiple locations and often replicated to the process space of the consuming application, the publisher doesn't have to send obese messages - applications merely change the data fabric and underneath the hood the right "delta" event gets propagated to the consuming applications. </li><li>The receiving application when notified can take immediate business decisions as the contextual information they need is cached right there.</li></ol><p>More later ....</p><p> </p>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-1158621053835324692006-09-18T16:08:00.000-07:002006-09-18T17:53:57.543-07:00Continuous Querying Article on August JDJ issue<blockquote>Myself and gideon puttogether a rather simple use case to illustrae the power of the continuous querying technology and reached out to Java Developer Journal. They apparantly found this to be powerful enough - it became the front page feature articel for their August issue.<br /><br /><a href="http://java.sys-con.com/read/260054.htm">Java Feature — Building Real-Time Applications with Continuous Query Technology</a><br />— The client/server development model prevalent in the mid-1990's resulted in extremely easy-to-build rich GUI applications that interacted directly with a relational database. 4GL tools such as Visual Basic and PowerBuilder let even junior developers visually compose both the presentation and most of the backend data binding. While this made for impressive Rapid Application Development (RAD) productivity, the client/server architecture was severely challenged when dealing with real-time environments where the data changes rapidly and applications require visibility to the correct data at all times. As a result, client applications were forced to poll the database continuously to check for changes.<br />......<br /><br /></blockquote>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-1154405502819649802006-07-31T21:02:00.000-07:002006-07-31T21:53:53.526-07:00SOA n GRID synergistic?<p class="MsoNormal">A colleague of mine brought this SOA vs GRID <a href="http://www.it-analysis.com/business/innovation/content.php?cid=8638">aricle </a>to my attention .... ' provides some interesting data on this topic and debunks certain popularly held beliefs..'<br /><br /><br /><br />---------<br />Here is my take on the SOA-GRID synergy or lack thereof in practice followed by a discussion on where and if a distributed cache can fit in such architectures …..<br /><br />Clearly the whole business of service orientation is based on the fact that one discovers a service based on desired operation and other QoS considerations, before binding. Service clients are loosely coupled to service providers by definition.<br />In the case where service clients come and go, say, like with a portal service that aggregates information from 10 other services, each request could be routed through a intermediary that isolates client from directly connecting to the server. This could be the UDDI registry for instance.<br /><br />You agree with this, ya? Now, the question is, in what percentage of such SOA architectures will one use a dynamic provisioning service (loosely a GRID)? i.e. every time I use the discovery service it may point me to a different server to get the request fulfilled. In many cases, the expected load might be very predictable or the service provider has to run on specific platform that is my legacy and cannot be deployed into a utility computing center. Folks talk about high availability and hence the need for the Grid, but, I think, that is just bull. All SOA architectures will use a JEE or .Net façade and inherently be HA. So, basically, there is no case for a Grid in this situation.<br /><br />But, more you look into the future, more likely I want my services to deployed using low cost commodity hardware, be bound by contracts on the QoS (availability and performance), forcing me to think about how my services can constantly be monitored and dynamically reprovisioned on the fly. The power of the Grid - well, actually a sophisticated provisioning and virtualization engine would become a necessity. I am not talking about a typical Grid solution, aka a compute grid - a scheduling engine.<br /><br />In any case, like the who's who within the OGSA community would say, there is natural synergy between SOA and GRID. SOA is about how to connect services together to realize higher value (integration) and GRID is merely a deployment strategy for such services.<br /><br />I thought, <st1:stockticker>IBM</st1:stockticker> was already on track to deliver on this promise within their WebSphere platform, circumventing the need for DS or EGO. Ahh! Who am I kidding?<br />Checkout Oracle's fusion strategy and how automatic provisioning and virtualization is integrated into 10G. Quite impressive.<br /><br />Now, let me shift gear and see if and where a distributed caching solution fits in a SOA architecture .....<br /><br />Where do databases and messaging solutions fit in? Just SOA architectures? Only in Compute Grid apps? Duh! everywhere. And, that is my position.<br />OK, here is the hiccup - again, SOA, by definition is all about getting apps (what do call this now … yeah! Services) talking to one another in a loosely coupled fashion. The service provider can change the underlying technology and behaviour any way it want and remains isolated from other services. I just have to make sure my contract is still intact. So, this cache thing breaks this, one could argue. The cynic would say 'I want a ESB bus that uses XML messaging not a bloody cache'.<br />Well, gentlemen, I got news for you. I think, all this loosely coupling stuff is baloney. Well, it holds water to a point, but no more.<br /><br />The fact of the matter is, in real life, application models and yes, data models change and undergo significant enhancements. XML or no XML, your apps, if they need to talk to one another have to change.<br /><br />You want to solve real problems, create a highly scalable, performant SOA architecture - use a memory based data fabric - An ESB designed for data intensive environments. You still want to use XML - shove data into the Enterprise Service Fabric (<st1:stockticker>ESF</st1:stockticker>) as XML. You don't like using APIs, but rather prefer the slippery <st1:stockticker>SOAP</st1:stockticker> (JAXM) route, use the web services interface.<br /><br />And, now, fire storms have erupted in <st1:state><st1:place>california</st1:place></st1:state> ….</p>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-1153442554531878792006-07-20T17:37:00.000-07:002006-07-20T21:02:18.273-07:00Introducing Distributed Data Fabric for the middle tier<p class="MsoNormal" style=""><span style=";font-family:Verdana;font-size:100%;" >You should <a href="http://jagslog.blogspot.com/2006/07/one-size-fits-all-relational-database.html">read the prior post</a> first, tell me what you agree with and where you disagree .....<br /></span></p><p class="MsoNormal" style=""><span style=";font-family:Verdana;font-size:100%;" ><br /></span></p><p class="MsoNormal" style="font-weight: bold;"><span style=";font-family:Verdana;font-size:100%;" >What is the GemFire Enterprise Data Fabric?<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >GemFire Enterprise Data Fabric is a high performance, distributed operational data management infrastructure that sits between your clustered application processes and back-end data sources to provide very low latency, predictable, high throughput data sharing and event distribution.<o:p></o:p></span></p><p class="MsoNormal"><span class="Heading2Char" style="font-size:100%;"><span style="font-family:Verdana;"><br /></span></span></p><p class="MsoNormal"><span class="Heading2Char" style="font-size:100%;"><span style="font-family:Verdana;">It is about operational data management </span></span><span style=";font-family:Verdana;font-size:100%;" >– </span><span style=";font-family:Verdana;font-size:100%;" >Unlike a Data warehousing system where terabytes (or petabytes) of data is consolidated from multiple databases for offline data analysis, the EDF is a real-time data sharing facility specifically optimized for working with operational data needed by real-time applications – it is the “now” data, the fast moving data shared across many processes and applications. It is a layer of abstraction in the middle tier that collocates frequently used data with the application and works with backend databases behind the scenes.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-weight: bold;font-size:100%;" class="Heading2Char" ><span style="font-family:Verdana;">Distributed Data Caching </span></span><span style=";font-family:Verdana;font-size:100%;" >– </span><span style=";font-family:Verdana;font-size:100%;" >the most important characteristic of the GemFire Data Fabric is that it is fast – many times faster than the traditional disk based database management system, because it is primarily main-memory based. Its engine harnesses the memory and disk across many clustered machines for unprecedented data access rates and scalability. It utilizes highly concurrent main-memory data structures to avoid lock contention and a data distribution layer that avoids redundant message copying, native serialization and smart buffering to ensure messages move from node to node faster than what traditional messaging would provide. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >It does this without compromising the availability or consistency of data – a configurable policy dictate the number of redundant memory copies for various data types, storing data synchronously or asynchronously on disk and uses a variety of failure detection models built into the distribution system to ensure data correctness.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-weight: bold;font-size:100%;" class="Heading2Char" ><span style="font-family:Verdana;">Key Database semantics are retained</span></span><span style=";font-family:Verdana;font-size:100%;" ><span style="font-weight: bold;"> </span>– </span><span style=";font-family:Verdana;font-size:100%;" >simple distributed caching solutions provide caching of serialized objects – simple key-value pairs managed in Hashmaps that can be replicated to your cluster nodes. GemFire, provides support for multiple data models across multiple popular languages – data can be managed as Java or C++ objects natively, native XML documents or in SQL tables. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Similar to a Database management system, distributed data in GemFire can be managed in transactions, queried upon, persistently stored and recovered from disk. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Unlike a relational database management system, where all updates are persisted and transactional in nature (ACID), GemFire relaxes the constraints allowing applications to control when and for what kind of data you need total ACID (provide link) characteristics. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >For instance, a very high performance financial services application trying to get price updates distributed what is most important is the distribution latency – there is no need for transactional isolation.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >The end result is a data management system that spends fewer CPU cycles for managing data and offering higher performance.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <h2 style="margin-left: 0in; text-indent: 0in;"><span style=";font-family:Verdana;font-size:100%;" >Continuous Analytics<o:p></o:p></span></h2> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >With data in the fabric changing rapidly as it is updated by many processes and external data sources it is important for real-time applications to be notified when events of interest are being generated in the fabric. Something a messaging platform is quite suited to do. GemFire data fabric takes this to the next level – applications can now register complex patterns of interest, expressed through SQL queries; Queries that are continuously running. Unlike a database system where queries have to be executed on resident data, here data (or events) is continuously evaluated by a query engine that is aware of the interest expressed by hundreds of distributed client processes. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <h2 style="margin-left: 0in; text-indent: 0in;"><span style=";font-family:Verdana;font-size:100%;" >Reliable messaging and routing <o:p></o:p></span></h2> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >When using a messaging platform, application developers expect reliable and guaranteed Publish-Subscribe semantics. The system has knowledge about active or durable subscribers and provides different levels of message delivery guarantees to subscribers. GemFire EDF incorporates these messaging features on top of what looks like a database to the developer. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Unlike traditional messaging where applications have to deal with piecemeal messages, message construction, incorporating contextual information in messages, managing data consistency across publishers and subscribers, GemFire enables a more intuitive approach - one where applications simply deal with a data model (Object or SQL), subscribe to portions of the data model and publishers make updates to the business objects or relationships. Subscribers are simply notified on the changes to the underlying distributed database. <o:p></o:p></span></p> <div style="border-style: none none solid; padding: 0in 0in 1pt;"> <p class="MsoNormal" style="border: medium none ; padding: 0in;"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> </div> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <h1 style="margin-left: 0in; text-indent: 0in;"><span style=";font-family:Verdana;font-size:100%;" >What makes GemFire EDF unique ?<o:p></o:p></span></h1> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><o:p> </o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >For the last two decades or so, relational database management systems have taken a "kitchen sink" approach trying to solve any problem associated with data management by bundling this as part of the database engine.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Relational databases are centralized and passive in nature. It does a good job in managing data securely, correctly and persistently, but does not actively push the data to applications that might be interested, now. Second, databases are designed to optimize access to disk and to guarantee the transactional properties at all times. This limits the speed and scalability of a database engine in a highly distributed environment. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Compare this to a data environment where data storage structures are highly optimized for management in memory and concurrent access. To notify applications instantaneously, GemFire immediately routes data to the right node through a data distribution layer that is designed to reduce contention points and avoid unnecessary copies of messages before being transported.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Messaging solutions are most suited for very loosely coupled applications. Though this has its benefits, applications are left with the tough job of managing contextual information to make decisions, often requiring round trips to a database. This eliminates any performance advantages that applications can derive from messaging. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Besides this, often, the asynchronous nature of messages can also result in inconsistencies – the contextual information in the database may not reflect the correct state when the message is received.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >GemFire provides an operational data infrastructure that brings data and events into one distributed platform – applications can focus on what matters most – operate on business objects and relationships. Interested applications are immediately notified as and when the data model changes. Data is co-located and accessible at memory speeds and data correctness is always ensured.<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Modern day Event Driven Architectures require applications to react to events being pushed at very high rates from multiple streaming data sources, aggregate this data with other slow moving data managed in databases and distribute data and events to many application processes. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >Traditional centralized databases simply are not designed to handle this mounting onslaught – what you need is a distributed memory based architecture that can analyze the incoming stream data, combine this with related information and present a consistent and correct data model to the application. <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" >What makes GemFire unique, is this ability to not just analyze fast moving data, but the ability to present a data model (like a database) and route data/events to applications with guaranteed reliability (the semantics of reliable messaging).<o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" > <o:p></o:p></span></p> <p class="MsoNormal"><span style=";font-family:Verdana;font-size:100%;" ><span style="font-weight: bold;">Bottomline</span>: Time has come for a middle tier data management layer to manage your operational data and events to enable a new generation of real-time applications with QoS guarantees on performance, continuous availability and scalability. You want to be able to do this, while retaining your investments in existing databases.<o:p></o:p></span></p> <span style=";font-family:Verdana;font-size:100%;" ></span>Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-1153442110739620842006-07-20T17:27:00.000-07:002006-07-20T17:35:10.743-07:00"One size fits all" Relational Database is dead. What say u?OK, this is my first blog.<br />I will mostly blog on my professional life and it has a lot to do with high performance distributed computing. Especially, main-memory based distributed data management systems.<br /><br />Let me begin by taking a look at the traditional relational database.<br /><br />For the last two decades or so, major Database vendors have taken a "kitchen sink" approach trying to solve any problem associated with data management by bundling this as part of the database engine. Don't take my word on this. Here is how Jim gray's put this "We live in a time of extreme change, much of it precipitated by an avalanche of information that otherwise threatens to swallow us whole. Under the mounting onslaught, our traditional relational database constructs—always cumbersome at best—are now clearly at risk of collapsing altogether" Checkout this <a href="http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=293&page=1.">article </a>. Adam Bosworth also has some interesting comments in his blog "<a href="http://www.adambosworth.net/archives/000038.html">Where have all the good databases gone</a>"<br /><br />Alright! What specifically I am talking about?<br /><br />Consider this for starters:<br /><br />Take a portal - today you want to build scalable systems using a clustered architecture where you can keep adding processing nodes in the middle tier so you can linearly scale as the number of users keeps going up.<br /><br />Now, I don't have to tell you how important availability is for folks like these. Always up, always predictable in terms of performance and responsiveness. Enormous attention has been paid to make the middle tier highly available. But, alas, when it comes to the backend data sources, it is left upto the DB vendor.<br /><br />The traditional database is built to do one thing very well - do a darn a good job in managing data securely, correctly and persistently on DISK. It is centralized by design. Everything is ACID. Ensuring high availability when you are constrained by the strong consistency rules (everything to disk) is very tough to manage. Replicate a database so you can failover to the replica during failure conditions, you are all of sudden left with random inconsistencies to deal with. Try to replicate synchronously, you pay a huge price in terms of performance. You want to provide dramatic scalability, through many replicated databases, you better be ready to live with a very compromised data consistency model.<br /><br />Ahh! so, something like Oracle RAC is the answer, one might argue? Yes, for a number of use cases. But, here is what one has to consider:<br />1) You are still dealing with a disk centric architecture where all disks are shared and each DB process instance has equal access to all disks used to manage the tablespaces.<br />Here are a few important points worth noting:<br /><br /><ul><li> The unit of data movement in the shared buffer cache across the cluster happens to be a logical data block, which is typically a multiple of 8KB. The design is primarily optimized to make disk IO very efficient. AFAIK, even when requesting or updating a single row the complete block has to be transferred. Compare this to an alternative offering (a distributed main memory object management system) where the typical unit of data transfer is an object entry. In a update heavy and latency sensitive environment, the unit could actually be just the "delta" - the exact change.</li></ul><br /><ul><li>In RAC, If accessing a database block of any class does not locate a buffered copy in the local cache, a global cache operation is initiated. Before reading a block from disk, an attempt is made to find the block in the buffer cache of another instance. If the block is in another instance, a version of the block may be shipped. Again, there is no notion of a node being responsible for a block; there could be many copies of the block depending on how the query engine parallelized the data processing. This will be a problem if the update rate is high - imagine distributed locks on coarse grained data blocks in a cluster of 100 nodes; for every single update?</li></ul><ul><li> At the end of the day, RAC is designed for everything to be durable to disk and requires careful thought and planning around a high speed private interconnect for buffer cache management and a very efficient cluster file system </li></ul><br />Yes, there have been tremendous improvements to the relational database, but, it may not the answer for all our data management needs.<br /><br />Consider this - How many enterprise class apps are being built that just depend on one database. Amazon's CTO, werner wogel talks about how 100 different backend services are engaged to just construct a single page that you and me see. Talk to a investment bank, events streaming in at mind boggling speeds have to be acted on not by one process, but, by potentially 100's of processes, instantenously. Everything needs to talk to one another in real-time. Consistent, correct data has to be shared across many processes built using heterogenous languages, and connected to heterogenous data sources all in real-time. Ahh! what do call this SOA, ESB, etc.<br /><br />You can come up with a massive loosely coupled architecture for the connecting everything with everything, but, you have to pay extra attention to how you will manage data efficiently in this massively distributed environment.<br /><br />Isn't it time for a middle-tier high performance data management layer that can capitalize on the abundant main-memory available on cheap commodity hardware today ? - a piece of middleware technology that can draw data from a multitude of data sources, move data between different applications with tight control on correctness and availability allowing all real-time operations to occur with a high and predictable Quality of Service.<br /><br />Is this a distributed cache? A messaging solution?<br /><br />OK, let me sell you what we do ....Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0tag:blogger.com,1999:blog-31429685.post-1153441579939907772006-07-20T17:19:00.000-07:002007-04-09T22:29:27.854-07:00Finally, I am bloggingBoy! this blog thing is new to me. Honestly, I have never blogged nor have I closely watched any blog. This nearly middle aged boy is "old school". Well, about me .... I do a lot of architecture for <a href="http://www.gemstone.com">GemStone </a> on paper :-) as their Chief Architect (<a href="http://www.gemstone.com">www.gemstone.com</a>). But, to my credit, I have been doing distributed data management stuff for, well... too long. I guess, <a href="http://www.google.com/search?hl=en&q=jags+ramnarayan&btnG=Google+Search">googling me</a> might give you a better idea ...<br /><br />I guess, I am laggard when it comes to blogging. Better now, than never. It is all about the mindshare.<br /><br />Serious bloggers, any advise to share ?Jags Ramnarayanhttp://www.blogger.com/profile/01088990426487508645noreply@blogger.com0