Recently, I’ve noticed more and more people looking for ways to create massively scalable applications. It is no well known that the database is more often than not the bottleneck. This is why caching became ubiquitous these years with lots of products available on the market, both open source or not.
Unfortunately, there are cases where caching is not enough: either because even with exceptional cache efficiency (98% cache hits or more) or because your application isn’t read-mostly, ie: when there isn’t a ratio of at least 50 reads per 1 write. In the latter case, a write will invalidate the cache entry and will seriously reduce the cache efficiency.
In those cases (but there could be others), the only way to make your application scale is to make the database scale. Databases clusters like Oracle RAC or MySQL memcached will help you to some extent but sometimes even those won’t be sufficient and that goes without saying that this solution is incredibly expensive.
One of the alternative designs proposed is called database shard. The basic idea is to divide your data between multiple cheap servers, each one handling a small fraction of your data. This allows near-linear cheap scalability of your database cluster as you just have to make data separation a bit finer each time you want to add more servers. Here is a good article explaining the concept in more details. Big names like Google, Flikr and LiveJournal as using this technology as of today so this isn’t something to come but already present and working in production environments.
In the Java world, there are at least two good products to build shards: Hibernate using Hibernate Shards and OpenJPA using Slice. Both are JPA implementations that are going to split the data between all you database nodes with only a matter of configuration and maybe a bit of extra code.
One of the problem with shard that I noticed is often not openly discussed is that since the data is split across many databases, you can loose atomicity as your application might read or write to multiple data sources in a single transaction.
Well, that’s counting without that good ol’ technology invented more than 20 years ago: 2PC / XA and its related Java standard: JTA. Thanks to the two-phase commit protocol, the atomicity of the transactions can still be preserved.
Slice explicitly states that XA is required to maintain atomicity between all nodes while Hibernate does not explicitly document this but the developers openly agreed that XA also is a prerequisite to atomicity, see http://groups.google.com/group/hibernate-shards-dev/browse_thread/thread/a1bf86180f6ecf63 and http://opensource.atlassian.com/projects/hibernate/browse/HSHARDS-49. You should have no worries here: a good transaction manager properly plugged in Hibernate Shards or Slice will work fully transparently.
If you’re looking to cut costs this could also be a good alternative to expensive traditional clusters. The two shard stack I mentioned are open source and there are at least two open source databases with good XA support you can use for your backend: PostgreSQL and FirebirdSQL.
Isn’t it a bit ironic that a technology regarded as one of the worst database speed hogs allows for massive database scalability ?