I got some flack a few months back when I posted an article about poor scalability when using Hibernate for a proof of concept I was working on at the time. For that particular project it seems that Hibernate probably wasn't the best fit. However, I've now had the opportunity to use Hibernate on another project and my experience has been much better this time. First of all, the productivity benefits of using Hibernate rather than writing Spring JDBC code are quite impressive. I was able to get a prototype web application up and running in just a couple of hours with Hibernate because I really did only have to focus on the business logic rather than writing low level code. Scalability isn't at all critical for this application since it will not have a huge number of users but I ran some tests anyway to compare one of the transactions with a hand-coded JDBC equivalent. Here are the results:



It's true - Hibernate does add some overhead to the transaction but it's a price I'm happy to pay for the time saved in development. I expect there are other transactions in the system where Hibernate will be faster than JDBC due to caching.

I still stand by my previous conclusion, which is that it is always worth benchmarking products/technologies against real use cases to make sure they are suitable choices. One size does not fit all.

Labels: hibernate, performance, scalability

A while ago, I posted a couple of blog entries about the fact that I was going through the learning curve with Hibernate. Since then I have talked to some friends who have used Hibernate on large projects and I have also been prototyping an application using both Hibernate and Spring JDBC to understand the pros and cons of each approach.

My initial reason for learning Hibernate was to make sure I knew enough in an interview situation if it came up. I also needed to recommend a persistence technology for a couple of projects that I am involved in and having heard so much about Hibernate I assumed that it would be a good fit.

The biggest advantage that I can see with Hibernate is that it allows for rapid prototyping and development of new applications. That alone might be sufficient reason to use it on some projects. However, once you get past the simple initial requirements of persisting objects and get to the point where you need to run complex queries across a number of tables and perhaps use aggregate functions and other SQL capabilities, things get much more complex because Hibernate uses its own HQL query language that the runtime translates into SQL and this forces the development team through yet another learning curve. Another issue with this approach is that you can’t simply develop and test SQL and then paste the working SQL into your source code or configuration files but have to then re-code them in HQL.

Spring JDBC isn’t directly comparable to Hibernate because it doesn’t have the same goals (it doesn’t provide caching or object relational mapping for instance). However, it is certainly an alternative choice for implementing persistence. Spring JDBC provides a very small abstraction layer over raw JDBC and does this in an elegant way. Setting up a new project does require more thought and effort than using Hibernate but that time is paid back once more complex application functionality is developed.

However, the real surprise for me was how the performance and scalability compared between the two approaches. I spent a couple of days building a realistic subset of the application I was recommending the technology for and then ran benchmarks against a large database on dedicated test hardware. The results showed that Hibernate didn’t scale at all well compared to Spring JDBC for this application.

I posted some questions to the Hibernate forum to validate the approach I had taken and I received the rather terse and unhelpful reply “This is called 'A useless microbenchmark' and has been addressed a thousand times.” from a member of the Hibernate team. I did some reading of the Hibernate Performance FAQ and found some interesting statements in there, such as "We claim that Hibernate performs well" and "Many people try to benchmark Hibernate. All public benchmarks we have seen so far had (and most still have) serious flaws.". It almost seems like it is taboo to question Hibernate performance. I don't understand this defensive stance since most developers would recognize that there is a trade off between ease of use and performance.

The bottom line for me is that for my application, Spring JDBC scales much, much better, which will save significant hardware dollars (or pounds). Spring JDBC also doesn't require much investment in terms of learning curves and I can have full control over the SQL I use which in turn will help me squeeze maximum throughput out of the hardware that is available to me.

If you're considering using Hibernate on your projects I highly recommend running some performance tests based on your application requirements before getting in too deep.

[14th Oct 2007] Wow! This article has certainly attracted more attention than I expected. I would like to draw attention to that fact that I state in the above article that Hibernate does not scale well for *this* *particular* *application*. Many people seem to think I'm saying that Hibernate doesn't scale well, period. That's not what I'm saying so please read again before posting comments :-)

Labels: hibernate, performance, persistence, spring

I've been researching some ideas in my spare time over the past couple of weeks where performance is critical so I decided to put together a framework for easily running performance and scalability tests. The idea is to provide a framework that is as easy to use as JUnit and that encourages performance testing as an integral part of the development process, rather than as an afterthought before a product release or deployment.

I've called the framework JPerf and I have made the code available on sourceforge - it's just a few classes at the moment but I will be evolving this as and when I need the functionality myself.

The key interface is PerfTest which provides setUp(), test() and tearDown() methods. The test runners call setUp() once and then repeatedly call test() for the specified period of time and then call tearDown() at the end to free up resources, such as database connections.

There are currently two test runners - PerfTestRunner and ScaleTestRunner, which measure performance and scalability, respectively. Here's some sample output from the example that ships with JPerf.

PerfTestRunner:

[java] org.jperf.example.SimpleDateFormatTest: 243,971 iterations in 500ms (486,968 iterations per second)
[java] org.jperf.example.MyDateFormatTest: 371,968 iterations in 500ms (742,451 iterations per second)

ScaleTestRunner:

[java] With 1 client threads, throughput is 100.0 tps and memory usage is 5137656
[java] With 2 client threads, throughput is 202.0 tps and memory usage is 5175144
[java] With 3 client threads, throughput is 300.0 tps and memory usage is 5223616
[java] With 4 client threads, throughput is 402.0 tps and memory usage is 5271592
[java] With 5 client threads, throughput is 504.0 tps and memory usage is 5330056
[java] With 6 client threads, throughput is 602.0 tps and memory usage is 4888336
[java] With 7 client threads, throughput is 702.0 tps and memory usage is 4979288
[java] With 8 client threads, throughput is 798.0 tps and memory usage is 5057864
[java] With 9 client threads, throughput is 900.0 tps and memory usage is 5137568
[java] With 10 client threads, throughput is 1000.0 tps and memory usage is 5215200


I'd be interested to hear ideas on what features people would like to see in this framework to make it useful for their projects. If you have any requirements please leave a comment here or post a feature request on sourceforge.

Labels: java, performance