Connection pool should be configurable for all FWs about frameworkbenchmarks

[pfalls1]

@cies Thanks for the suggestion. I like the idea of easily configuring the connection pool, but that one might be too much of a hassle to implement in the growing number of tests that we have.

As for having a consistent pool size across all frameworks, there's probably some benefit to that, and I think in general all frameworks should have their pool set to 256 (which is the highest concurrency that we reach in our tests). I'm hesitant to force that restriction on all frameworks though, in the off chance that a framework does indeed perform better with a different sized pool. I do think it's a worthwhile exercise to ensure that every framework has their connection pool configured and set to a reasonable size.

[cies]

As long as the pools are not equal in size for the contenders, the benchmark is testing "default configurations" which is not very interesting.

I think 256 is a bit large for a pool. 10-30 will do for my gut feeling. There are aslo some other conf vars governing the connections (timeouts, minimum size, etc.), best is to set them all to the same values for all FWs.

See:
http://docs.oracle.com/cd/E19316-01/820-4343/abehs/index.html

[bhauer]

Hi @cies. Pat has made a separate issue (#118) for normalizing the connection string used for all the JVM frameworks. So far, in our spot checking, the connection string changes have very little impact. Still, I don't think the tuning options in question will hurt any of the frameworks, so it seems reasonable to normalize this.

Other JDBC configuration options such as transaction isolation level should be managed by the ORM provided by the framework, or in the case of servlet-raw (or any other raw JDBC tests), managed directly in the code.

Where you see 256 as large for a connection pool, I consider it fairly typical. We often deploy applications to production environments with a minimum connection pool size of 200+, reason being our applications tend to use databases for put and get operations and not complex queries. I feel it's fair to say there is not widespread agreement on a good application-agnostic connection pool size (among the hundreds of other tuning parameters for database connectivity and beyond).

Tuning can always be done with knowledge of the application particulars, and we encourage that here, especially in light of a desire to see each framework perform as well as possible. The original spirit of the exercise was to benchmark each framework using what we believed was a reasonable production-grade configuration given the general best-practices prescribed by the framework's documentation. If from there, thanks to community contributions, we end up with more tests that are fine-tuned for the benchmark particulars, I think that's ultimately a good thing. But we do need to keep re-evaluating how and where to paint the line between a standard test and a "stripped" test. I think it's useful to many readers to see how every framework works out of the box when you select its "production" deployment profile (if one exists) versus how it performs with application-specific tuning (which we've called "stripped" here).

All that said, my suspicion is that most frameworks will benefit from having a connection pool sized at least equal to the concurrency level of the test (assuming no server-side concurrency fan-out). Such a configuration allows each of the N requests the opportunity to be begin a query without the risk of waiting for an on-demand connection if the other N-1 requests are all mid-query already.

[slorber]

256 concurrent users = 256 connections -> not sure this rule should apply for an asynchronous driver

[xxgreg]

PostgreSQL recommends ~2*cores connections for optimal throughput. 256 is far too high.

https://wiki.postgresql.org/wiki/Number_Of_Database_Connections

@slorber Handling 256 HTTP requests concurrently does not imply that 256 concurrent database transactions will result in optimal database throughput. A larger number of concurrent queries will increase contention, which increases the amount of work that the database must do. This applies to both a sync and an async driver.

[xxgreg]

For the record, it appears that many postgresql benchmarks do not use 256 connections.

Servlet-postgresql-raw uses a default which is 5. (This could probably be set higher for better performance)

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/servlet/src/main/webapp/WEB-INF/resin-web.xml
http://www.caucho.com/resin-3.1/doc/database-tags.xtp#maxcreateconnections

Undertow uses a default too. Not sure what it is.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a919d8945a1a46826cf551811ccae805ad756144/frameworks/Java/undertow/src/main/resources/hello/server.properties

Aspnet also uses a default. Not sure what it is.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/a919d8945a1a46826cf551811ccae805ad756144/frameworks/C%23/aspnet/src/Web.config