vladmihalcea / flexy-pool

FlexyPool adds metrics and failover strategies to a given Connection Pool, allowing it to resize on demand.

Initial examples

My personal blog features some code examples that are included in this repository.

• customer-lock - contains concurrency examples
• db-facts - contains DB related examples
• hibernate-facts - contains various Hibernate examples
• misc - contains unclassified examples
• mongodb-facts - contains MongoDB examples

Are you struggling with application performance issues?

Imagine having a tool that can automatically detect if you are using JPA and Hibernate properly. No more performance issues, no more having to spend countless hours trying to figure out why your application is barely crawling.

Imagine discovering early during the development cycle that you are using suboptimal mappings and entity relationships or that you are missing performance-related settings.

More, with Hypersistence Optimizer, you can detect all such issues during testing and make sure you don't deploy to production a change that will affect data access layer performance.

Hypersistence Optimizer is the tool you've been long waiting for!

Training

If you are interested in on-site training, I can offer you my High-Performance Java Persistence training which can be adapted to one, two or three days of sessions. For more details, check out my website.

Consulting

If you want me to review your application and provide insight into how you can optimize it to run faster, then check out my consulting page.

High-Performance Java Persistence Video Courses

If you want the fastest way to learn how to speed up a Java database application, then you should definitely enroll in my High-Performance Java Persistence video courses.

High-Performance Java Persistence Book

Or, if you prefer reading books, you are going to love my High-Performance Java Persistence book as well.

flexy-pool's Issues

Investigate HikariCP Java version changes and document/adapt accordingly

Fix C3P0 test that fails spuriously

The C3P0 test fails from time to time because the pool is not properly initialized by the time we want to fetch the first connection.

Add a Developer Guide wiki page

For those who want to contribute there should be a developer guide to help them kick off.

Add support for Micrometer?

I am using Spring Boot2.
In Spring Boot 2, it uses Micrometer instead of dropwizard metrics.

Do you have any plan to support for micrometer?

ref: https://micrometer.io/

Odd Dependency Structure within HikariCP JAR

While starting to integrate flexy-pool with a Java SE app, I found the following CNFE:

Caused by: java.lang.NoClassDefFoundError: com/codahale/metrics/JmxReporter
	at com.vladmihalcea.flexypool.metric.codahale.JmxMetricReporter.init(JmxMetricReporter.java:28)
	at com.vladmihalcea.flexypool.metric.codahale.CodahaleMetrics.<init>(CodahaleMetrics.java:87)
	at com.vladmihalcea.flexypool.metric.codahale.CodahaleMetrics.<init>(CodahaleMetrics.java:102)
	at com.vladmihalcea.flexypool.metric.codahale.CodahaleMetrics$ReservoirMetricsFactory.newInstance(CodahaleMetrics.java:45)
	at com.vladmihalcea.flexypool.config.Configuration$Builder.build(Configuration.java:164)

This was confusing, since I didn't declare any metrics runtime (yet), and was planning to use DWM4.

The following is part of the dependency tree I see within my app:

[INFO] +- com.vladmihalcea.flexy-pool:flexy-hikaricp:jar:1.3.0:compile
[INFO] |  +- com.vladmihalcea.flexy-pool:flexy-pool-core:jar:1.3.0:compile
[INFO] |  +- com.vladmihalcea.flexy-pool:flexy-codahale-metrics:jar:1.3.0:compile
[INFO] |  +- com.vladmihalcea.flexy-pool:flexy-dropwizard-metrics:jar:1.3.0:compile
[INFO] |  \- io.dropwizard.metrics:metrics-core:jar:4.0.2:compile

I wouldn't expect a hard compile time dependency on these components, they don't seem like they're hard dependencies either within the code. So should they be removed? Or excluded?

HikariCP returns also SQLTransientConnectionException when connection can be retried

HikariCP after version 2.4.1 (issue brettwooldridge/HikariCP#349) can also return SQLTransientConnectionException. Seems it could also be handled same way as SQLTimeoutException.

Usage of DataSourceFactory

Nice work;
It should be interesting to define an implementation of org.osgi.service.jdbc.DataSourceFactory

Throw UnsupportedOperationException for BoneCp and Vibur DBCP on pool resize

BoneCP and Vibur DBCP does not support pool resizing natively, as C3P0. This way, it's impossible to guarantee what will happen to the current acquired connections one the pool has to be destroyed and recreated, only to take into consideration the new pool size.

Therefore, the safest approach is to throw an UnsupportedOperationException whenever the max pool size is about to be changed and document the behavior.

Add support for Agroal

https://github.com/agroal/agroal

Add support for Vibur-DBCP

Add time threshold pool size grow strategy

In our current enterprise system we've been struggling to lower the poolingDataSource.acquisitionTimeout from the default 30s to some more appropriate value (1-5s).

For "classic" DataSource.getConnection() scenarios it was possible, thanks to using the IncrementPoolOnTimeoutConnectionAcquiringStrategy.

The only problem is the recovery thread, since we cannot intercept the time-out exceptions and there is no default retrying mechanisms or the possibility for using a separate connection from the regular pooled connections.

So we get:

bitronix.tm.recovery.RecoveryException: cannot start recovery on a PoolingDataSource containing an XAPool of resource dtfDataSource with 7 connection(s) (0 still available)
at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:288) ~[btm-2.1.3.jar:2.1.3]
at bitronix.tm.recovery.Recoverer.recover(Recoverer.java:258) [btm-2.1.3.jar:2.1.3]
at bitronix.tm.recovery.Recoverer.recoverAllResources(Recoverer.java:226) [btm-2.1.3.jar:2.1.3]
at bitronix.tm.recovery.Recoverer.run(Recoverer.java:142) [btm-2.1.3.jar:2.1.3]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]
Caused by: bitronix.tm.internal.BitronixRuntimeException: XA pool of resource dtfDataSource still empty after 1s wait time
at bitronix.tm.resource.common.XAPool.waitForConnectionInPool(XAPool.java:423) ~[btm-2.1.3.jar:2.1.3]
at bitronix.tm.resource.common.XAPool.getInPool(XAPool.java:374) ~[btm-2.1.3.jar:2.1.3]
at bitronix.tm.resource.common.XAPool.getConnectionHandle(XAPool.java:123) ~[btm-2.1.3.jar:2.1.3]
at bitronix.tm.resource.jdbc.PoolingDataSource.startRecovery(PoolingDataSource.java:284) ~[btm-2.1.3.jar:2.1.3]
... 4 common frames omitted

So, this is what we need:

• Add timeoutMillis to IncrementPoolOnTimeoutConnectionAcquiringStrategy, that should check if the current connection acquiring time is grater than a threshold and attempt to grow the pool according to the remaining overflow size.

This way we could still have a large Bitronix poolingDataSource.acquisitionTimeout and grow the pool even before we get a Timeout.

The Bitronix retrying policy is actually happening in the XAPool.getConnectionHandle while(true) loop, that keeps on attempting until it hits the timeout threshold. The new strategy will allow to grow the pool size even if we managed to get a connection after 1-5s.

Add support for Atomikos using non-XA JDBC implementations

Looks like AtomikosPoolAdapter could work with the Non-XA DataSource as well.

Include uniqueName in the metric names

Hello and thank you guys for your great work. Flexypool is amazingly helpful.
My problem is that I have multiple connection pools(different datasources) and they all want to use the same metric name since they aren't factoring the uniqueName into it. I was able to figure out how to do this on my own, it just felt a bit more difficult than I'd like and it seems like including the uniqueName in the metric name might be good default behavior. Perhaps I am missing an easier way to do it?

For reference, here's how I got around it:

 configurationBuilder.setMetricsFactory(new MetricsFactory() {
  public Metrics newInstance(com.vladmihalcea.flexypool.common.ConfigurationProperties configurationProperties) {

   CompositeMeterRegistry globalRegistry = io.micrometer.core.instrument.Metrics.globalRegistry;

   return new MicrometerMetrics(configurationProperties, globalRegistry){
     @Override
     public Histogram histogram(String name) {
       return super.histogram(poolName + StringUtils.capitalize(name));
     }

     @Override
     public Timer timer(String name) {
       return super.timer(poolName + StringUtils.capitalize(name));
     }
   };
  }
});

Add code coverage support for the Travic-CI integration

When building the project with Travis, we should have code coverage running and also badge on the front page

Add a new ProxyFactory to use a ConnectionDecorator object instead of a Proxy

For performance reasons, a Proxy will never be as fast as an actual delegate class. While the Proxy allows the library to be compatible with future Java versions that keep on adding methods to the Connection class hierarchy (as we got used to it by now), the performance penalty might be to much for paying of.

The ConnectionDecorator simply delegates all calls to the actual Connection object. The newer Java 1.7 and 1.8 methods are invoked using reflection, to allow compiling the project with Java 1.6

Add Maven Central and License badges

Add link to Maven central and the license badge

Does flexy-pool supports DB2 and Oracle Datasources?

We uses the below connection pools in our application

1. DB2DataSource 2) OracleDataSource

Does current flexy-pool supports DB2 and Oracle Datasources?
Can we implement flexypool metrics and failover strategies to the above connection pools?
I didn't find any specific doc or example code related to it under vladmihalcea's flexypool github repository.

Could you please suggest on this?

Investigate if ViburDBCP 4.0 poolReducerClass can help resize the pool on timeout

The poolReducerClass is configurable in ViburDBCP, so it might be useful for re-sizing the pool.

Add support for Dropwizard Mertics 4

As explained in this issue, DW Metrics 4 has added several changes:

1. They did not rename the package after all which makes the DW FlexyPool module dependency useless
2. They moved the JmxReporter to a new package.

So, we need to release version 2.0 for that and change the DW module to reflect all these changes.

Dropwizard Metrics

We are using Flexypool with HikariCP for our dropwizard app. We push our dropwizard metrics to a graphite server.

Wanted to push flexy pool metrics to the same graphite server, how do we proceed with the same. Can we pass the Dropwizard metrics registry to flexypool metric factory?

Tomcat CP support

Add support for Tomcat database connection pool.

Testimonial of FlexyPool usage from Scentbird

Hello, Vlad Mihalcea!

We have tried FlexyPool in Scentbird, after one week of intensive testing we decided switch it to FlexyPool + HikariCP in production mode, and everything works very robust and fast, we so happy about that! Thank you for this library.

This is link to our logo:
http://cdn.scentbird.com/github-logo-no-shadow.svg

And description:
Scentbird.com is a website that allows its subscribers to try hundreds of designer fragrances at very affordable prices, every month for just $14.95.

— Andrey Rebrov, CTO, Scentbird

Btw, we want a free copy of your book, High-Performance Java Persistence. ;-)

Spring Boot support?

Any plan to implement flexy-pool-spring-boot-starter? It will be very convenient for microservices application developers. Developers just import flexy-pool-spring-boot-starter and all will be ready for usage and metrics.

Dead link for HikariCP on README

Amendment below:

• HikariCP

For some reason I couldn't make a PR for it as I couldn't push change/branch upstream...

remote: Permission to vladmihalcea/flexy-pool.git denied to Aeonitis.

Integrate Travis-CI and add build pass badge

The project should be build continuously using Travis and the build status should be available on the project front page

Add support for Atomikos5

Atomikos 5 has been released, and it has some changes in its API, that made flexy-atomikos incompatible with it.

https://www.atomikos.com/Blog/TransactionsEssentials5dot0

Add documentation for release 1.2.0

The 1.2.0 release adds a lot of new features that require detailed explanations.

Update documentation according to release 1.2.2

Increase test coverage

The utilities constructors and the Vibur and BoneCP max pool size getters and setters and not covered by tests.

Add support for Narayana TM

The JBoss Narayana TM should be supported by FlexyPool, to make it available to JBoss and Wildfly Application servers.

Ability to override MetricsFactory used

While looking at https://github.com/vladmihalcea/flexy-pool/blob/master/flexy-pool-core/src/main/java/com/vladmihalcea/flexypool/metric/MetricsFactoryResolver.java I saw that:

• This is a final class, not extensible
• There's no way to manipulate the behavior, override the instance used
• It has multiple responsibilities
• Has no interface, no way to replace at runtime what instance will be used

Basically, everything against SOLID

I was hoping that at runtime, I could replace the resolver to use my own resolver, or somehow replace the behavior. To do that, I would need to change something within Flexy Pool. That's not possible. Ideally, what I'd like to do is extend the DWM metrics implementation, since my metrics are based on it, but use my CDI provided MetricsRegistry as the backing object instead. There's no way to do that, without duplicating a lot of logic.

Remove version tags from JavaDocs

The version tags bring no value to the current Java Documentation

Add javadoc badge

The javadoc badge should point to the javadoc.io project location

Add build scripts to automate the release procedure

Add support for Dropwizard Metrics 4.0.0

Dropwizard Metrics 4.0.0 has changed all Metrics packages from:

com.codahale.metrics

to

io.dropwizard.metrics

Check the Maven repository for more information:

https://oss.sonatype.org/content/repositories/snapshots/io/dropwizard/metrics/metrics-core/

Add proxy support for Java EE

The XAConnection is only used internally, since Application Servers must supply a javax.sql.DataSource to the clients, even when the underlying XAResource is enlisted in a Transaction. So the javax.sql.DataSource is the one we need to monitor anyway.

It's probably better to supply a Generic DataSource adaptor that handles only monitoring and cannot auto-scale on demand. This could probably be the best way to integrate AS DataSources.

MicrometerMetrics should have a "flexypool" prefix and a poolname=uniqueName parameter in the metrics

How to configure flexy-pool if my connection pool is not supported now?

Hello, I see that flexy-pool does not support druid now, and I have no idea how to configure flexy-pool? (sorry for my poor English)

Broken link for Mitch&Mates in README

The mitchandmates link in the readme file is broken:
https://github.com/vladmihalcea/flexy-pool/blob/master/README.md

Update Dropwizard Metrics dependency to use the released version

When Dropwizard Metrics 4.x.x version is released to Maven Central, it will be time to switch from using the SNAPSHOT dependency.

Add support for event notifications

FlexyPool should allow registering event listeners (e.g. connection acquire failures, connection acquire max threshold, connection lease time max threshold) and propagate these events to the application logic. This way, the application developer can leverage some alert mechanism in case of connection depletion.

Update Wiki to include 1.2.3 and 1.2.4 release changes

The 1.2.3 release has added two major features:

• connection decorators instead of dynamic proxies
• event notifications

These changes should be documented in the project Wiki.

Clear guide on how to use

Can we get a clear guide on how to use the flexy pool on java 6?
I tried adding this dependency but it says artifact missing.

com.vladmihalcea.flexy-pool flexy-dropwizard3-metrics ${flexy-pool.version}

Add Release Notes wiki page

There should be a Release Notes wiki page instructing what features are provided when migrating to a given release.

Review all JavaDocs

Review all current JavaDocs and make them as informative as possible

Flexypool not showing any stats

Hi,
I have added flexypool into my applicaiton but when I observe the MBeans on Jconsole , it doesn't show any activity.

Here is what I did:
In pom added below entry:

<dependency>
   <groupId>com.vladmihalcea.flexy-pool</groupId>
	<artifactId>flexy-dbcp</artifactId>
    <version>1.3.0</version>
    <exclusions>
        <exclusion>
            <groupId>com.vladmihalcea.flexy-pool</groupId>
            <artifactId>flexy-dropwizard-metrics</artifactId>
        </exclusion>
    </exclusions>
</dependency>

In ApplicationContext.xml added below entries:

<bean id="configurationBuilder" class="com.vladmihalcea.flexypool.config.Configuration$Builder">
	<constructor-arg value="uniqueId"/>
	<constructor-arg ref="oracleDataSource"/>
	<constructor-arg value="#{ T(com.vladmihalcea.flexypool.adaptor.DBCPPoolAdapter).FACTORY }"/>
</bean>

<bean id="configuration" factory-bean="configurationBuilder" factory-method="build"/>

<bean id="dataSource" class="com.vladmihalcea.flexypool.FlexyPoolDataSource" init-method="start" destroy-method="stop">
	<constructor-arg ref="configuration"/>
	<constructor-arg>
		<array>
			<bean class="com.vladmihalcea.flexypool.strategy.IncrementPoolOnTimeoutConnectionAcquiringStrategy.Factory">
				<constructor-arg value="5"/>
			</bean>
			<bean class="com.vladmihalcea.flexypool.strategy.RetryConnectionAcquiringStrategy.Factory">
				<constructor-arg value="2"/>
			</bean>
		</array>
	</constructor-arg>
</bean>

oracleDataSource here refers to the datasource of my application

After this.. I started the server, and navigated through my application but in JConsole's Mbeans.. I dont see any activity.

Note: I am on weblogic and jdk 1.6

Flexy-pool with hibernate 4.2

We are using an old version of Hibernate (4.2.14.Final) with JBoss EAP 6.1-Final.

I wanted to give a try to flexy-pool to test some of the performance hints given by your great book (High-Performance Java Persistence). However when I try to start our application following exception is thrown:

Caused by: java.lang.NoClassDefFoundError: org/hibernate/engine/jdbc/connections/internal/DatasourceConnectionProviderImpl

Is possible to use flexy-pool with Hibernate 4.2?

flexy-pool-core-java9 requires Java 9, but should require Java 8 according to the documentiation

Major version number of class file is 53, which leads to errors when used with JRE 8.

Couldn't load ConnectionDecoratorFactoryService on the current JVM
java.lang.UnsupportedClassVersionError: com/vladmihalcea/flexypool/connection/Java9ConnectionDecoratorFactoryService 
has been compiled by a more recent version of the Java Runtime (class file version 53.0), 
this version of the Java Runtime only recognizes class file versions up to 52.0 
(unable to load class [com.vladmihalcea.flexypool.connection.Java9ConnectionDecoratorFactoryService])
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2342) ~[catalina.jar:8.5.32]
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:829) ~[catalina.jar:8.5.32]
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1278) ~[catalina.jar:8.5.32]
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137) ~[catalina.jar:8.5.32]
	at java.lang.Class.forName0(Native Method) ~[?:1.8.0_181]
	at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_181]
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370) ~[?:1.8.0_181]
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_181]
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_181]
	at com.vladmihalcea.flexypool.connection.ConnectionDecoratorFactoryResolver.resolve(ConnectionDecoratorFactoryResolver.java:38) ~[flexy-pool-core-2.0.0.jar:?]
...

Add support for Atomikos TransactionsEssentials

Support for multiple JNDI Data Sources in Java EE

A Java EE application might have multiple DataSources that need monitoring. If the default properties can be overridden, then we can use the FlexyPool Hibernate ConnectionProvider to pass specific properties via the persistence.xml configuration file.