Active Objects runs out of connections when migrating enough times

Description

Our tests run and cause the AO migrate to happen... a lot.

Java Compatibility Information : JIRA version = 1000.5.0, Java Version = 1.8.0_74

Eventually..... we get this

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] 2016-04-21 06:52:19,457 active-objects-init-JiraTenantImpl{id='system'}-0 ERROR admin 405x2679x1 bkrp7b 127.0.0.1 /rest/testkit-test/1.0/dataImport [NoModule] There was an error getting a DBCP datasource. build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:60) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:317) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:81) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:164) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:136) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.jira.ofbiz.DefaultOfBizConnectionFactory.getConnection(DefaultOfBizConnectionFactory.java:44) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.jira.JiraTenantAwareDataSourceProvider$OfBizDataSource.getConnection(JiraTenantAwareDataSourceProvider.java:103) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.DataSourceProviderActiveObjectsFactory$ActiveObjectsDataSource.getConnection(DataSourceProviderActiveObjectsFactory.java:93) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 2 filtered build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:498) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.builder.DelegatingDisposableDataSourceHandler.delegate(DelegatingDisposableDataSourceHandler.java:37) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.builder.DelegatingDisposableDataSourceHandler.invoke(DelegatingDisposableDataSourceHandler.java:30) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy3243.getConnection(Unknown Source) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.DatabaseProvider.getConnection(DatabaseProvider.java:864) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.DatabaseProvider.hasIndex(DatabaseProvider.java:1442) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.db.PostgreSQLDatabaseProvider.renderDropIndex(PostgreSQLDatabaseProvider.java:252) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.DatabaseProvider.renderAction(DatabaseProvider.java:238) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.schema.SchemaGenerator$2.apply(SchemaGenerator.java:126) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.schema.SchemaGenerator$2.apply(SchemaGenerator.java:124) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.google.common.collect.Iterators$8.transform(Iterators.java:799) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.schema.SchemaGenerator.migrate(SchemaGenerator.java:90) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at net.java.ao.EntityManager.migrateDestructively(EntityManager.java:141) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.migrateDestructively(EntityManagedActiveObjects.java:54) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 3 filtered build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:498) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.pocketknife.api.ao.util.ActiveObjectsKit.invokeMigrateDestructively(ActiveObjectsKit.java:101) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.pocketknife.api.ao.util.ActiveObjectsKit.migrateDestructively(ActiveObjectsKit.java:81) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.pocketknife.api.ao.util.ActiveObjectsKit.migrateDestructively(ActiveObjectsKit.java:75) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.servicedesk.internal.ao.upgrade.task006.AoUpgradeTask006.doUpgrade(AoUpgradeTask006.java:31) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.pocketknife.spi.ao.AbstractActiveObjectsUpgradeTask.upgrade(AbstractActiveObjectsUpgradeTask.java:63) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl$1.doInTransaction(ActiveObjectUpgradeManagerImpl.java:68) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl$1.doInTransaction(ActiveObjectUpgradeManagerImpl.java:64) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.jira.DefaultHostContextAccessor.doInTransaction(DefaultHostContextAccessor.java:34) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 2 filtered build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:498) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy477.doInTransaction(Unknown Source) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 2 filtered build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:498) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy477.doInTransaction(Unknown Source) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 2 filtered build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:498) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy893.execute(Unknown Source) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:42) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:16) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:198) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl.upgrade(ActiveObjectUpgradeManagerImpl.java:64) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl.upgrade(ActiveObjectUpgradeManagerImpl.java:36) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.AbstractActiveObjectsFactory.upgrade(AbstractActiveObjectsFactory.java:87) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.AbstractActiveObjectsFactory.create(AbstractActiveObjectsFactory.java:67) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.internal.DelegatingActiveObjectsFactory.create(DelegatingActiveObjectsFactory.java:32) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:91) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:42) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.util.concurrent.FutureTask.run(FutureTask.java:266) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at java.lang.Thread.run(Thread.java:745) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:142) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:321) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:318) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:54) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] Caused by: java.util.NoSuchElementException: Timeout waiting for idle object build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) build 21-Apr-2016 10:52:19 [INFO] [talledLocalContainer] ... 89 more build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Database configuration OK build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] ___ Database Configuration _________________ build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Loading entityengine.xml from : file:/opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/SDHMASTER-SDHMASTER3086-JAVAINTCLIENT3/tests/integration-tests-parent/integration-tests/target/container/tomcat8x/cargo-jira-home/webapps/jira/WEB-INF/classes/entityengine.xml build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Entity model field type name : postgres72 build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Entity model schema name : public build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Database Version : PostgreSQL - 9.2.16 build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Database Driver : PostgreSQL Native Driver - PostgreSQL 9.1 JDBC4 (build 903) build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Database URL : jdbc:postgresql://localhost:15432/testservicedeskdb build 21-Apr-2016 10:27:51 [INFO] [talledLocalContainer] Database JDBC config : postgres72 jdbc:postgresql://localhost:15432/testservicedeskdb

Environment

None

Testing Notes

Add notes...

Status

Assignee

Unassigned

Reporter

Brad Baker

Labels

None

Add-on Type

None

Team

None

CC

None

Risk factor

None

QA Kickoff Status

None

QA Demo Status

None

Priority

Major
Configure