Details

    • Type: Bug Bug
    • Status: In Progress
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.1
    • Fix Version/s: None
    • Security Level: public (Anyone can view the issue)
    • Labels:
      None

      Description

      The hosted instance on onconfluence.com was hanging on to database connections and eventually started running really slow. Turns out that part of the culprit was the Contributors Plugin. The following was found in the thread-dump, actually a few of them :

      "resin-tcp-connection-j2ee.www.onconfluence.com:6802-155" daemon prio=1 tid=0x054eac50 nid=0x5f10 runnable [0x032e5000..0x032e7e20]
      	at net.sf.hibernate.type.TypeFactory.findDirty(TypeFactory.java:225)
      	at net.sf.hibernate.persister.AbstractEntityPersister.findDirty(AbstractEntityPersister.java:275)
      	at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2536)
      	at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2486)
      	at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2281)
      	at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1815)
      	at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1578)
      	at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1543)
      	at net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:49)
      	at org.springframework.orm.hibernate.HibernateTemplate$22.doInHibernate(HibernateTemplate.java:774)
      	at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:364)
      	at org.springframework.orm.hibernate.HibernateTemplate.findByValueBean(HibernateTemplate.java:769)
      	at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateSpacePermissionDao.findPermissionTypes(HibernateSpacePermissionDao.java:86)
      	at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateSpacePermissionDao.hasPermission(HibernateSpacePermissionDao.java:27)
      	at com.atlassian.confluence.security.CachingSpacePermissionManager.findPermission(CachingSpacePermissionManager.java:148)
      	at com.atlassian.confluence.security.CachingSpacePermissionManager.hasPermissionInDao(CachingSpacePermissionManager.java:49)
      	at com.atlassian.confluence.security.DefaultSpacePermissionManager.groupHasPermission(DefaultSpacePermissionManager.java:139)
      	at com.atlassian.confluence.security.DefaultSpacePermissionManager.hasPermissionViaGroups(DefaultSpacePermissionManager.java:350)
      	at com.atlassian.confluence.security.DefaultSpacePermissionManager.hasPermission(DefaultSpacePermissionManager.java:98)
      	at com.atlassian.confluence.security.SpacePermissionCheckDispatcherCoordinator.hasPermission(SpacePermissionCheckDispatcherCoordinator.java:31)
      	at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:585)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
      	at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
      	at $Proxy18.hasPermission(Unknown Source)
      	at com.atlassian.confluence.security.delegate.AbstractPermissionsDelegate.hasSpaceLevelPermission(AbstractPermissionsDelegate.java:19)
      	at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.canView(PagePermissionsDelegate.java:17)
      	at com.atlassian.confluence.security.Permission$1.checkAgainst(Permission.java:18)
      	at com.atlassian.confluence.security.DefaultPermissionManager.hasPermission(DefaultPermissionManager.java:31)
      	at com.atlassian.confluence.security.DefaultPermissionManager.getPermittedEntities(DefaultPermissionManager.java:49)
      	at sun.reflect.GeneratedMethodAccessor492.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:585)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
      	at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
      	at $Proxy32.getPermittedEntities(Unknown Source)
      	at com.atlassian.confluence.contributors.macro.ContributorsSummaryMacro.execute(ContributorsSummaryMacro.java:324)
      	at com.atlassian.renderer.v2.macro.ResourceAwareMacroDecorator.execute(ResourceAwareMacroDecorator.java:45)
      	at com.atlassian.renderer.v2.components.MacroRendererComponent.processMacro(MacroRendererComponent.java:254)
      	at com.atlassian.renderer.v2.components.MacroRendererComponent.makeMacro(MacroRendererComponent.java:150)
      	at com.atlassian.renderer.v2.components.WikiContentRendererHandler.handleMacro(WikiContentRendererHandler.java:18)
      	at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:126)
      	at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:114)
      	at com.atlassian.renderer.v2.WikiMarkupParser.handlePotentialMacro(WikiMarkupParser.java:99)
      	at com.atlassian.renderer.v2.WikiMarkupParser.parse(WikiMarkupParser.java:60)
      	at com.atlassian.renderer.v2.components.MacroRendererComponent.render(MacroRendererComponent.java:46)
      	at com.atlassian.renderer.v2.V2Renderer.render(V2Renderer.java:53)
      	at com.atlassian.renderer.v2.V2RendererFacade.convertWikiToXHtml(V2RendererFacade.java:57)
      	at sun.reflect.GeneratedMethodAccessor379.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:585)
      

        Issue Links

          Activity

          Hide
          Shannon Krebs added a comment -

          From the thread dump here:

          at $Proxy32.getPermittedEntities(Unknown Source)
          at com.atlassian.confluence.contributors.macro.ContributorsSummaryMacro.execute(ContributorsSummaryMacro.java:324)

          It looks like it got into this line (line numbers don't match up in latest version in svn)

          	pageList = permissionManager.getPermittedEntities(AuthenticatedUserThreadLocal.getUser(), Permission.VIEW, pageList);
          

          The performance of the contributors plugin is pretty bad to begin with and is not really suited to large page sets, but if there was a big enough list of pages could that cause a problem with permissionManager.getPermittedEntities ?

          Show
          Shannon Krebs added a comment - From the thread dump here: at $Proxy32.getPermittedEntities(Unknown Source) at com.atlassian.confluence.contributors.macro.ContributorsSummaryMacro.execute(ContributorsSummaryMacro.java:324) It looks like it got into this line (line numbers don't match up in latest version in svn) pageList = permissionManager.getPermittedEntities(AuthenticatedUserThreadLocal.getUser(), Permission.VIEW, pageList); The performance of the contributors plugin is pretty bad to begin with and is not really suited to large page sets, but if there was a big enough list of pages could that cause a problem with permissionManager.getPermittedEntities ?
          Hide
          Jens Schumacher added a comment -

          Our server logs are full with the following exception:

          @4000000047afaa83378007dc com.atlassian.user.impl.RepositoryException: org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: 
          Could not execute query; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; can't create statement from closed connection.; nested exception is java.sql.SQLException: 
          can't create statement from closed connection.
          @4000000047afaa833780177c 	at com.atlassian.user.impl.hibernate.HibernateUserManager.getUser(HibernateUserManager.java:123)
          @4000000047afaa833780d2fc 	at com.atlassian.user.impl.hibernate.properties.HibernatePropertySetFactory.getPropertySet(HibernatePropertySetFactory.java:52)
          @4000000047afaa833780dacc 	at com.atlassian.user.impl.hibernate.properties.HibernatePropertySetFactory.getPropertySet(HibernatePropertySetFactory.java:34)
          @4000000047afaa83378109ac 	at com.atlassian.user.impl.cache.properties.CachingPropertySetFactory.getPropertySet(CachingPropertySetFactory.java:28)
          @4000000047afaa833781117c 	at com.atlassian.user.impl.delegation.properties.DelegatingPropertySetFactory.getPropertySet(DelegatingPropertySetFactory.java:34)
          @4000000047afaa833781194c 	at bucket.user.DefaultUserAccessor.isDeactivated(DefaultUserAccessor.java:663)
          @4000000047afaa83378134a4 	at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)
          @4000000047afaa8337813c74 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          @4000000047afaa833781405c 	at java.lang.reflect.Method.invoke(Method.java:585)
          @4000000047afaa833781482c 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
          @4000000047afaa8337815f9c 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
          @4000000047afaa8337816b54 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
          @4000000047afaa8337817324 	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
          @4000000047afaa8337819a34 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
          @4000000047afaa833781a5ec 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
          @4000000047afaa833781a9d4 	at $Proxy19.isDeactivated(Unknown Source)
          @4000000047afaa833781b1a4 	at com.atlassian.confluence.security.DefaultSpacePermissionManager.hasPermission(DefaultSpacePermissionManager.java:68)
          @4000000047afaa833781ccfc 	at com.atlassian.confluence.security.SpacePermissionCheckDispatcherCoordinator.hasPermission(SpacePermissionCheckDispatcherCoordinator.java:31)
          @4000000047afaa833781d4cc 	at sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source)
          @4000000047afaa833781dc9c 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          @4000000047afaa833781f40c 	at java.lang.reflect.Method.invoke(Method.java:585)
          

          coming from:

          @4000000047afaa833783d09c 	at com.atlassian.confluence.contributors.macro.ContributorsMacro.execute(ContributorsMacro.java:246)
          @4000000047afaa833783d86c 	at com.atlassian.renderer.v2.macro.ResourceAwareMacroDecorator.execute(ResourceAwareMacroDecorator.java:45)
          @4000000047afaa833783e03c 	at com.atlassian.renderer.v2.components.MacroRendererComponent.processMacro(MacroRendererComponent.java:254)
          @4000000047afaa833783e80c 	at com.atlassian.renderer.v2.components.MacroRendererComponent.makeMacro(MacroRendererComponent.java:150)
          @4000000047afaa833783ff7c 	at com.atlassian.renderer.v2.components.WikiContentRendererHandler.handleMacro(WikiContentRendererHandler.java:18)
          @4000000047afaa8337840b34 	at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:126)
          @4000000047afaa8337840f1c 	at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:114)
          @4000000047afaa83378416ec 	at com.atlassian.renderer.v2.WikiMarkupParser.handlePotentialMacro(WikiMarkupParser.java:99)
          @4000000047afaa8337843244 	at com.atlassian.renderer.v2.WikiMarkupParser.parse(WikiMarkupParser.java:60)
          @4000000047afaa833784362c 	at com.atlassian.renderer.v2.components.MacroRendererComponent.render(MacroRendererComponent.java:46)
          @4000000047afaa8337846124 	at com.atlassian.renderer.v2.V2Renderer.render(V2Renderer.java:53)
          @4000000047afaa83378468f4 	at com.atlassian.renderer.v2.V2RendererFacade.convertWikiToXHtml(V2RendererFacade.java:57)
          

          I'm not certain whether permissionManager.getPermittedEntities results in a big performance hit. With large number of pages this is certainly possible.

          Show
          Jens Schumacher added a comment - Our server logs are full with the following exception: @4000000047afaa83378007dc com.atlassian.user.impl.RepositoryException: org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Could not execute query; uncategorized SQLException for SQL []; SQL state [ null ]; error code [0]; can't create statement from closed connection.; nested exception is java.sql.SQLException: can't create statement from closed connection. @4000000047afaa833780177c at com.atlassian.user.impl.hibernate.HibernateUserManager.getUser(HibernateUserManager.java:123) @4000000047afaa833780d2fc at com.atlassian.user.impl.hibernate.properties.HibernatePropertySetFactory.getPropertySet(HibernatePropertySetFactory.java:52) @4000000047afaa833780dacc at com.atlassian.user.impl.hibernate.properties.HibernatePropertySetFactory.getPropertySet(HibernatePropertySetFactory.java:34) @4000000047afaa83378109ac at com.atlassian.user.impl.cache.properties.CachingPropertySetFactory.getPropertySet(CachingPropertySetFactory.java:28) @4000000047afaa833781117c at com.atlassian.user.impl.delegation.properties.DelegatingPropertySetFactory.getPropertySet(DelegatingPropertySetFactory.java:34) @4000000047afaa833781194c at bucket.user.DefaultUserAccessor.isDeactivated(DefaultUserAccessor.java:663) @4000000047afaa83378134a4 at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source) @4000000047afaa8337813c74 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) @4000000047afaa833781405c at java.lang.reflect.Method.invoke(Method.java:585) @4000000047afaa833781482c at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287) @4000000047afaa8337815f9c at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181) @4000000047afaa8337816b54 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148) @4000000047afaa8337817324 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) @4000000047afaa8337819a34 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170) @4000000047afaa833781a5ec at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176) @4000000047afaa833781a9d4 at $Proxy19.isDeactivated(Unknown Source) @4000000047afaa833781b1a4 at com.atlassian.confluence.security.DefaultSpacePermissionManager.hasPermission(DefaultSpacePermissionManager.java:68) @4000000047afaa833781ccfc at com.atlassian.confluence.security.SpacePermissionCheckDispatcherCoordinator.hasPermission(SpacePermissionCheckDispatcherCoordinator.java:31) @4000000047afaa833781d4cc at sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source) @4000000047afaa833781dc9c at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) @4000000047afaa833781f40c at java.lang.reflect.Method.invoke(Method.java:585) coming from: @4000000047afaa833783d09c at com.atlassian.confluence.contributors.macro.ContributorsMacro.execute(ContributorsMacro.java:246) @4000000047afaa833783d86c at com.atlassian.renderer.v2.macro.ResourceAwareMacroDecorator.execute(ResourceAwareMacroDecorator.java:45) @4000000047afaa833783e03c at com.atlassian.renderer.v2.components.MacroRendererComponent.processMacro(MacroRendererComponent.java:254) @4000000047afaa833783e80c at com.atlassian.renderer.v2.components.MacroRendererComponent.makeMacro(MacroRendererComponent.java:150) @4000000047afaa833783ff7c at com.atlassian.renderer.v2.components.WikiContentRendererHandler.handleMacro(WikiContentRendererHandler.java:18) @4000000047afaa8337840b34 at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:126) @4000000047afaa8337840f1c at com.atlassian.renderer.v2.WikiMarkupParser.makeMacro(WikiMarkupParser.java:114) @4000000047afaa83378416ec at com.atlassian.renderer.v2.WikiMarkupParser.handlePotentialMacro(WikiMarkupParser.java:99) @4000000047afaa8337843244 at com.atlassian.renderer.v2.WikiMarkupParser.parse(WikiMarkupParser.java:60) @4000000047afaa833784362c at com.atlassian.renderer.v2.components.MacroRendererComponent.render(MacroRendererComponent.java:46) @4000000047afaa8337846124 at com.atlassian.renderer.v2.V2Renderer.render(V2Renderer.java:53) @4000000047afaa83378468f4 at com.atlassian.renderer.v2.V2RendererFacade.convertWikiToXHtml(V2RendererFacade.java:57) I'm not certain whether permissionManager.getPermittedEntities results in a big performance hit. With large number of pages this is certainly possible.
          Hide
          Jens Schumacher added a comment -

          I had to disable the plugin on our hosted instance due to performance issues related to the database connections. Changed the priority to critical since it affects the whole system.

          Show
          Jens Schumacher added a comment - I had to disable the plugin on our hosted instance due to performance issues related to the database connections. Changed the priority to critical since it affects the whole system.
          Hide
          Per Fragemann added a comment -

          I have disabled the plugin on EAC and on CAC as well until a solution is found. We didn't experience any problems so far because the plugin is not used anyway, but in case someone would use it - i will rather be on the safe side with our production systems

          Show
          Per Fragemann added a comment - I have disabled the plugin on EAC and on CAC as well until a solution is found. We didn't experience any problems so far because the plugin is not used anyway, but in case someone would use it - i will rather be on the safe side with our production systems
          Hide
          Matt Ryall added a comment -

          I came across this plugin in a support case, where it was causing lots of problems in a large instance of Confluence.

          The main problem with the implementation is that the PageSearchHelper isn't using the search index. In some cases, it can retrieve an insane amount of data from the database in order to answer its queries.

          This class needs to be rewritten to use the SmartListManager and ListQuerys to work efficiently in large instances. We have a ContentPermissionsFilter which can be used to restrict access to search results efficiently. See the SearchQueryBean in Confluence for a sample implementation, and how we do our queries effectively for labels, pages, blogs, etc.

          Show
          Matt Ryall added a comment - I came across this plugin in a support case, where it was causing lots of problems in a large instance of Confluence. The main problem with the implementation is that the PageSearchHelper isn't using the search index. In some cases, it can retrieve an insane amount of data from the database in order to answer its queries. This class needs to be rewritten to use the SmartListManager and ListQuerys to work efficiently in large instances. We have a ContentPermissionsFilter which can be used to restrict access to search results efficiently. See the SearchQueryBean in Confluence for a sample implementation, and how we do our queries effectively for labels, pages, blogs, etc.
          Hide
          Nate Nash added a comment -

          I will second that is kills our 2-node clustered instance as well.

          Show
          Nate Nash added a comment - I will second that is kills our 2-node clustered instance as well.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jens Schumacher
            • Votes:
              6 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Who's Looking?