End of Life healthcheck upgrade task blocks JIRA startup

Description

Step to reproduce

  1. Install JIRA 7.0.1 on a server without any internet access

  2. upgrade to version 7.2.x

Environment

  • jira-healthcheck-plugin ver 2.1.11+

Expected behaviour

JIRA should startup correctly.

Actual behaviour

JIRA stuck in startup page forever, the log stops with the following line:

1 2017-06-01 03:55:19,745 JIRA-Bootstrap INFO [c.a.j.p.healthcheck.upgrade.InstanceHealthCopyEolFileTask] Running JIRA Instance Health upgrade task 1. Copying the default eol JSON file into local home directory

Diagnostic

The following thread from thread dump output, it running forever:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 "JIRA-Bootstrap" #17 daemon prio=5 os_prio=0 tid=0x000000001fe1a800 nid=0xfb0 runnable [0x00000000205bb000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) - locked <0x00000000fc32cb18> (a java.lang.Object) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) - locked <0x00000000fc32cbd8> (a java.lang.Object) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.jav a:134) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java: 353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.atlassian.jira.plugins.healthcheck.util.SupportEolCheckUtil.fetchAndProcessVersions(SupportEolCheckUtil.j ava:65) at com.atlassian.jira.plugins.healthcheck.upgrade.InstanceHealthCopyEolFileTask.doUpgrade(InstanceHealthCopyEolF ileTask.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.jav a:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java: 60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor .java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.ja va:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLI nterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.j ava:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvic e.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor .java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.ja va:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy836.doUpgrade(Unknown Source) at com.atlassian.sal.core.upgrade.PluginUpgrader.doUpgrade(PluginUpgrader.java:72) at com.atlassian.sal.core.upgrade.PluginUpgrader.upgrade(PluginUpgrader.java:57) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager$1.doInTransaction(DefaultPluginUpgradeManager.java :198) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager$1.doInTransaction(DefaultPluginUpgradeManager.java :173) at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTem plate.java:21) at com.atlassian.jira.DefaultHostContextAccessor.doInTransaction(DefaultHostContextAccessor.java:34) at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocati onHandler.java:26) at com.sun.proxy.$Proxy454.doInTransaction(Unknown Source) at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(Hos tComponentFactoryBean.java:136) at com.sun.proxy.$Proxy454.doInTransaction(Unknown Source) at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java :18) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager.upgradePlugin(DefaultPluginUpgradeManager.java:173 ) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager.upgradeInternal(DefaultPluginUpgradeManager.java:1 52) at com.atlassian.sal.jira.upgrade.JiraPluginUpgradeManager.upgradeInternal(JiraPluginUpgradeManager.java:49) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager.upgrade(DefaultPluginUpgradeManager.java:136) at com.atlassian.sal.core.upgrade.DefaultPluginUpgradeManager.onStart(DefaultPluginUpgradeManager.java:80) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:277) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$4.consume(DefaultLifecycleManager.java:274) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecyleAware(DefaultLifecycleManager.java:303 ) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyOnStartIfStartedAndEnabled(DefaultLifecycleMan ager.java:273) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.access$300(DefaultLifecycleManager.java:49) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:235) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager$3.evaluate(DefaultLifecycleManager.java:232) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyLifecycleAwares(DefaultLifecycleManager.java:2 58) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.notifyStartableLifecycleAwares(DefaultLifecycleManag er.java:231) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.startIfApplicationSetup(DefaultLifecycleManager.java :219) at com.atlassian.sal.core.lifecycle.DefaultLifecycleManager.start(DefaultLifecycleManager.java:210) at com.atlassian.sal.jira.lifecycle.JiraLifecycleManager.onJiraStart(JiraLifecycleManager.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker .java:36) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299) at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:10 7) at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160) at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79) at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:73) at com.atlassian.jira.upgrade.PluginUpgradeLauncher.start(PluginUpgradeLauncher.java:30) at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:51) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:178) at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$423/83691638.run(Unknown Source) at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139 ) at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55) at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgrad eManager.java:42) at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:171) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:159) at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$422/1114914902.run(Unknown Source) at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55) at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:156) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145) at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$36/249444439.run(Unknown Source) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManag erImpl.java:298) at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConf igurationManagerImpl.java:194) at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135) at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:101) at com.atlassian.jira.startup.DefaultJiraLauncher$$Lambda$3/1906317836.run(Unknown Source) at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31) at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:99) at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149) at com.atlassian.jira.startup.LauncherContextListener$$Lambda$2/1250664460.run(Unknown Source) at java.lang.Thread.run(Thread.java:745)

Workaround solution

  1. Disable the support tools plugin via database query.

  2. Set the DNS host file to:

    1 127.0.0.1 marketplace.atlassian.com

    which will force the request to failed and bypass the upgrade task.

Environment

None

Testing Notes

None

Status

Assignee

Unassigned

Reporter

Yilin Mo

Add-on Type

None

Team

None

CC

None

Risk factor

None

QA Kickoff Status

None

QA Demo Status

None

Affected Product/s

None

Fix versions

Priority

Major
Configure