Random error codes for "Too many connection problem"

Description

With more customers we have more and more problems with stability of the API. It is almost impossible to predict when an app will reach API limits. Error codes are not helping. It is very common to get 500 status code or even 403 and 400

I will put examples in comments. I can provide url of instances if that helps

Environment

None

Activity

Show:
Maciej Dudziak [Deviniti]
August 21, 2019, 9:53 AM

We have added a simple retry mechanism for various status codes and now it is clearly visible that 403 is random:

GET:

2019-08-20T11:49:55+02:00 issue-templates-cloud app/web.1 get {"url":"/rest/atlassian-connect/1/addons/com.intenso.jira.issue-templates/properties/field","headers":{"Accept":"application/json","Content-Type":"application/json","x-atlassian-force-account-id":true}}

OUTPUT:

Aug 20 11:49:57 issue-templates-cloud app/web.1: <html>

Aug 20 11:49:57 issue-templates-cloud app/web.1: <head> Aug 20 11:49:57 issue-templates-cloud app/web.1: <title>Forbidden (403)</title>

Aug 20 11:49:57 issue-templates-cloud app/web.1: <!--[if IE]><![endif]-->

Aug 20 11:49:57 issue-templates-cloud app/web.1: <script type="text/javascript" >

….. trimmed scripts for better visibility

<body id="jira" class="aui-layout aui-style-default page-type-message" data-version="1001.0.0-SNAPSHOT" >

Aug 20 11:49:57 issue-templates-cloud app/web.1: <div class="aui-page-panel"><div class="aui-page-panel-inner">

Aug 20 11:49:57 issue-templates-cloud app/web.1: <section class="aui-page-panel-content">

Aug 20 11:49:57 issue-templates-cloud app/web.1: <header class="aui-page-header"><div class="aui-page-header-inner">

Aug 20 11:49:57 issue-templates-cloud app/web.1: <div class="aui-page-header-main">

Aug 20 11:49:57 issue-templates-cloud app/web.1: <h1>Forbidden (403)</h1>

Aug 20 11:49:57 issue-templates-cloud app/web.1: </div><!-- .aui-page-header-main -->

Aug 20 11:49:57 issue-templates-cloud app/web.1: </div><!-- .aui-page-header-inner --></header><!-- .aui-page-header -->

Aug 20 11:49:57 issue-templates-cloud app/web.1: <div class="aui-message aui-message-warning warning">

Aug 20 11:49:57 issue-templates-cloud app/web.1: <p>Encountered a <code>&quot;403 - Forbidden&quot;</code> error while loading this page.</p>

Aug 20 11:49:57 issue-templates-cloud app/web.1: <p><a href="/secure/MyJiraHome.jspa">Go to Jira home</a></p>

Aug 20 11:49:57 issue-templates-cloud app/web.1: </div>

Aug 20 11:49:57 issue-templates-cloud app/web.1: </section><!-- .aui-page-panel-content -->

Aug 20 11:49:57 issue-templates-cloud app/web.1: </div><!-- .aui-page-panel-inner --></div><!-- .aui-page-panel -->

Aug 20 11:49:57 issue-templates-cloud app/web.1: </body>

Aug 20 11:49:57 issue-templates-cloud app/web.1: </html>

RETRY AFTER 2 seconds:

2019-08-20T11:49:59+02:00 issue-templates-cloud app/web.1 get {"url":"/rest/atlassian-connect/1/addons/com.intenso.jira.issue-templates/properties/field","headers":{"Accept":"application/json","Content-Type":"application/json","x-atlassian-force-account-id":true}}

This time success:

2019-08-20T11:49:59+02:00 issue-templates-cloud app/web.1 [34mdebug[39m: GET 200 {"key":"field","value":"customfield_12535","self":"https://OURCUSTOMER.atlassian.net/rest/atlassian-connect/1/addons/com.intenso.jira.issue-templates/properties/field"}

Maciej Dudziak [Deviniti]
August 19, 2019, 2:08 PM

App using ACE (com.intenso.jira.issue-templates):

put {"url":"/rest/api/2/issue/265962?notifyUsers=true&overrideScreenSecurity=true","body":"{\"fields\":{\"summary\":\"Revoke :: CCC CCC :: Zoom Account\"},\"update\":{}}","headers":{"Accept":"application/json","Content-Type":"application/json","x-atlassian-force-account-id":true}}

response:

PUT 400 {"errorMessages":["Exception occurred: org.ofbiz.core.entity.GenericDataSourceException: Unable to establish a connection with the database. (PSQL_TOO_MANY_CONNECTIONS Exception already occurred in this workcontext, skipping next getConnection)"],"errors":{}}

Maciej Dudziak [Deviniti]
August 7, 2019, 10:14 AM
Edited

App using ACE (com.intenso.jira.issue-templates):

request authenticated as addon:

put {"url":"/rest/api/2/issue/264152?notifyUsers=true&overrideScreenSecurity=true","body":"{\"fields\":{\"summary\":\"Summary\"},\"update\":{}}","headers":{"Accept":"application/json","Content-Type":"application/json","x-atlassian-force-account-id":true}}

App updates 33 subtasks in the same project under the same parent, all request looks the same way except id

28 are successful, 4 of them fails like this:

PUT 403 <!doctype html><html lang="en"><head><title>HTTP Status 403 – Forbidden</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 403 – Forbidden</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The server understood the request but refuses to authorize it.</p><hr class="line" /><h3>Apache Tomcat/9.0.14</h3></body></html>

1 fails like:

PUT 400 {"errorMessages":["Exception occurred: org.ofbiz.core.entity.GenericTransactionException: Error occurred while starting transaction. (PSQL_TOO_MANY_CONNECTIONS Exception already occurred in this workcontext, skipping next getConnection)"],"errors":{}}

Maciej Dudziak [Deviniti]
August 6, 2019, 9:45 AM
Edited

GET /rest/api/2/issue/createmeta

{"message":"org.postgresql.util.PSQLException: FATAL: too many connections for role \"GL62fZE7FxApdF2RarwDJW\"","status-code":500,"stack-trace":"com.atlassian.jira.exception.DataAccessException: org.postgresql.util.PSQLException: FATAL: too many connections for role \"GL62fZE7FxApdF2RarwDJW\"

App key: com.intenso.jira.plugins.jsd-extender (Cloud)

 

Assignee

Unassigned

Reporter

Maciej Dudziak [Deviniti]