This incident has been resolved.

Issue Summary

On 7 Oct, 2021 1530 IST Chatwoot started experiencing very high response times and elevated error rates. The incident lasted till 1842 IST(3 Hours 12 minutes).

We apologize for any impact during this incident. We will be working proactively to prevent such incidents from occurring and improve our emergency response against them.

Root cause and Remediation

This was a minor outage as Chatwoot was available for the whole duration with performance degradation. Our engineers were alerted within a few minutes once the incident began and started working towards a resolution.

The incident started with a misconfigured implementation from a customer who started DDOS'ing Chatwoot servers. This in turn led to bursts of traffic which started queuing up the router and also filling up the database connections. All these had a cascading effect which in turn led to very high response times and timeouts.

By 1730 IST, engineers identified the root cause and started working towards a fix. The customer was informed and we deployed a fix by 1756 IST. Chatwoot app returned to normal limits by 1842 IST.

Prevention and Followup

We deployed an immediate fix to rate-limit the APIs in question. We will be reviewing our existing rate limits against all APIs and ensure we have reasonable limits set to prevent one tenant from bringing the entire app down. We have identified some bottlenecks in our infrastructure and will be upgrading the same.

Also, we will be reviewing our emergency response process and alerting to ensure we are in a position to prevent this from reoccurring.

Resolved: Chatwoot performance has improved in 0a63c3e.

In 32ff0ee, Chatwoot (https://app.chatwoot.com) experienced degraded performance:
- HTTP code: 503
- Response time: 29640 ms