@martins.viksna:

Post mortem - Edit sync variant incident on Aug 3

Incident summary

• Between 12:47 and 14:47 users could not edit sync variants. Modal just didn’t open.
• The event was triggered by a minor fix for Feature Flag to not load independently from other designmaker Feature Flags
• The bug was caused by trying to access vuex store when it was not yet initialised
• Issue was detected by sentry, and less than 60 users got affected. At 14:18 our dev team received the first report from CS team. I immediately knew where the possible issue could be and reverted the changes.

Leadup

I added The Feature Flag fix as a part of different issue. Unfortunately, I didn’t understand correctly the scope of this minor change and that lead to insufficient tests to catch the issue.

Fault

The issue was that there was a flow that called “inside label color group” Feature Flag before generator and vuex module was initialised. (All flags for designmaker are called there in a bundle and then stored in vuex)

Impact

Affected less than 60 users according to sentry.

Detection

The incident was reported to our dev team on Aug 3, 2022, 14:18 GMT+0200. We immediately looked into it and soon found a cause.

Recovery

Fastest way to fix the issue was to revert the changes. Because the previous variant, when not finding cached Feature Flag, made a request to get it from DB. After that, the fix was confirmed locally. deployed to production and confirmed in the production.

Timeline
Aug 3, 2022, 12:47 GMT+0200 fix for the inside label native templates was deployed.
Aug 3, 2022, 14:18 GMT+0200 the Account Specialist reported the issue
Aug 3, 2022, 14:47 GMT+0200 issue was fixed in the production

Lessons learned

• The scope of the changes was bigger than expected
• Always look at sentry after deploying a fix

Corrective actions

• If change might affect more than one flow, that should be noted and tested
• Always keep sentry open after a bug fix