@martins.viksna:
Post mortem - Edit sync variant incident on Aug 3
Incident summary
Leadup
I added The Feature Flag fix as a part of different issue. Unfortunately, I didn’t understand correctly the scope of this minor change and that lead to insufficient tests to catch the issue.
Fault
The issue was that there was a flow that called “inside label color group” Feature Flag before generator and vuex module was initialised. (All flags for designmaker are called there in a bundle and then stored in vuex)
Impact
Affected less than 60 users according to sentry.
Detection
The incident was reported to our dev team on Aug 3, 2022, 14:18 GMT+0200. We immediately looked into it and soon found a cause.
Recovery
Fastest way to fix the issue was to revert the changes. Because the previous variant, when not finding cached Feature Flag, made a request to get it from DB. After that, the fix was confirmed locally. deployed to production and confirmed in the production.
Timeline
Aug 3, 2022, 12:47 GMT+0200 fix for the inside label native templates was deployed.
Aug 3, 2022, 14:18 GMT+0200 the Account Specialist reported the issue
Aug 3, 2022, 14:47 GMT+0200 issue was fixed in the production
Lessons learned
Corrective actions