Sync variants couldn't be edited for brief time.
Incident Report for Printful


Post mortem - Edit sync variant incident on Aug 3

Incident summary

  • Between 12:47 and 14:47 users could not edit sync variants. Modal just didn’t open.
  • The event was triggered by a minor fix for Feature Flag to not load independently from other designmaker Feature Flags
  • The bug was caused by trying to access vuex store when it was not yet initialised
  • Issue was detected by sentry, and less than 60 users got affected. At 14:18 our dev team received the first report from CS team. I immediately knew where the possible issue could be and reverted the changes.


I added The Feature Flag fix as a part of different issue. Unfortunately, I didn’t understand correctly the scope of this minor change and that lead to insufficient tests to catch the issue.


The issue was that there was a flow that called “inside label color group” Feature Flag before generator and vuex module was initialised. (All flags for designmaker are called there in a bundle and then stored in vuex)


Affected less than 60 users according to sentry.


The incident was reported to our dev team on Aug 3, 2022, 14:18 GMT+0200. We immediately looked into it and soon found a cause.


Fastest way to fix the issue was to revert the changes. Because the previous variant, when not finding cached Feature Flag, made a request to get it from DB. After that, the fix was confirmed locally. deployed to production and confirmed in the production.

Aug 3, 2022, 12:47 GMT+0200 fix for the inside label native templates was deployed.
Aug 3, 2022, 14:18 GMT+0200 the Account Specialist reported the issue
Aug 3, 2022, 14:47 GMT+0200 issue was fixed in the production

Lessons learned

  • The scope of the changes was bigger than expected
  • Always look at sentry after deploying a fix

Corrective actions

  • If change might affect more than one flow, that should be noted and tested
  • Always keep sentry open after a bug fix
Posted Aug 03, 2022 - 13:39 UTC

We saw an issue where sync variants in store could not be edited, this was a regression from another launch earlier today.
Bugfix for this issue was deployed live in around an hour and sync variant editing in store is currenty working again.
Posted Aug 03, 2022 - 14:00 UTC