Action Required - Youreka Service Interruption - March 2024

Hello Youreka Customers,

On Mar 11, 2024 at 4:44 pm eastern time, Youreka experienced an interruption to our license provisioning system. As a result your PRODUCTION INSTANCE along with your sandbox instances MAY HAVE BEEN IMPACTED. Please assume your production instance has been impacted and take the steps below immediately.

Impact:

• Youreka access is expired - all users will not be able to access the application.
• Templates that were ‘Published/In Use’ have been set to an inactive status.

Solution:

Part 1 - Addressing Expired Licenses

1. Log in to your Salesforce instance as a System Administrator.
2. Navigate to the Youreka Licenses tab.
3. Click the ‘Refresh License Information’ button on the top right of the page.
4. Confirm that the ‘Premium Expiration Date’ in the top center of the page is in the future.
5. Inform any impacted mobile end users to close and re-open the Youreka mobile application after you’ve completed all the steps in this section and the steps to Re-enable templates. End users should not log out or uninstall the application.

*If you are having trouble, or if you need help please contact your CSM and we will provide a technical resource to assist you.

Part 2 - Re-enabling templates

1. Navigate to the Form Templates tab as a user with a Youreka Template Builder license
2. Create a Form Template list view with the following filters
   1. filter by owner - all form templates
   2. last modified date - greater or equal YESTERDAY (March 11, 2024)
   3. status - equals inactive
3. Select fields to display
   1. form template name
   2. status
   3. last modified
4. Click the pencil icon to edit a status (do this for each template you wish to make available, one at a time)
   1. select “In use”
   2. click “save”

*If you are having trouble, or if you need help please contact your CSM and we will provide a technical resource to assist you.

Once you have ensured all desired templates have been set to a status of ‘In Use’ you should have fully recovered from this interruption. At this point you can repeat these steps in your sandbox environments. Please note, not all instances were impacted. You may discover that none, one, many or all of your instances were impacted so please check your production instance first, followed by all of your active sandbox instances.

We are extremely sorry for this interruption of service. As of 8:20 am eastern time this morning (Mar 12, 2024) we have remedied the issue that caused this interruption. We will follow up with additional information shortly. We are prioritizing re-establishing service as our highest priority.

------------

Youreka Service Interruption Update - March 12, 2024

Introduction

Unfortunately many Youreka customers experienced production and/or sandbox service interruptions today, March 12, 2024, for several hours due to a bug introduced to Youreka’s license provisioning system on March 11, 2024.

Once aware of the issue, we opted to address it in two phases. Phase one - contain the impact by correcting the defect and restoring access for all customers. Phase two - identify the root cause, remediation plan, and communicate details to our customers and partners.

What happened

It is unacceptable that we introduced the underlying bug and have conducted an internal post-mortem session to uncover the flaw in our deployment process that made the introduction of the bug possible. A change was made to how we verify licenses in the Youreka license provisioning system. Unfortunately, a quality assurance step was missed prior to the deployment on March 11, 2024 at 4:44 PM ET. We have remedied the flaw to ensure this type of issue does not repeat.

Timeline (all times in eastern time)

March 11, 2024

• 4:44 pm - a bug was introduced to Youreka’s internal license provisioning system
  Any Youreka customer orgs (sandbox or production) that completed an automated daily license check after 4:44 pm may have received inaccurate license expiration data causing the org to deactivate both licensing and published form templates
• 10:00 pm - Youreka received the first case related to the issue, however, we did not have enough information at this time to assess the potential severity or impact

March 12, 2024

• 3:31 am - Youreka received the second case related to the issue and began to treat the occurrence as a Severity 1 issue with potential to impact all customers
• 7:25 am - Root cause identified by Youreka engineering team
• 8:25 am - Resolution deployed to Youreka’s license provisioning system and began to craft customer communications
• 9:10 am - Youreka Customer Success Managers began to broadcast messaging, impact, and resolution steps to all customers
• 9:30 am - Mass email communicating the issue, impact, and resolution steps to all customers. Customer and Technical Success Managers began to provide virtual assistance to customers and partners

Impact

On Mar 11, 2024 at 4:44 pm eastern time, Youreka introduced a defect to our license provisioning system. Any Youreka customer orgs (sandbox or production) that completed an automated daily license check after 4:44 pm may have received inaccurate license expiration data causing the org to deactivate both licensing and published form templates. We advised customers and partners to assume your production and sandbox instances have been impacted and take appropriate remediation steps immediately.

Impact to affected orgs:

• Youreka access is expired - all users will not be able to access the application.
• All Templates that were ‘Published/In Use’ have been set to an inactive status.

Remediation steps distributed earlier today are here for your reference.

Conclusion

We are extremely sorry for the interruption in service and the inconvenience this has created for you, your organization, and your end users. Your trust is of the utmost importance to us and we take the responsibility of delivering stable and trustworthy solutions very seriously. As a result of this unacceptable situation, we must and will hold ourselves to a higher standard moving forward. Thank you for your patience and understanding. We welcome your feedback and are happy to answer any questions you may have.

Sincerely,

Dan Bergner
Founder, Chief Product Officer