Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on April 19th, 2023 described below.

Root Cause:

Based on our investigation the reason for the failure was the repeated failures to uninstall the Exacta Application. While the uninstall was successful, part of the application remained resulting in repeated installation failures. As part of the manual clean-up process, the import/export services were removed and are not automatically reinstalled as part of the installer process. 

Action Items:

As for the extended downtime caused by the issues with rollback, Bastian has identified multiple opportunities for improvement in our process in this case.

1. Bastian will force all promotes to be installed and tested within the customer's dev environment. This will help minimize issues impacting production. 

2. Bastian Leadership will develop and share with team members a guide and checklist and general troubleshooting guide to be used with installers. 

3. Bastian to provide additional training to team members on the installation process and ways we can quickly roll back if issues arise. 

4. Improve the installer process to not overwrite installation logs but append to them so we don't lose history from previous failures. 

Timeline of Events:

Apr 19, 2023 3:04 PM
Curtis calls Metro Supply and leaves a message letting the site know that the promote is staged and will be installed during the scheduled downtime between 3 and 4 pm. 

Apr 19, 2023 3:00-3:40 PM
Curtis attempted to uninstall the core Exacta functions in the ExactaAppServer folder per promote instructions. The uninstall was unsuccessful due to services starting after being stopped. Repeated the uninstallation process multiple times. Once the uninstall was successful, Curtis began the installation of the promote. Installation was unsuccessful.  He validated install parameters had been set correctly. Curtis attempted to reinstall the promote a couple of times with the same result. Curtis contacted the Bastian QA and Dev team for a consultation at 3:40 pm.

Apr 19, 2023 3:40-4:30 PM

Bastian QA team recommended a more complete uninstall of Exacta services, and a re-run of the initial installer. However, these could not be uninstalled using the Add/Remove Programs tool in Windows, due to the incomplete state of the install attempt. The team then attempted to roll the promote back using backup files, but they still could not run services.

Apr 19, 2023 4:30-5:15 PM

After consulting with the QA resources again, Curtis manually removed the remaining services. Out of concern for the impact the extended downtime was causing, Curtis elected to run the last known working installer, which ran successfully. 

Curtis then started services, which also ran successfully with the exception of ExactaLightController. Additional troubleshooting of LightController took around 15 minutes. At 5:15 the system appeared to be up and running normally, with the exception of the Exacta Connect API. However, this was not noticed by the Bastian team at the time, as the standard Export/Import services were present.

Apr 19, 2023 5:30-7:00 PM
Metro Supply observed ongoing issues with import and export of data to Exacta and reached out for escalation with Bastian management. Management engaged the original project manager, Kyle Grant, who installed the correct import/export service package and restored full functionality to the system. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective to provide our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.