At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on April 19th, 2023 described below.
Based on our investigation the reason for the failure was the repeated failures to uninstall the Exacta Application. While the uninstall was successful, part of the application remained resulting in repeated installation failures. As part of the manual clean-up process, the import/export services were removed and are not automatically reinstalled as part of the installer process.
As for the extended downtime caused by the issues with rollback, Bastian has identified multiple opportunities for improvement in our process in this case.
1. Bastian will force all promotes to be installed and tested within the customer's dev environment. This will help minimize issues impacting production.
2. Bastian Leadership will develop and share with team members a guide and checklist and general troubleshooting guide to be used with installers.
3. Bastian to provide additional training to team members on the installation process and ways we can quickly roll back if issues arise.
4. Improve the installer process to not overwrite installation logs but append to them so we don't lose history from previous failures.
Timeline of Events:
Apr 19, 2023 3:04 PM
Apr 19, 2023 3:00-3:40 PM
Apr 19, 2023 3:40-4:30 PM
Bastian QA team recommended a more complete uninstall of Exacta services, and a re-run of the initial installer. However, these could not be uninstalled using the Add/Remove Programs tool in Windows, due to the incomplete state of the install attempt. The team then attempted to roll the promote back using backup files, but they still could not run services.
Apr 19, 2023 4:30-5:15 PM
After consulting with the QA resources again, Curtis manually removed the remaining services. Out of concern for the impact the extended downtime was causing, Curtis elected to run the last known working installer, which ran successfully.
Curtis then started services, which also ran successfully with the exception of ExactaLightController. Additional troubleshooting of LightController took around 15 minutes. At 5:15 the system appeared to be up and running normally, with the exception of the Exacta Connect API. However, this was not noticed by the Bastian team at the time, as the standard Export/Import services were present.
Apr 19, 2023 5:30-7:00 PM
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective to provide our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.