At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on February 21st, 2023 described below.

Summary:

February 21st, Troy Battey called to report that the system was experiencing extreme slowness and bringing operations almost to a stop. It was also noted that they had seen these delays for the past few days and they had attempted to resolve the issue with a reindex of data. 

Steven Wonderley logged in and reviewed the Bastian Delete & Archive + Delta Purge Test as well as the Bastian Delete & Archive + Delta Purge job that was mentioned and found that neither had completed running in the past 6 days. Troy was going to touch base with a college to determine if this A&D was expected.

Troy called back to report that the system had become inoperable. T1 resource attempted to restart the PTL system to see if this would resolve the issue. In the meantime, it was escalated to a T2 resource. T2 resource determined that the Bastian Delete & Archive + Delta Purge Test job had been accidentally configured on 02/14 and was run, interfering with the regularly scheduled Job. This was immediately disabled and the Bastian Delete & Archive + Delta Purge job was canceled. This resolved the issues with the PTL system. 

Root Cause:

The Bastian Delete & Archive + Delta Purge Test job had been incorrectly configured on the live system, when both jobs attempted to run it caused the Bastian Delete & Archive + Delta Purge Test process to fail in a hung state. Once this was stopped operations were back to normal. The Bastian Delete & Archive + Delta Purge Test job has been disabled and the Bastian Delete & Archive + Delta Purge Test process has since run with no further issues. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.