At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on Thursday, January 21st, 2021 described below.

Summary:

Thursday, January 21th

8:11 am

Inmar called into support state that boxes were not dropping and that the AOR was sending constant destinations of 0. Support agent began initial investigation

8:30 am

After reviewing the AOR and the AOR logs, the support agent, along with Inmar decided to restart the AOR. Once this was complete, boxes began to receive a few destinations, but the problem resurfaced shortly after.

8:35 am

The support agent began to review older tickets that dealt with the error messages below:

Line 26240: 07:45:53.653 [34] (null) ERROR Bastian.Exacta.Inmar.LaneAssignment.InmarLaneAssignment - Did not find information for ?????. Request for ?????. XTray False. Recirc False. Side A..

07:46:10.481 [1] (null) INFO  Equipment 1 - 07:46:10:481 - Could not find lane [00000000-0000-0000-0000-000000000000] - 07:46:10:421

In a previous instance the Bastian support agent reached out to euro sort and then never heard back from Inmar or euro sort further on the issue.

8:59 am

The Bastian support agent reached out to the euro sort hotline. Euro sort state that they would loop in their software team and give Hiram a call back.

9:03 am

The Bastian support agent emailed Hiram letting him know that euro sort would be reaching out to him, and that if he did not hear from Hiram or euro sort he would reach back out to check to see if euro sort had called Inmar.

9:24 am

After not hearing from Inmar or euro sort, Bastian reached back out to Inmar and they stated that euro sort said they were successfully sending us destinations and that the problem lied within Bastian's software.

The Bastian support agent then escalated internally.

9:32 am

After confirming that we were not receiving destinations with Bastian's escalation resource, the Bastian support agent reached back out to euro sort to see if they could look into the issue again.

9:40 am

Euro sort stated that everything looked fine on their end other than when they send us an induction message, they stated that they did not receive a response back.

10:00 am

The Bastian support agent reached out internally to the Bastian controls team to help determine why we were receiving no reads. The three agents then reviewed the logs and investigated together. 

10:29 am

The Bastian support agent, his escalation resource, and the Bastian controls agent decided to reboot the AOR and the AOR PC with permission from Inmar. This did not make any changes. 

10:33 am

Bastian joined a call with Inmar and euro sort to troubleshoot further with the end result being euro sort restarting their services and the sorter. 

10:49 am

Euro sort decided to restart the sorter and their services. The sorter started back up fine, but there was a bit of delay for the euro sort services to restart. "USS Sorter Service" was the specific service restarted on the euro sort side.

11:24 am

After the sorter and services had been restarted on the euro sort side, the Inmar team confirmed that everything was working properly again. No services were restarted on the Bastian side other than the AOR around 8:30 A.M. EST.

Root Cause:

Network Disturbance (Either communication from EuroSort to AOR or AOR to Eurosort) Need to continue to monitor for repeated issues

Resolution:

Restarted both AOR and EuroSort "USS Sorter Service" to get connect established

Next Steps and Preventive Actions:

EuroSort and Bastian developed a plan to not restart services until both parties are engaged and investigating. Once both sides are logged in we will pull logs and investigate. Once information is pulled we will get site processing and investigate log files.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.