Published August 24th, 2022 at approximately 5 p.m. EST. 
Last updated on August 24th, 2022.

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on August 18th and 19th, 2022 described below.
 

Summary:

12:58 PM EST (8/18/2022)

Software Support received a call stating that Lane 13 was disabled in AOR. We quickly engaged Controls Support to have them check PLC to make sure everything was online on their side. 

1:23 PM EST (8/18/2022)

At the request of Controls Support Engineer, the call was transferred to our Controls team to check everything on their end. 

2:05 PM EST (8/18/2022)

The call was transferred back to Software Support to resume troubleshooting. In coordination with Controls, Lane 13 was enabled. It was discovered that the lane was disabling based on the the lane full tags. 

2:20 PM EST (8/18/2022)

The site reported seeing cartons with an expected divert code of 91 going to Jackpot and reported that orders were not being manifested.

2:40 PM EST (8/18/2022)

Reviewing our application logs and the examples provided, Software Support saw that logs were not missing a divert code. Reviewed previous incidents to confirm if this was problem with Exacta or Manhatten.

3:10 PM EST (8/18/2022)

Site began confirming with Manhattan that they were sending divert codes. Manhatten was sending divert code of 91.

3:40 PM EST (8/18/2022)

Bastian researched Import logs.Restarted services and ran more tests. Reviewed logs to see if we were recieving proper information to begin manifesting. 

4:15 PM EST (8/18/2022)

Bastian Support continued researching logs against the examples that were given to us from the site to examples prior to the issue. 

4:40 PM EST (8/18/2022)

Bastian software support agents, after notes from the site in regards to missing weight values, collaborate statement with AOR logs and ask Bastain controls team to investigate.

5:10 PM EST (8/18/2022)

Bastian controls team verify there is not an issue with the scale on their end. Software support begin checking for valid communication between controls and software.
5:30 PM EST (8/18/2022)

Bastian software transition issue over to second shift, and get team up to speed.

5:45 PM EST (8/18/2022)

Site mentions that they now have issues with induction and also mention there had been changes made to a few scanners. Support begins looking into induction while collecting more detail soon the scanner configurations adjustments made.

5:55 PM EST (8/18/2022)

Support finds issues in induction logs that point to issues with services on the AOR machine.

6:10 PM EST (8/18/2022)

Support confirms issues with induction from the logs and finds related service and configuration that is causing induction issues. Configuration had been updated immediately before issues began on 8/18. After comparing it to an older copy of the configuration, an update was made to the configuration.

6:25 PM EST (8/18/2022)

The Specialist asks for Rene to test after configuration changes were made.

6:45 PM EST (8/18/2022)

Rene confirms with the Specialist that the issue is resolved. 
8:45 AM EST (8/19/2022)

Bastian Solutions Controls Support team engaged Software Support to assist in troubleshooting a diverting issue within shipping sorter AOR.

9:00 AM EST (8/19/2022)
The Software Support Specialist reached out to Rene Campos to assist with the ongoing issues.

9:10 AM EST (8/19/2022)
The Software Support Specialist determined with Rene that containers were supposed to be diverting to lane 13, but it was showing as currently disabled on the AOR. The issue the site was seeing was that when Lane 13 was either disabled or full, containers would divert to the next available lane rather than recirculating. 

9:20 AM EST (8/19/2022)
The Controls Support team enabled lane 13 via the HMI, and the site reported that all containers were diverting correctly.

9:25 AM EST (8/19/2022)
The Software Support Specialist and Controls Engineer began looking into possible root causes and how to prevent this issue moving forward.

9:58 AM EST (8/19/2022)

The Controls Engineer implemented a temporary fix to prevent lane 13 from disabling.

9:59 AM EST (8/19/2022)

The Software Support Specialist implemented changes to set Lane 13 to be set to the error lane per Rene's request.

10:50 AM EST (8/19/2022)

The Software Support Specialist updated Lane 13 to the error lane/restarted the AOR and the Controls Engineer reverted their temporary changes. Site confirmed that the changes were successful.

1:31 PM EST (8/19/2022)
Rene reached back out to the Software Support reporting that containers were again diverting to incorrect lanes whenever the new error lane is full rather than recirculating. Operations cleared the line to free up space in lane 13 and containers began to divert correctly again.

2:30 PM EST (8/19/2022)
The Specialist consulted with Senior resources as they continued their investigation as to what was causing containers to divert to incorrect lanes.

3:18 PM EST (8/19/2022)
The Software Support Specialist escalated the issue to the Director of Software Development to have a developer assigned to this issue.

3:30 PM EST (8/19/2022)
The Developer began assisting Bastian support agent in finding a root cause/permanent fix.

4:00 PM EST (8/19/2022)
The developer found the issue and proposed that the Shipping sorter scanner record within the ExactaDB and scanner_routing_map table be updated to 99(recirculate) from 13(Lane 13).

4:30 PM EST (8/19/2022)
The Specialist made the configuration change in the database and restarted the AOR.

4:44 PM EST (8/19/2022)

Rene and the operations team confirmed that the database update worked and containers were recirculating even when lane 13 became full.

Root Cause:

For the incident reported on 8/18/2022, the root cause was determined to be a configuration file that had been updated on the AOR workstation shortly before the initial call to support. The configuration was updated incorrectly, removing a tag, which caused AOR to enter a faulted state. We confirmed that the configuration had been opened and reviewed by a member of the project team but was unaware that the file had been modified when closing it.

The root cause for issues experienced on August 19th was determined to be a misconfiguration within the database. Containers that went past the shipping sorter scanner were programmed to go to lane 13 if they were deemed in need of quality control. This logic was fine until lane 13 became full, then containers would divert to the next available lane rather than recirculating.

Resolution:

Resolution for the issues experienced on August 18th was to correct the configuration file and restore the deleted tags. After correcting the configuration and restarting the AOR software, we received confirmation from the site that the issue was resolved. 

Resolution for the continued issues experienced on August 19th was a change that was made in relation to the shipping sorter scanner and lane 13, this scanner was programmed to route containers to Lane 13, but once lane 13 was full it would route containers to the next available lane(Resulting in the issues we saw on the 18th). The two changes we made occurred in the database. First, Lane 13 was set as the Quality Control Lane. Then the shipping sorter scanner was configured to have containers recirculate when lane 13 was full rather than having them divert to a separate lane, per Rene's request.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.