Published August 24th, 2022 at approximately 5 p.m. EST.
At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on August 18th and 19th, 2022 described below.
12:58 PM EST (8/18/2022)
Software Support received a call stating that Lane 13 was disabled in AOR. We quickly engaged Controls Support to have them check PLC to make sure everything was online on their side.
1:23 PM EST (8/18/2022)
At the request of Controls Support Engineer, the call was transferred to our Controls team to check everything on their end.
2:05 PM EST (8/18/2022)
The call was transferred back to Software Support to resume troubleshooting. In coordination with Controls, Lane 13 was enabled. It was discovered that the lane was disabling based on the the lane full tags.
2:20 PM EST (8/18/2022)
The site reported seeing cartons with an expected divert code of 91 going to Jackpot and reported that orders were not being manifested.
2:40 PM EST (8/18/2022)
Reviewing our application logs and the examples provided, Software Support saw that logs were not missing a divert code. Reviewed previous incidents to confirm if this was problem with Exacta or Manhatten.
3:10 PM EST (8/18/2022)
Site began confirming with Manhattan that they were sending divert codes. Manhatten was sending divert code of 91.
3:40 PM EST (8/18/2022)
Bastian researched Import logs.Restarted services and ran more tests. Reviewed logs to see if we were recieving proper information to begin manifesting.
4:15 PM EST (8/18/2022)
Bastian Support continued researching logs against the examples that were given to us from the site to examples prior to the issue.
4:40 PM EST (8/18/2022)
Bastian software support agents, after notes from the site in regards to missing weight values, collaborate statement with AOR logs and ask Bastain controls team to investigate.
5:10 PM EST (8/18/2022)
Bastian controls team verify there is not an issue with the scale on their end. Software support begin checking for valid communication between controls and software.
Bastian software transition issue over to second shift, and get team up to speed.
5:45 PM EST (8/18/2022)
Site mentions that they now have issues with induction and also mention there had been changes made to a few scanners. Support begins looking into induction while collecting more detail soon the scanner configurations adjustments made.
5:55 PM EST (8/18/2022)
Support finds issues in induction logs that point to issues with services on the AOR machine.
6:10 PM EST (8/18/2022)
Support confirms issues with induction from the logs and finds related service and configuration that is causing induction issues. Configuration had been updated immediately before issues began on 8/18. After comparing it to an older copy of the configuration, an update was made to the configuration.
6:25 PM EST (8/18/2022)
The Specialist asks for Rene to test after configuration changes were made.
6:45 PM EST (8/18/2022)
Rene confirms with the Specialist that the issue is resolved.
Bastian Solutions Controls Support team engaged Software Support to assist in troubleshooting a diverting issue within shipping sorter AOR.
9:00 AM EST (8/19/2022)
9:10 AM EST (8/19/2022)
9:20 AM EST (8/19/2022)
9:25 AM EST (8/19/2022)
9:58 AM EST (8/19/2022)
The Controls Engineer implemented a temporary fix to prevent lane 13 from disabling.
9:59 AM EST (8/19/2022)
The Software Support Specialist implemented changes to set Lane 13 to be set to the error lane per Rene's request.
10:50 AM EST (8/19/2022)
The Software Support Specialist updated Lane 13 to the error lane/restarted the AOR and the Controls Engineer reverted their temporary changes. Site confirmed that the changes were successful.
1:31 PM EST (8/19/2022)
2:30 PM EST (8/19/2022)
3:18 PM EST (8/19/2022)
3:30 PM EST (8/19/2022)
4:00 PM EST (8/19/2022)
4:30 PM EST (8/19/2022)
4:44 PM EST (8/19/2022)
Rene and the operations team confirmed that the database update worked and containers were recirculating even when lane 13 became full.
For the incident reported on 8/18/2022, the root cause was determined to be a configuration file that had been updated on the AOR workstation shortly before the initial call to support. The configuration was updated incorrectly, removing a tag, which caused AOR to enter a faulted state. We confirmed that the configuration had been opened and reviewed by a member of the project team but was unaware that the file had been modified when closing it.
The root cause for issues experienced on August 19th was determined to be a misconfiguration within the database. Containers that went past the shipping sorter scanner were programmed to go to lane 13 if they were deemed in need of quality control. This logic was fine until lane 13 became full, then containers would divert to the next available lane rather than recirculating.
Resolution for the issues experienced on August 18th was to correct the configuration file and restore the deleted tags. After correcting the configuration and restarting the AOR software, we received confirmation from the site that the issue was resolved.
Resolution for the continued issues experienced on August 19th was a change that was made in relation to the shipping sorter scanner and lane 13, this scanner was programmed to route containers to Lane 13, but once lane 13 was full it would route containers to the next available lane(Resulting in the issues we saw on the 18th). The two changes we made occurred in the database. First, Lane 13 was set as the Quality Control Lane. Then the shipping sorter scanner was configured to have containers recirculate when lane 13 was full rather than having them divert to a separate lane, per Rene's request.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.