At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the sporadic disruptions in service on April 22th through May 5th, 2023 described below.
Tickets Included in this RCA: 222223, 222387, 223043
April 22, 1:57 PM EST (Ticket #222223)
The client notified Bastian Support through an e-mail ticket that Lanes 3 and 9 were not loading at the shipping sorter.
April 22, 2:06 PM EST
The client called in inquiring as to the status of the issue with Lanes 3 and 9, but there was no resolution at this time.
April 22, 3:42 PM EST
The client called in to have Bastian Support join a Teams bridge while the issue was being examined. During this bridge, Bastian Support determined that the lanes were not receiving divert confirmations.
April 22, 5:14 PM EST
Bastian Support restarted the AOR software, and divert confirmations began being received by Exacta for Lane 9, but not for Lane 3.
April 22, 5:43 PM EST
Bastian Support restarting the AOR software, and re-initialized the Kepware tags, but were still not receiving divert confirmations for Lane 3. At this point, Bastian Controls was engaged.
April 22, 6:31 PM EST
Bastian Controls identified that five boxes did not receive a divert confirmation because the gap between the boxes was too small for the divert to be performed, and Lane 3 was closed when one of the example boxes had gone through. Bastian Controls also noticed that there was a stuck tag, which was additionally preventing the divert confirmations for the lane from being sent. Bastian Controls cleared the data in the tag from the PLC, and this resolved the issue for the day.
April 24, 9:35 PM EST (Ticket #222387)
The client notified Bastian Support through an e-mail ticket that the ship sorter chutes 3, 5, and 7 were diverting to the correct location, but not loading.
April 24, 10:55 PM EST
Bastian Support identified a database failover issue was present with the production database. The database was failed back over to the correct server nodes. Afterwards, the AOR software, all Exacta services, and IIS (Portal) were restarted.
April 24, 11:30 PM EST
The client conducted some limiting testing to determine if the issue was resolved, but encountered a no-read streak during testing. Production was complete for the night at this time. The client was advised to call back in the next day if they continued to experience issues with the chutes not loading, as it was unclear if the issue was resolved at that time.
April 25, 8:43 AM EST
The client sent an e-mail in indicating they were not receiving divert confirmations for Lane 5, but had not reported any issues with Lanes 3 and 7.
April 25, 9:14 AM EST
The client called in to engage Bastian Support regarding the issue with Lane 5 not receiving divert confirmations. Bastian Support was having difficulty connecting to the site’s AOR computer at the time.
April 25, 12:00 PM EST
Bastian Support was able to get connected to the site and worked on the issue during a Teams bridge. Bastian Support observed that Lane 5 was disabled. The client indicated that Lane 5 had been disabled because it was not receiving divert confirmations. The client was asked to call back in to reach the controls department since the support agent was currently active on the bridge and could not transfer from the bridge.
April 25, 1:00 PM EST
Bastian Controls verified with the client that they had closed Lane 5 because they were not receiving divert confirmations for the lane. Bastian Controls had the client open Lane 5, and observed the PLC during the time. Bastian Controls validated all items were receiving divert confirmations with no additional action having been taken.
May 2, 4:39 PM EST (Ticket #223043)
The client sent an e-mail indicating that Ship Sorter Lane 3 is diverting, but not loading the packages.
May 2, 4:58 PM EST
The client called in to have Bastian Support join a Teams bridge regarding the issue with Lane 3.
May 2, 6:05 PM EST
Bastian Support restarted the AOR computer, which resolved the issue with Lane 3.
May 2, 7:15 PM EST
Bastian Support continued examining the logs, and identified that the AOR was sending divert requests to controls for Lane 3 for the latest incident, but that divert confirmations were not being received back from controls.
May 3, 2:00 AM EST
Bastian Support continued gathering information and created a work item for development to review. Additionally, Bastian Support contacted a member of Bastian Controls that had previously worked on the issue to gather more information, and notified other members of Bastian Support to engage Bastian Controls if the issue occurred again prior to performing any resolution steps, such as rebooting the AOR computer, in order for Bastian Controls to observe the issue as it occurred.
May 5, 10:33 AM EST
The client called Bastian Support due to an issue with Lane 9 not receiving divert confirmations. Bastian Support engaged Bastian Controls at this time.
May 5, 11:47 AM EST
Bastian Controls identified data stuck in the divert confirmation tag for Lane 9. At this time, Bastian Controls added a timer to the divert confirmation tags for each shipping lane to automatically clear the data from the divert confirmation tags if the data for the divert was still present in the tag one second after the divert was to be performed.
May 8, 9:00 AM EST
Bastian Controls added the updated controls logic for divert confirmations to the Belleville site. No issues had yet been reported at the Belleville location.
The divert confirmation tags for the respective lanes were occasionally not being automatically cleared from the PLC when the divert confirmations were sent back to the AOR. The data remaining in the divert confirmation tag would prevent the tag from correctly updating when new divert confirmations were requested and would subsequently prevent further divert confirmations from being sent.
Bastian Controls added a timer to the PLC logic automatically clear any data remaining in the shipping lane divert confirmation tags one second after the divert confirmation was sent at both the Dubuque and Belleville sites. The Belleville site had not yet reported any issues, so the timer was added as a precautionary measure to that site.
Next Steps and Preventive Actions:
Occasionally, there may be rare occurrences of network communication issues and Kepware glitches that can disrupt the handshake between Controls and Software. To mitigate this potential problem, Bastian Controls implemented a timer in the PLC logic. This timer serves as a proactive measure to automatically clear any remaining data in the shipping lane divert confirmation tags, ensuring smooth operations and preventing any issues. Additionally, Bastian Support will engage Bastian Controls if further issues are observed regarding the divert confirmations.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.