Knowledgebase Article Number

4415355553431

Description

November 29th, 2021, at approximately 7:00 p.m. EST. 

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on November 29th, 2021 described below.

Summary:

Monday, November 29th

7:00 PM EST
Calab Honeycutt called the Bastian support line. He said that the sorter wasn't running and had a T-manager fault. I jumped online to see what the issue was. I could see one of the T-manager AOI's had a fault. The AOI was GetData_Scanner2. First I tried just resetting the fault but it would not clear. This is the same thing that the customer experienced. While trying to see what needed to be done the customer insisted we got on a bridge call while my investigating took some time. This was advised 30 min after looking into the issue. 

7:30 PM EST

I joined the bridge call but Calab did not. There were 3-4 managers on the call looking for updates but I was still actively working the issue. The block was throwing fault was throwing fault code 102 for stsAdapterFault. But the Adapter was ok. The block was still triggered by GetData_Scanner2_cmdTrigger so I cleared that and then the block would reset. Upon resetting the block we ran some packages but they still were not getting correct destinations. I went to look if we were getting good scans in the PLC and notice so odd data. A normal scan was just numbers ex: 520872277, but this data was K&1 W+4Y003$'. I quickly cleared this data out and cleared out the FIFO and made sure the T-manager was good. After clearing everything out we did start getting good destinations again. 

8:00 PM EST
Trying to get the sorter back working the packages that didn't get diverted went to recirculation. This overloaded recirculation because it took some time to get things moving and diverting correctly. The curve at the outfeed of the sorter was found to be running backwards. The direction is not controlled by the PLC and is just a DIP switch setting on the card. We tried changing it to see if the conveyor would change direction and it did but there were about 30 cards affected. One of the managers suggested not to change the DIP switches and this gave time for the maintenance guy to look at the cards. 

8:30 PM EST
Two of the cards at the exit of the sorter had a fault. The maintenance guy was able to find some replacements and upon replacing the 2 bad cards and a power cycle the line ran the correct way. This looks to have been a separate issue that arose from overloading the line. Everything looks to have been working around 8:45PM. 

Root Cause:

There were 2 issues that occurred but were separate issues. The first issue with the T-manager looks like it occurred because of corrupt data from the scanner. The 2nd issue with the MDR cards running backwards looks like was caused by overloaded cards and caused 2 cards to fault, which caused the reset of the cards to run the wrong direction. 

Resolution:

Issue 1: Resolution was to clear out the corrupt data and the FIFO. 

Issue 2: Replace the bad MDR cards

Next Steps and Preventive Actions:

Issue 1: I would like to add a check for corrupt data before moving this data to the T-manager block. This check will look for only numbers. If there is data in there that is not a number it will be registered as a no-read. 

Issue 2: For the bad MDR cards we just need to make sure there are enough spare parts to cover in case this happens again. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.