Knowledgebase Article Number
Published April 1, 2020, at approximately 11 a.m. EST.
|At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on March 23rd until March 25th, 2020 described below.
Brendan O'Neill, Controls Support Engineer III, received a call from Saul who was not onsite at Amazon Mexico about totes diverting to the wrong lanes. All the events took place between 3/23-3/25/2020. Below are the steps that were taken by Brendan while working on the problem.
March 23rd 2020
Received the initial call from Saul at this time. Saul was not onsite. Saul worked with the site team to get a code for Bastian to get connected to the PLC. The report is that all product was going to the destination of the previous container.
It should be noted, that over the weekend a call came in about product not diverting properly. Joe Doyle worked that call, and an encoder was replaced and seemed to correct the issue.
While waiting connect, Saul got word that the product was now properly diverting, but nothing was changed, and no troubleshooting had began.
At this point, Bastian connected and quickly checked the HMI for read rate of any faults but did not see any. Ticket #140964 shows the numbers as well as the conveyor status.
After checking stats we moved on to identifying the conveyors and began researching the Destinations contained in the PLC via the Destination Array.
There was logic present to count the pulses for the physical handoff from conveyor SRT1_8 onto the Sorter. Ran this test and saw the handoff was typically happening at 65 pulses. The moving of data into the handoff register occurred at the 63 pulse mark, meaning there typically was only 2 pulses to move the data, then have the tote clear the handoff eye, which is too tight of a window to accommodate this activity.
Did 2 things at this point. Moved the handoff load location to 61 pulses, and created a FIFO to load data and then unload it, just in case there were instances where the data was loading twice before clearing the eye (this later proved to not be the case)
Joined the Amazon bridge call with the Amazon Team, and gave them a rundown of what may be causing this, while continuing to troubleshoot. Things continued to run well throughout the call, and we disconnected about 30 minutes later.
Noticed 2 Encoder consistency errors, though they did not cause any fault that would stop the conveyors. It seems the tests were never dialed in. Set those tests up properly, so if that fault returns it should be a genuine concern.
March 24th 2020
Got a ticket update from Saul that the issue had returned. Later found out, according to site, it had been happening for 90 minutes before support was contacted. Requested another code to connect, and got online.
After dealing with some required PC restarts based on one time required updates, was finally able to get connected.
Cleared the FIFO and added logic to auto clear, should this happen again. This way the product would go to the proper destination. Note* This counter seemed to hold constant and does not appear to be a regular issue
At this time, moved the handoff Load position up two more pulses to 59, about 6/7 pulses before getting moved to the sorter array. We have not seen the issue reoccur after making this change.
March 25th 2020
The rest of the week connected and monitored the area, without a single instance of this reoccurring.
Next Steps and Preventive Actions:
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent re-occurrence.