Knowledgebase Article Number

5603848662679

Description

April 20th, 2022, at approximately 8:00 p.m. EST. 

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on April 20th, 2022 described below. 

The ticket that was generated for this case has been documented as ticket number: 190072

Summary:

Wednesday, April 20, 2022

8:05 PM EST
Parth Joshi called the Bastian support line. He claimed that there was a servo motor fault running. I proceeded with remoting into the Chewy FC10 site system and viewing the application to troubleshoot the issue. After noticing the fault on the PLC application, I proceeded with recommending the exchange of the motor to perhaps see if the issue follows. After the replacement was installed, we struggled with uploading the parameters of the VFD motor, so I was required to inhibit the module and then perform an upload to the VFD drive within the network IO tree. After attempting to start the system once again, the fault continued to persist and would not be removed. 

9:30 PM EST
After attempting to start the system after replacing the motor, I was instructed by Parth Joshi to enter a zoom bridge call to discuss the matter with other Chewy employees. The call included Parth Joshi, Brent Yoder, Dean Bates, Tom Mussar, and other Chewy employees. The meeting started by explaining the background to the issue and what was done thus far. After explaining the issue, Dean Bates, I assume, requested to have additional information regarding a timer preset value that had been modified. I was not involved in the previous discussions/development of this timer and was not aware of any ticket. It is assumed that this timer change/addition was included by the Chewy internal controls team. After discussions of this change, Bastian continued to monitor the issue and realized that there was an input custom fault that was latched on. The engineer that was on call proceeded with the reverse engineering of this fault and located the bit that would be able to unlatch the fault.  

10:00 PM EST

Chewy FC10 controls internal team members decided to proceed with reverting back to the previous timer change value which was 4 seconds. After this change, the Bastian Engineer realized that the site was nearly complete with the replacement of yet another motor exchange. The engineer decided to clear all faults in the queue of the VFD, upload the parameters of the motor, unlatch the custom fault bit, and note the timer change that was made by internal Chewy team members. 

10:30 PM EST

Shortly after completing the exchange of the motor and performing all the required resets, a test was made to the system to see if the fault would continue. After testing the system, the fault was not present and the site continued with its production activities. 

Root Cause:

There were a number of factors that could have caused this issue to occur.  Hence applying these actions led to the fix. First, having the servo motor properly wired and installed was required. Clearing the faults in the VFD fault queue was required, along with resetting the local fault and all custom faults. Identifying the proper timer pre-set values that were modified by site personnel is also critical in the optimization of the site equipment. 

It is suspected that the latch of the custom fault continued to be latched on due to the motor being out of service and not properly configured/uploaded. Once inhibiting the VFD and uploading the parameters, we were able to unlatch the fault along with editing a timer that the internal Chewy automation team included. The custom fault reset was done among 5 other procedures that took place and this could led to all 6 procedures needing to occur to successfully clear the fault. 

Resolution:

Issue 1: Faults within the fault queue in the VFD queue

Issue 2: Faulty suspected servo motor

Issue 3: VFD motor parameters not configured within the newly replaced motor

Issue 4: Improper the timer Pre-set value

Issue 5: AOI fault

Issue 6: Latched input custom fault

Next Steps and Preventive Actions:

Issue 1: Clear all faults within the VFD queue

Issue 2: Replace the faulty suspected servo motor

Issue 3: Upload the VFD motor parameters properly by inhibiting the module

Issue 4: Change the timer Pre-set value and note the ticket that changes are made so engineers can refer to it for future reference 

Issue 5: Perform a local reset of the AOI fault

Issue 6: Unlatch the input custom fault

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.