Knowledgebase Article Number

1500003615141

Description

Published March 12th, 2021, at approximately 9:30 a.m. EST. 

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on March 9th, 2021 described below.

Summary:

Tuesday, March 9th

11:40 PM EST
Ingram Micro call came in to the software support queue.

11:45 PM EST

Recognized as a controls issue and Donovan was brought into the issue.  The reported issue was a loss of communication to MPE PLC Controller.  Donovan began trying to get online with the site’s PC to troubleshoot. 

11:50 PM EST
Donovan was receiving a message through the Citrix website that the username/password was incorrect for the VPN remote access.  Eventually he was able to get the Password reset.  

12:05 PM EST
Donovan opened up the program stored on the local PC and tried to go online with the MPE PLC which was the controller not communicating.  The programming software (RSLogix 5000) reported that the controller did not have a program in memory and had a non-recoverable fault.

12:10 PM EST

Donovan cleared the fault and started a download of the most recent copy of the program on the local PC to the Controller.

12:15 PM EST

The download completed and Donovan was able to verify that the PLC was back up and communicating to the rest of the system.

12:20 PM EST

I tried to reach back out to Ryan Misner the original Requester who called in but the phone call had been disconnected.

12:25 PM EST

Donovan called Ryan Misner back and inquired about the status of the system after the PLC had recovered.  It was reported that the system was back up and running and the onsite team would monitor the rest of the night.

Root Cause:

The PLC was faulted.  A "Power-up Fault" (Type 01, Code 62) was the recorded fault code.  This fault causes the memory to clear out the program.

Resolution:

Downloading the program to the processor after clearing the fault gets the controller up and running again.

Next Steps and Preventive Actions:

We would suggest having a spare Processor onsite if this Controller is at fault.  Make a note and monitor for future occurrences.  We can get on and load a current copy of the program to the non volatile memory (usually an SD card if installed).  If you do not have an SD card we would suggest purchasing one and installing.  If a fault happens on the processor, cycling the power will clear the fault and if no program is detected then the backup saved to the SD card will load and start.  This will allow the facility to get running again and we can more formally deal with the issue later.

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.