Knowledgebase Article Number

360057125674

Description

Published November 12th, 2020, at approximately 9:30 a.m. EST. 
Last updated on November 12th, 2020.

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on November 9th-10th, 2020 described below.

Summary:

Monday, November 9th

9:10 PM EST
HBC called call came in to the software support queue

10:00 PM EST

Recognized as a controls issue and Donovan was brought into the issue.  Donovan got online with the site’s PC to troubleshoot.  Began trying to power cycle faulted equipment to clear errors and faults.  Also began checking for network issues at the ethernet switches.

10:15 PM EST
Was able to see that many of the faults (safety prox fault at the sorters, healthchecks, VFD Active faults) were cleared and needed reset. To test, Donovan attempted to reset a few of the faults. Then tried to restart the system. Many of the same faults returned.

10:40 PM EST
HBC onsite team initiated a bridge call.

10:50 PM EST

Site team indicated a "Point Bus Status" fault on the Point I/O.  We tracked this down for a while to find causes, until the onsite team said this fault had already been cleared by cycling the power on the AENTR modules

11:00 PM EST
Learned that a message was scrolling across the safety PLC indicating an issue.  Due to this, it looked like a networking issue.

11:20 PM EST

Had onsite team confirm the message on the safety PLC "Safety Task Inoperative CPU01-0204 unconnected".

11:30 PM EST

Unable to find anything in Rockwell's knowledgebase. Donovan tried to go online with the processor and found that the control PLCs were version 30 and the safety PLC was 29, so the onsite programming software would not work with the safety PLC.

12:00 AM EST

Reached out to Jason Beisel and other support engineers to see if anybody had seen this fault before.  Donovan began studying CP01 for faults and found nothing wrong.  Donovan also looked at an offline copy of the Safety PLC.

12:30 AM EST

Believing the only path forward was checking the status of the safety PLC, Donovan tried to initiate the old VPN to use his computers programming software (which has 29 installed).  The HBC IT was able to confirm that the VPN accounts had been disabled due to inactivity. HBC's IT attempted to turn the VPN account on but were unable to at the time.

1:00 AM EST

As one of the major issues we were seeing was loss of power to all VFDs in the facility it was asked if the onsite team had anybody qualified to check Voltage on the 480VAC supply to determine if the power was with the safety relays or with the overall power in the panels.  Donovan was informed that the onsite team did not have anybody available to use a multimeter to do the checks.

1:20 AM EST

Bastian started downloading the Version 29 software on the local PC

1:40 AM EST

Bastian got online with the Safety PLC and found a Safety Task Fault.  The fault was unrecoverable and required a download to clear

2:00 AM EST

After the download completed and we confirmed the Safety PLC was without errors or faults, The onsite team tried to restart the system and it started as expected

2:20 AM EST

After posting a few notes on the faults and issues found, everyone signed off the bridge call.

Root Cause:

The safety PLC had a watchdog timer fault, because it took too long to execute a safety task.

Resolution:

We will need to set up the safety signature so that a download is not necessary.  This should be done during a shutdown.

Next Steps and Preventive Actions:

Having a qualified system expert onsite for each shift is necessary.  Many times in troubleshooting the first step is to test voltage.  If 2nd/3rd doesn't have anyone qualified it really limits Support's ability to help.

By downloading version 29 of Studio5000, getting online with the safety PLC won't be an issue in the future.

Version 29 of Studio5000 needs to be installed onto the backup PLC/HMI Server PC so this problem won't be an issue when you swap PCs. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.