Knowledgebase Article Number
Published November 12th, 2020, at approximately 9:30 a.m. EST.
At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on November 9th-10th, 2020 described below.
Monday, November 9th
9:10 PM EST
10:00 PM EST
Recognized as a controls issue and Donovan was brought into the issue. Donovan got online with the site’s PC to troubleshoot. Began trying to power cycle faulted equipment to clear errors and faults. Also began checking for network issues at the ethernet switches.
10:15 PM EST
10:40 PM EST
10:50 PM EST
Site team indicated a "Point Bus Status" fault on the Point I/O. We tracked this down for a while to find causes, until the onsite team said this fault had already been cleared by cycling the power on the AENTR modules
11:00 PM EST
11:20 PM EST
Had onsite team confirm the message on the safety PLC "Safety Task Inoperative CPU01-0204 unconnected".
11:30 PM EST
Unable to find anything in Rockwell's knowledgebase. Donovan tried to go online with the processor and found that the control PLCs were version 30 and the safety PLC was 29, so the onsite programming software would not work with the safety PLC.
12:00 AM EST
Reached out to Jason Beisel and other support engineers to see if anybody had seen this fault before. Donovan began studying CP01 for faults and found nothing wrong. Donovan also looked at an offline copy of the Safety PLC.
12:30 AM EST
Believing the only path forward was checking the status of the safety PLC, Donovan tried to initiate the old VPN to use his computers programming software (which has 29 installed). The HBC IT was able to confirm that the VPN accounts had been disabled due to inactivity. HBC's IT attempted to turn the VPN account on but were unable to at the time.
1:00 AM EST
As one of the major issues we were seeing was loss of power to all VFDs in the facility it was asked if the onsite team had anybody qualified to check Voltage on the 480VAC supply to determine if the power was with the safety relays or with the overall power in the panels. Donovan was informed that the onsite team did not have anybody available to use a multimeter to do the checks.
1:20 AM EST
Bastian started downloading the Version 29 software on the local PC
1:40 AM EST
Bastian got online with the Safety PLC and found a Safety Task Fault. The fault was unrecoverable and required a download to clear
2:00 AM EST
After the download completed and we confirmed the Safety PLC was without errors or faults, The onsite team tried to restart the system and it started as expected
2:20 AM EST
After posting a few notes on the faults and issues found, everyone signed off the bridge call.
The safety PLC had a watchdog timer fault, because it took too long to execute a safety task.
We will need to set up the safety signature so that a download is not necessary. This should be done during a shutdown.
Next Steps and Preventive Actions:
Having a qualified system expert onsite for each shift is necessary. Many times in troubleshooting the first step is to test voltage. If 2nd/3rd doesn't have anyone qualified it really limits Support's ability to help.
By downloading version 29 of Studio5000, getting online with the safety PLC won't be an issue in the future.
Version 29 of Studio5000 needs to be installed onto the backup PLC/HMI Server PC so this problem won't be an issue when you swap PCs.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.