|
At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on February 9, 2022 described below.
Summary:
7:20 AM EST
The client notified Bastian that the site had experienced a power outage and that labels were not printing on either Print and Apply.
7:25 AM EST
Bastian began troubleshooting the incident by reviewing AOR logs, it was determined that logging was set to INFO so it was changed to DEBUG
7:40 AM EST
Bastian software restarted the AOR application, AOR PC, and the Application Server to start with a clean reboot of every machine after the power outage.
7:55 AM EST
Bastian software determined through logs that the AOR was not receiving all necessary data from the PLC, Bastian software engaged Bastian controls to help assist in troubleshooting, and to provide more visibility as to what the PLC was showing.
8:07 AM EST
Bastian software and controls began testing with the site to review live examples together.
8:28 AM EST
Bastian software escalated the issue to a Tier 2 resource.
8:33 AM EST
Escalation resource was assigned and brought up to speed by the Tier 1 resource
8:42 AM EST
Bastian software began reviewing Kepware tags while controls continued testing on their side.
8:46 AM EST
Bastian software began reviewing Kepware tags with controls to verify if data was being communicating properly.
8:53 AM EST
Bastian software attempted to manually update Kepware tags to see if Bastian controls could see the updates and had the site continue to run live examples through for additional testing.
9:00 AM EST
Bastian Software and Controls were able to see the manual updates being sent back and forth between the two systems. The Bastian controls agent relayed to the software support team that they were not able to find any issues on the PLC side.
9:20 AM EST
Bastian software escalation resource began reaching out to additional support resources to assist in troubleshooting.
10:30 AM EST
Bastian software escalation reached out to our development team to engage an additional resource to assist in troubleshooting.
10:50 AM EST
Bastian development resource was assigned and began troubleshooting the issue with Bastian's Tier 1 and 2 resources.
12:00 PM EST
The Bastian developer believes that the issue is caused by tracking ID's being sent by controls is out of sync with what software is expecting.
12:30 PM EST
Bastian software Tier 1 resource noticed that the application server date was set to 12/22/2022, and while proactively checking the other servers noticed that the Case Dispenser AOR had the same older date as well.
12:35 PM EST
Bastian software changed the data and time on the Application server to the current data and time, Bastian software did not have access to change the data and time on the Case Dispenser AOR PC.
01:00 PM EST
SGWS IT changed the date and time on the Case Dispenser AOR PC to the current date and time.
01:07 PM EST
Bastian software restarted the AOR to initialize the date and time changes.
01:15 PM EST
Bastian software began testing to see if this would remedy the sequencing issues.
01:35 PM EST
Bastian software reengaged Bastian controls to continue to assist in troubleshooting due to see continuing sequencing issues.
02:15 PM EST
Bastian teams and SGWS team determined that the best course of action would be to purge all of last nights data and to reimport a test wave so we could test with clean data.
02:30 PM EST
Bastian software continued to work with controls to test, and began putting together adequate testing data relevant to using the PANDAs.
03:45 PM EST
Bastian software completed the testing data and attempted to import, but the files were not being picked up by the Exacta import services
04:05 PM EST
Bastian Software restarted the import and import adapter services and continued to review logs to see why the file was not being picked up.
05:52 PM EST
Testing data began to import.
06:15 PM EST
Bastian software successfully imported testing data and began to test.
06:50 PM EST
Labels began printing on the D11 PANDA, but not on the D12 PANDA.
07:10 PM EST
Bastian Software continued to work with Bastian Controls and Bastian's development teams to determine what the difference was between the D11 PANDA and the D12 PANDA
08:04 PM EST
The Bastian Development team found that the AOR and not receiving all necessary information from Kepware
8:45 PM EST
Bastian software found stuck Kepware tags that communicated with the PLC.
8:54 PM EST
Bastian controls cleared the tags and cleared the buffer lane, then began to test with the site.
9:18 PM EST
Both D11 and D12 PANDAS began printing labels as expected.
9:30 PM EST
Bastian software purged the testing data to prepare for production.
Root Cause:
Onsite power outage caused the initial printing problems and was believed to have caused the date and time change on the application and AOR PC.
Resolution:
Ensure all Window PCs Bios times are up to date, and after power outages, ensure that time on all PCs are current.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.
|
Comments
0 comments