Knowledgebase Article Number

Description

Resolution 

At Bastian Solutions, we continually strive to provide outstanding customer service, not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on April 12th, and 13th of 2020.
 

Summary:

April 12

8:07 AM EST

The client notified Bastian to join a bridge call due to document inserters not printing out collates. 

8:20 AM EST

Bastian joined the bridged call requested by Usama. Upon joining the conference call, HBC representatives were discussing possible network related issues such as, checking the switch that the printers are routed to as well as using network commands to identify other network issues.

8:30 AM EST

The on-call escalation representative for the weekend was notified about this being a high level priority and system down issue.

8:45 AM EST

The client mentioned a DNS change that was implemented on Thursday of the previous week. After that, a traceroute test was conducted by the client and there were some sort of failure mentioned on the bridged call by the client. Shortly after, the client concluded that there were no network issues on their end. 

9:00 AM EST

Bastian began troubleshooting the communication between the AOR software and the document inserter printers. We were able to successfully ping all of the printers from the AOR as well as a traceroute test. Bastian wasn't able to connect to the HMI PC. The client then rebooted the HMI PC and now we were able to establish a connection to the HMI PC. 

9:25 AM EST

At this point in time, the goal was to have the client back up and running as soon as possible. A full services restart on the application server was initiated. 

9:40 AM EST 

Escalation assisted with the full services restart. All services were fully back up and running. Bastian then restarted the AOR software. All required services and the AOR were now running. Bastian's controls team was notified about the ongoing issue.

9:45 AM EST

The client started their conveyor back up and confirmed that the first several cartons had collates applied to them. The client and Bastian continued with additional testing to conclude whether or not the document inserters were properly functioning. 

10:08 AM EST

The client confirmed to Bastian that the document inserters were fully functioning without any issues. 

10:30 AM EST

Bastian reviewed AOR logs and a consistent error in the logs stated that "Exception Thrown Trying to get status of printer." Relevant logs were pulled to continue the investigation. 

 April 13

7:22 AM EST

Bastian was briefed about the document inserters not functioning like the previous day. The client confirmed that they restarted services and the AOR. The client was running without issues but wanted to notify us about the incident occurring once again. 

7:30 AM EST

Bastian reviewed additional logs and the same errors from the previous day persisted. 

9:15 AM EST

Bastian continued investigation by reviewing logs and worked with the client by gathering additional details. 

2:15 PM EST

Through investigation, Bastian noticed that when the document inserters weren't printing based on the client's details, the logs point to the arrival scanners not seeing the cartons, possibly due to the HMI reboots. A test is scheduled tomorrow morning for the client to call Bastian before the HMI PC will be restarted. 

9:45 AM EST April 12

The HMI PC was restarted by the client, along with Bastian restarting services on the application server and a restart of the AOR software.

Root Cause: 

Communication to the PLC was broken due to faulty HMI restarts

Resolution:

A test is scheduled tomorrow morning to confirm if the HMI restart is causing a broken communication to the PLC as well as verifying if the DNS servers being incorrect is causing issues connecting to the Exacta Database server.

Next Steps and Preventive Actions:

Our commitment to you:

Bastian Solutions understands the impact of the downtime that occurred because of the document inserters not functioning as intended, which in turn affected your business processes for your organization. It is our primary objective in providing our clients with outstanding customer service and we assure you that we are taking the required preventative measures to prevent this issue from reoccurring.