Description

B&H Orders not importing

Support Ticket 190863

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service due to Order Cubing. 

Summary:

April 29th 

12:30 PM EST

Bastian Support received a call reporting that Orders were not imported into Exacta from the host system. T1 resources began troubleshooting the issue by investigating order examples on checking logs.

Exacta Import and Import Adapter were restarted as part of the troubleshooting process. 

After further troubleshooting, it was determined that order imports were failing because they already existed in the Exacta Database. 

1:30 PM EST

1 hour into the investigation the issue was escalated to a T2 resource. 

Bastian identified multiple log examples of timeouts and connection errors. Around the same time, the B&H DBA team found that the Database server was spiking at 100% usage due to a long-running query. 1:45 PM EST B&H DBA team restarted the server.

This cleared the Usage issue but orders were still not processing. Bastian Identified that during the DBA spike many orders were left OnHold status. These were released and finished processing. Further instances of timeouts and connection issues were present in CICM around the timeframe of the reported issues. 

All relevant logs from the incident have been pulled back and attached to the Support ticket for future reference. 

Conclusion

At this time, It appears as though the Database usage issue prevented orders from properly being assigned at the time of their import. Further attempt to resend the orders caused the second batch of orders to fail as they were already in the system but not completely processed yet. Once the Database server / Application servers were restarted, and the existing orders were taken off on OnHold status remaining work began to process, and work resumed. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.