Knowledgebase Article Number

Description

Published October 23, 2020, at approximately 9:30 a.m. EST. 
Last updated on October 23, 2020.

Resolution 

At Bastian Solutions, we continually strive to provide superb customer service at the highest standard and not only reactive but proactive support. Transparency is a key component in helping achieve this objective, and with that in mind, we are providing a full root cause analysis for the disruptions in service on March 14th and 16th, 2020, described below.

Summary:

Friday, October 23, 2020

11:06 AM EST
A site called Bastian Support and spoke to Nathan Ernst, Support Specialist I, to report that pick stations were unable to pick confirm at all workstations. 

11:20 AM EST
Mr. Ernst connected to the system and started an investigation via logs and SQL queries. 

12:00 - 12:30 PM EST

Nathan worked with the site to restart Exacta services. Also attempted to reprocess failed imports from the host system. Also located additional errors related to order_journal records.

12:45 2:30 PM EST

Support Analyst I, escalation resource Owen Graham engaged. Located duplicate entries attempting to be inserted into the Order_journal table. Found that Order_journal sequence numbers had reached their maxed value. 

2:40 -  3:45 PM EST

Mr. Graham transitioned the issue to Support Analyst I resource Andrew Downey and our Software Support Manager, Travis Coleman, to clean up old records causing the issue. The final restart of all Exacta services / AOR / Workstations to confirm that the system was now working properly. 

Saturday, October 24, 2020

11:40 AM - 2:00PM EST

Support Specialist I,  John Will Martin, received a call from the site reporting one workstation was unable to pick complete orders. John reviewed the case history began investigating the issue. Support Analyst I, resource Barry Stone and our Software Support Manager, Travis Coleman, were contacted. Some additional records were located that needed to be archived, and the Seq_num was reset again. Reimported the failed records and confirmed that the site was operational. 

Root Cause:

The ExactaDB Order_Journal table has an identity column called seq_num. This is an autoincrementing field that exceeded its max value. This table is populated on imports, picks, exports, etc. 

Resolution:

Archived records that would conflict with our range and purged those records from the ExactaDB. Performed a reset on the seq_num column back to zero.

Next Steps or Preventive Actions:

Added SQL job that will periodically check the identity column, which will automatically reset the seq_num column if we exceed the max value. 

Our commitment to you:

Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. Our primary objective is to provide our clients with superb customer service, and we assure you we're taking the required preventative measures to prevent reoccurrence.