Parts Town 2022-02-03-RCA CNA34

Timeline	Hindsight	Moving Forward
2:30 pm Tuesday
Kenny helped James get access to the server so James could make changes on CNA34 PANDA. James mentioned planned changes before 11 am when the sales side starts.	This approval discussion should have been its own dedicated meeting, with all stakeholders fully aware of the breadth of changes and timing.	Change control process, formal approvals by all stakeholders.
Wednesday
7:20 AM
James put in the changes for CNA34
Ricky Castillo in putaway emailed about errors at putaway ports.
Ricky called John Martin
John called Kenny saying the server services were down
Kenny didn't know why services would be down, asked John to reboot the server in case services were unstable
John rebooted
James called Kenny to explain he was making the aforementioned updates
Miscommunication: Kenny and James discussed different options about moving forward, Kenny remembers asking to complete the updates on the database side at this time. James remembers Kenny saying to wait until after hours to complete the database updates.	At this point, had the database updates been made, AOR may have worked as intended, because the Parts Town operation had not processed any shipments yet in Packing. Shipments after the config updates would have contained the necessary information to route properly at the ship sorter ... (note: the work categories missing the Import Adapter config would have caused the same downstream effects on picking and putaway orders getting stuck. We also are not sure why there was any effect on AOR heartbeat, see below)	If a change is ever at the "halfway in" point, it is incumbent on the Bastian Project Manager to inform the customer exactly what the consequences operationally would be of rolling back vs. fully implementing the change (rolling forward).
11:20 AM
Jeremy called Kenny to say all ship sorter boxes were going to Jackpot
Kenny called James and Michael to decide next steps. James had not put in database updates yet, so Kenny green-lighted that.
With CNA34 fully implemented at that point, AOR heartbeat wouldn't come back on despite a full restart of the panel with Troy.	We don't know why the heartbeat wouldn't come back on. This needs to be understood before we go in the next time.
11:50 AM
Kenny and Troy made the decision to roll back
Michael wrote [in real time] the necessary database rollbacks, James handled the server rollback with Kenny	Because Michael had to rush in writing rollbacks for several different tables/configs, the Import Adapter config was missed during the rollback.	Rollback scripts should be written in advance for all database updates. This saves precious minutes during a rollback and minimizes the risks involved with trying to write code in a pressure situation.
12:20 PM
We came back up having rolled back, AOR working, boxes sorting normally again at ship sorter
Everything from this point was due to the botched rollback. We missed work categories in the Import Adapter config
1:30 PM
Erick called Kenny saying AutoStore Continuous Putaway was not sending work down		Whenever the operation at Parts Town is saying "certain types of work aren't coming down" or "we aren't getting any work", our first check should be the AutoStore Console to see what we have there, then order_header statuses, then Import Adapter. Everyone working the Parts Town account should be made aware of customer category requirements.
Kenny asks John, Michael, James to join Webex
Kenny decides a reboot of PRDEXAPP01 might fix that and was quick to do – but restart had no effect on the putaway issue
1:45 PM
Kenny asks John to escalate
We get the word that only certain types of work are showing up in Continuous Picking and see that orders are stuck in status 100
Thinking these issues are probably related to the rollback in the morning, Kenny called Tom to see if we had a snapshot of the server … we did, from 1/23, so we reverted to that snapshot
2:00 PM
Andy Downey joins in the background for Bastian senior support resource
2:30 PM
Everything is back online with the reverted snapshot in place
Issues at both picking and putaway ports persist, no change
Kenny brings in Dave Strawser onto the Webex
3:00 PM
Terry joins on Bastian side in the background
3:15 PM
James had saved the Import Adapter configuration in Production prior to the changes in the morning. Michael compared that backup version of the config to the one currently active and saw that we were missing the customer category (aka work category) configuration section. This meant that PreProcessor couldn't process the picking or putaway orders since 11:20 when the CNA34 database updates went in.
Michael then completed the rollback of the Import Adapter config, which immediately fixed incoming work from that point on. That left us with about 2000 pick orders with NULL customer categories, and 10 putaway orders to remediate, but new work was getting assigned from this point.
3:45 PM
Michael and Terry worked on the database updates needed to pair translated import XML with the order_header, giving customer categories their proper values. Following this update, picking work began PreProcessing normally.
We were unable to fix putaway orders with a database update, so the group on the call made the decision to cancel and resend all open putaway orders. Nestor on the Parts Town side joined to assist with that effort.
4:30 PM
All systems green.
Root Cause
Partial Rollback occurred which caused all new orders that were imported to come over with missing work categories.
Next Steps and Preventative Actions
Parts Town and Bastian working to implement a new change request process.

1-844-227-8426 (BASTIAN)

Customer Service & Support

Comments