At Bastian Solutions, we continually strive to provide superb customer service at the highest standard, and not only reactive but proactive support. Transparency is a key component in helping achieve this objective and with that in mind, we are providing a full root cause analysis for the disruptions in service on January 6th, 2019 described below.
10:30 AM: Initial Call in to support to report issue
10:40 AM: Call was ended so that they could create a bridge
11:00 AM: I received the call for the bridge and started working on the issue
11:05 AM: Seeing that we had an error related to IIS being able to access the portal folder I worked with Usama to have them shut down picking for a moment so I could reset IIS
11:20 AM: I advised we bring on network and security teams to see if any changes were made to the system over the weekend
11:25 AM: We noticed that permissions were different on the Canada site when compared to the Pottsville site and advised for an IT member to add the permissions back
11:30 AM: Working with member of HBC IT team we added in Usama's account to the portal folder but this did not resolve the issue
11:40 AM: I officially escalated this issue internally to our Tier 2 team and brought in Jason to assist in troubleshooting.
11:45 AM: We worked to find any issue with the permissions that were different from PADC and HBC IT worked to get the Bastian Solutions folder available to the "Everyone" group
12:00 PM: The Bastian folder permissions did not work for the Usama account but we advised to add the Everyone group to the permissions since it is IIS that is accessing it, this resolved the issue
12:00-12:15 PM: The HBC team worked to figure out what account made changes and the bridge call was ended
Someone on site at HBC looks to have removed the permissions to the Bastian solutions folder on the application server.
Worked with HBC team to get everyone permissions back so that IIS can load Portal properly.
Next Steps and Preventive Actions:
HBC IT determine when change was made and validate permissions and access are secure for the future.
Our commitment to you:
Bastian Solutions understands the impact of the disruption that occurred and affected operations for your organization. It is our primary objective in providing our clients with superb customer service and we assure you we're taking the required preventative measures to prevent reoccurrence.