Telefoniestoring NXTvoice

NummerSRV/24/INC/MAJ/09401
Referentie
Datum25-03-2024 08:03
TypeIncident: Major
Tags
Status
Afgerond
Afmelding
@ 28-03-2024 14:23

RFO heeft op de website gestaan voor een maand. Geen vragen meer gehad zover ik zag.
Reason for Outage (RFO)
@ 28-03-2024 14:27

 
RFO NXTvoice incident Levelfour SRV/24/INC/MAJ/09401

Incident
Start Date | Time 25-03-2024 06:06 hour
End Date | Time 25-03-2024 09:41 hour
Customers Involved All customers using NXTvoice
Impact for customers No service via NXTvoice
 
Incident Report
Description
In the time frame mentioned above all customers using NXTvoice of Levelfour experienced a total loss of service. 
 
Cause
The outage was caused by a an issue with one of our database servers that caused a blocking incident with our Voice cluster.
 
Route Cause Analysis
 
Timeline
06:06 Monitoring is reporting issues from failing DB server. The impact of this was not yet clear due to the expected redundancy.
07:45 First customer is reporting loss of service.
07:46 Immediately scaled up to priority 1 incident. Engineers are investigating the issue.
08:08 Search area narrows to one of the cluster servers.
08:40 Search area further reduced, problem seems to be in the database of the server application. Services are not starting.
09:07 First time reboot of the servers, calls were immediately possible again. Continue to monitor.
09.25 Preventive second reboot of the servers, continue to monitor.
09:41 Telephony is working properly again and the devices are registered on the platform again. Problem has been resolved.
 
Problem solving
After the issue had been found we removed the DB server from the cluster and restarted the voice services. We were able to quickly repair the issue on the failed db server and it was put back in service the same morning.
 
Preventive measures
Together with our supplier we are working on implementing a patch to prevent this from happening in the future.
 
 
 
 
In case of questions regarding this RFO please send an e-mail to: servicecenter@levelfour.nl with subject RFO incident SRV/24/INC/MAJ/09401
Status update
@ 25-03-2024 09:18

Alle systemen en diensten zijn weer operationeel. Momenteel zijn we de situatie aan het monitoren. Indien er meer informatie bekend is vermelden wij dat bij deze incidentmelding.
Status update
@ 25-03-2024 09:08

De oorzaak van de storing is gevonden en we zijn bezig om de diensten weer operationeel te krijgen.

Wanneer we nieuwe informatie hebben zullen we de status bijwerken.

Statusupdate
@ 25-03-2024 08:05

Momenteel is er een grote storing gaande op ons NXTvoice platform. Hierdoor kan er momenteel niet worden gebeld.  Er wordt gezocht naar de oorzaak.

Wij zullen u op de hoogte houden van de voortgang.

Excuses voor het ongemak.

Aanmelding
@ 25-03-2024 08:03

Telefonie storint nxtvoice.
Hoe tevreden bent u over deze afhandeling?