Important: Network Issue
We are aware that there is a major network issue this morning (14 November) which is affecting a number of our Horizon customers, including ourselves, meaning that we are currently unable to answer phones. Please be assured that our supplier is working urgently to restore service as quickly as possible. Also please see below for updates:
Update @ 10.29:
Update from our supplier that they believe they’ve identified the root cause which relates to one of their application clusters. Their engineering team and vendor are coordinating an emergency update to the impacted cluster to restore the impacted services.
Update @ 11.00:
Update from our supplier: An emergency patch was applied to the impacted application cluster and we have monitored a positive recovery, we are now replicating this work across remaining application clusters, to ensure full stability.
Update @ 11.30:
Update from our supplier: Our engineering teams are continuing to work with our vendor whilst they replicate the earlier work across the remaining clusters to ensure full stability. We apologise for the ongoing impact to service, this incident is being managed as our highest priority.
Update from our supplier: We are continuing to see some intermittent issues across our Horizon base and are continuing to work on full restore as a priority.
Update from our supplier: We have identified and patched the root cause of the issue and we are in the process of taking steps to stabilise call volumes across Horizon services
Update from our supplier: We do not have a current indication of when all services will be fully restored, but we are taking further steps to ensure this happens as quickly as possible. We continue to monitor the situation and will advise on the recovery progress within the next 30 minutes.
Update @ 13.31:
Update from our supplier: We are still monitoring a slow but positive recovery of Horizon services following the earlier work. We are now taking some additional measures with our technology partner to expedite the recovery. Whilst the recovery is underway, customers may still experience partial or no service on the Horizon platform currently.
Update @ 14.15:
Update from our supplier: We have now taken further action with our technology partner to further stabilise the Horizon service and we expect to start to see an improvement on the ability for customers to make and receive calls consistently by 15:00.
Update @ 14.20:
Please note that if you have access to the Horizon portal you may now be able to place a divert. This was not working at all earlier in the day but now seems more stable (although very slow).
Update @ 14.55:
Update from our supplier: We continue to prioritise this incident and work toward full restoration for those customers that remain affected. We apologise for the ongoing disruption and we will provide a further update soon. Thank you for your patience.
Update @ 16:23
We are starting to see evidence of phones rebooting and coming live. This looks like a slow process and not all phones, on a site basis, will recover at once. Hopefully all sorted by the morning. If in the morning then you still have phones not working then try to reboot a handset or two.