Important Update regarding Network Issue
As many of you are aware, our supplier experienced a major outage on the Horizon service yesterday (14 November), resulting in the loss of the Horizon service for many of our customers, including ourselves. We have been advised by our supplier at 7.04 this morning that patches were applied and work completed successfully with no impact to service. They are continuing to monitor the platform and expect to provide a further update shortly.
Many of our customers are finding that their service is restored this morning. If you have any issues with your Horizon phones we would suggest rebooting them, and if that still doesn’t resolve the issue please contact us on 01453 704820 or by emailing firstname.lastname@example.org.
When we have further information regarding this incident we will provide an update. In the meantime we would like to apologise for the issues many of you were experiencing yesterday.
Update @ 14.00:
Our CEO has written to all customers affected by the issue with some more information regarding the outage. Our supplier has so far confirmed the following:
Overnight on Tuesday/Wednesday a planned, routine upgrade, one of over 200 in the past 5 years, was applied to the platform; this upgrade contained a software bug. This was the initial cause of the problem. Unfortunately the patch to correct the bug was not applied at the time of the upgrade and it was only on the Wednesday morning that the issues became apparent. The consequent failure of this update caused an increased load across the platform which meant that devices were unable to register; the Horizon portal became overloaded and effectively inaccessible to users and suppliers alike. Engineers worked methodically to ensure that the resolution was both effective and permanent and this took longer than hoped.
We are assured that lessons have already been learned in terms of both the application of updates and responses to any failures. We are further assured that the network believe that there is little or no likelihood of any similar incident occurring and that, in the unlikely event of a similar issue arising, the response and fix time will be much more swiftly handled and communicated.