[RESOLVED] Broadband Connectivity Issue

We are currently investigating a connectivity issue affecting Broadband products only; ADSL, FTTC, FTTP. The issue started around 11:30 am. All other services are unaffected.
Some network routes are unavailable, while others are OK.
An upstream network supplier is investigating and we will update when we have more information.

We have seen normal service from 15:00 when the upstream supplier re-routed traffic around the affected device to alleviate the issue.

Update 11:00

The line card in the bypassed router was replaced overnight and has been brought back into service.

Update 18:00

We have continued to see normal service from 15:00 since the upstream supplier re-routed traffic around the affected device to alleviate the issues, but they are still working on root cause of the issue.

Update from the supplier as follows

————————-

Current Action Plan:
Following discussions between NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. The second action was performing a full remote reboot of the affected line card.

During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.

Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.

————————-

————————————————————————————

---

UPDATE 15:30

The upstream network supplier has identified a router which is causing the issue and has re-routed traffic around the affected device to alleviate the issues, and we saw the service return to normal from 15:00, however they are still working on the issue. Update from the upstream network supplier below:

————————-

Current Action Plan: NOC have been investigating further and have determined that the issue is emanating from a specific location with the core network. As such, this device has been checked and the hardware supplier has been contacted directly for a Priority 1 case to be raised with them. During this time, NOC have attempted to alleviate some issues across the core network by performing a workaround to reroute traffic to avoid the suspected affected device.

All of the requisite logs have been provided and NOC are working alongside the hardware supplier to determine our next course of action. As of yet, it is still unclear as to when full service can be restored however we will endeavour to provide this as soon as we have it available.

We apologise for the disruption and hope to have a further update soon.

Next Update: 17:30

————————-

————————————————————————————

UPDATE 15:00

Service appears to be back to normal, although we are still awaiting an update from the upstream network supplier to confirm the resolution of the issue.

————————————————————————————

Update 13:45

The upstream network supplier causing the routing issue on our FTTC and ADSL broadband products has issued the following update:

————————-

Completed Actions:

• Reports of circuit impact into the TSC
• TSC engaged NOC for initial investigations
• NOC confirmed an issue seen on active monitoring
• MI process engaged
• MI accepted
• Internal Bridge call scheduled
• NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

Current Action Plan: At this juncture, it has been determined that several tunnels are currently flapping following initial investigations by NOC.

We are working to determine as to why the tunnels are flapping in order for restoration works to begin. At this point, we are unable to provide an estimated time of repair however once one is available we will provide it as soon as possible.

We apologise for the disruption and hope to have a further update soon.

Next Update: 15:30

————————-

————————————————————————————

UPDATE 12:45

The upstream network supplier causing the routing issue on our FTTC and ADSL broadband products has issued the following update:

————————-

Description: Major Incident – INC0107444 – Multiple circuit outage

Current Impact: Broadband. Unknown number as of yet

Incident Start Time: 22/07/2022 11:57:59

Major Incident Raised: 22/07/2022 12:25:43

Major Incident Manager: Sam Abberley

Location/City: Nationwide

Completed Actions:
– Reports of circuit impact into the TSC
– TSC engaged NOC for initial investigations
– NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled

Current Action Plan:
A Major Incident has been declared as of 12:25 following the confirmation of an issue affecting our core network. Initial impact assessment has determined that broadband circuits are affected.

Currently, we are investigating the issue alongside the NOC team and are working to restore service as quickly as possible. An internal bridge call is being held at 12:45 to discuss the current impact and determine a plan of action for restoring service.

We apologise for the disruption and hope to have a further update soon.

Next Update: 13:30

22/07/2022 12:39

————————-

————————————————————————————

UPDATE 12:33
The routing issue affecting FTTC and ADSL broadband products has been traced to an upstream network supplier. They have issued the following update
————————-