Plusnet Service

Some customers are seeing lower than expected speeds and performance issues on our 20CN broadband network.

This P1 problem has been raised to track the issues that are being reported, investigate the causes of the problems and report on the progress we've made to resolve each of the issues identified.

There is no single cause to the problem, we will document each issue and the outcome of each.

If you were logged in to the portal, you could associate your username with this problem.

We've made some routing changes and added extra capacity to our network in order to aid the resolution of this issue.

Please visit http://status.plus.net for more information.

The following work has been done to improve performance on our broadband network.

We have identified an issue on some of our core network devices which was causing some packet loss of customers' traffic. As this was happening before reaching our traffic management equipment the packet loss was across all different kinds of traffic.

UDP traffic like DNS and gaming was seeing more problems than TCP traffic because the TCP traffic will retransit and thus only see an increase in latency whereas the UDP would drop so cause DNS lookup failures for example and for browsing mean that pages just wouldn't load.

We have moved some of the traffic that was going through the congested routes on to a different route through our secondary Ellacoyas which has significantly improved the performance of traffic, we are continuing to work on this until we have eliminated the packet loss.

We have added additional capacity to both our 21CN network and our 20CN network and moved some customers from the 20CN network to 21CN thus giving more capacity on the 20CN customers. Further migrations to 21CN from 20CN and more bandwidth on 21CN are due next month.

With the addition of more bandwidth we have made changes to the queue weightings on our network to ensure that the bandwidth is spread correctly across the available queues on our network.

Further work has also been done on our DNS platform to improve performance.

We have also been closely monitoring the network balance to ensure that the spread of customers is even across our network. A number of pipe drops recently has meant that the network has been out of balance meaning that customers will see noticeable performance differences across different gateways.

There have also been periods of increased demand such as the Microsoft patch Tuesday last month and this week, which also coincided with an Apple OS update and the release of the game Modern Warfare 2. We are expecting a significant increase in demand this weekend with both Top Gear and Doctor Who on Sunday in HD and the new series of I'm a Celebrity get me Out of Here.

Further updates will be posted as more work is done.

As separate issues to this problem we have encountered 2 individual problems with 2 of our gateways - PCL-AG03 and PCL-AG02. The problem with PCL-AG03 has been resolved while the problem with PCL-AG02 is currently being worked on.

We have moved some of the capacity on PCL-AG02 to different gateways however the problems mean that the network is out of balance and some customers may see a reduction in speed because of this.

We have completed some maintenance work on our network to enable some additional Juniper routers to be able to take customer connections from any 622 pipe. This work will allow us to have a more even spread of users across each of the Juniper routers on our network and a more even spread of the traffic.

Work has also been done on the problem with the router pcl-ag02. It was anticipated that customers connected to this router may be disconnected, fortunately this didn't happen. We are waiting for feedback from our suppliers regarding the logs that were generated during the maintenance. Customers connecting on the 622 Plusnet 9 that normally connects to pcl-ag02 will be routed to one of 3 other routers over the weekend. Customers connected on pcl-ag02 will remain connected to that router.

We have also moved some Madasafish traffic away from our traffic management platform to reduce the CPU load as the traffic was only being monitored.

The network has been put back in to Plan A configuration for the weekend, we will review the weekend's performance on Monday and post a further update.

We have further work scheduled next week to make further improvements, further details on these changes will be posted next once the testing and change controls process has been completed.

Yesterday we made some changes to our traffic management platform to do better detect traffic from Content Distribution Networks (CDNs) Akamai and Limelight networks and be able to diffentiate between streaming, browsing and file downloads.

We have moved the browsing traffic into the gold queue for the following products:

Premier (all varients)
Unlimited
Premium
BBYW3
BBYW4

Traffic for Business, PAYG, Plusnet Pro and BBYW Pro will already be in gold.

We are now analysing the remaining traffic and will look at moving streaming next week.

We encountered a problem yesterday affecting 2 of our 622 pipes (Plusnet 9 and Plusnet 14) which meant that some customers were seeing slower than normal speeds when connected to one of these 622s. This problem was resolved yesterday evening and our monitoring reports that the traffic levels have returned to normal.

Further capacity has been ordered on both our 20 and 21CN networks and is due to be delivered next week. Bulk moves to 21CN from 20CN are due to begin next Monday.

We will post further updates as more work is done to improve the performance of the network.

We have completed a significant amount of work this week to improve the performance that customers have been seeing. The changes made include the following:

We have upgraded part of our DNS platform to try and reduce load and increase performance, we will review the performance and compare this against the performance of the part we haven't upgraded to see the difference this has made.

The load balancer at one of our data centres have been upgraded which should give a further performance uplift on DNS.

Upgraded the line cards on the gateway routers at one of our data centres which was seeing small amounts of packet loss at busy times, this should again improve performance on DNS but also gaming and other applications that noticed problems due to small increases in latency and packet loss.

The replacement hardware for the gateway PCL-AG02 has been installed and is currently ongoing testing and configuration. We expect to bring this into service shortly.

We have made changes to our traffic management database to better detect content from CDNs (content distribution networks) such as Akamai and Limelight networks, in particular streaming and browsing. This includes detection of the new iPlayer application on the Wii. We've also added a new batch of games and better signature detection for Rapidshare and Megaupload.

Further migrations from 20CN to 21CN are continuing and to support these migrations we have increased the bandwidth across our 21CN host links. Further increases are due later this month. We will post an announcement when these changes are made.

Further work will be done next week, in particular on the DNS platform and further migrations to 21CN.

We will post another update next week.