Metronet - Priority Problems

Problem: [62178] Data centre problems

Status: Closed

Problem description Added on: 03/06/2010 @ 02:15

At approximately 4am, we suffered a partial outage affecting one of our data centres in Sheffield. The problem followed shortly after a line card failed in one of our core routers. This will be affecting access to certain services for some customers.

If you were logged in to the portal, you could associate your username with this problem.

Comment updated on: 03/06/2010 @ 10:26

Monitoring flags numerous problems between 3am-4am.

Comment updated on: 03/06/2010 @ 10:26

Investigation suggests faulty line card at one of our Sheffield data centres is contributing to the problem.

Comment updated on: 03/06/2010 @ 10:26

Engineer arrives onsite at 5.30am to reseat affected card.

Comment updated on: 03/06/2010 @ 10:27

Work begins at 6.00am to bring affected servers back into service.

Comment updated on: 03/06/2010 @ 10:28

Second card failure shortly after 9.00am, case raised with supplier.

Comment updated on: 03/06/2010 @ 10:28

Service Status thread published here - http://usertools.plus.net/status/archive/1275553909.htm

Comment updated on: 03/06/2010 @ 11:34

We're continuing to investigate routing problems on the network.

A replacement line card is sceduled to arrive on site shortly after midday.

Comment updated on: 03/06/2010 @ 12:00

Email has been queuing on our anti-spam appliances since about 9.00am this morning. This will be leading to delays in the arrival of email for anybody who has spam filtering switched on for their account.

Comment updated on: 03/06/2010 @ 12:43

Customers may be experiencing IMAP connection errors when logging into Webmail.

Comment updated on: 03/06/2010 @ 13:36

Followup Service Status announcement published here - http://usertools.plus.net/status/archive/1275567344.htm

Comment updated on: 03/06/2010 @ 13:49

Restoring the new card caused a short storage blip that will have affected connectivity to our website and some other services.

Unfortunately, this has created an issue with access to our Plesk/PAYH hosting platform that is currently under investigation.

Comment updated on: 03/06/2010 @ 18:07

Access to our Plesk/PAYH platforms has been restored although we will need to recut the database as it has become corrupted.

Comment updated on: 03/06/2010 @ 18:09

Webmail access restored and all mail has been dequeued from our anti-spam appliances.

This email still needs to filter through to customers' mailboxes and until it does access to our mail collection servers may take longer than usual and you may still see some emails arrive that have been delayed in transit.

Comment updated on: 03/06/2010 @ 19:11

We have now restored all service, we hace had to temporarily disable POP for the following VISPs, however they are still available over IMAP and Webmail:

globalnet.co.uk
freenetname.co.uk
totalise.co.uk
care4free.net
madasafish.com

Comment updated on: 03/06/2010 @ 20:16

We have restored service to all VISPs now.

Comment updated on: 03/06/2010 @ 20:20

Unfortunately this has recurred and we have had to stop POP for the same VISPs, as before IMAP and webmail should still be working.

Comment updated on: 04/06/2010 @ 18:33

We have now restored POP access to all VISPs

Go to current priority problems | Go to archived priority problems

Plusnet Service

Problem Detail