Maintenance Type: Switch Maintenance
Expected effect on your service: No Connectivity
Expected duration: <2 Minutes
Reason for the reboot: To activate new firmware.
This will occur between: 02:00 and 06:00 on 09/03/2010
Over the last day or so, a few people have been unable to stop their AutoDJ stream. I’ve looked in to this and it seems like one user’s process was hogging the cpu (not a fault of their own). I’ve restarted the offending AutoDJ process and the server is now stable. If you still experience problems stopping or starting your AutoDJ stream then please open a support ticket.
We are aware some clients are still having connectivity issues. To rectify any remaining issues we are working on adding a connection to Cogent who is On-Net within our datacenter. They are currently not part of our bandwidth providers.
Abovenet has updated us they are trying to reroute the cut fiber using an alternate path because they are unable to gain access to the fiber for repair due to fire.
We do not have any ETA/ETR estimates at this time. As soon as we have more information we will send another update.
The network issue is a bend in one of our fibers to NYC, provided by Abovenet. Techs have identified where the problem is, have prepped for the work in the manhole they need to be in.
The fiber cut is being workedon by Abovenet. They are saying that the c rew is in place to replace the span of fiber that is damaged.
They are not able to give us an estimated time of repair yet.
There seems to be an issue affecting some routes to the USA premium network, affecting the following servers:
usa1-pn, usa2-pn, usa1-adj, usa2-adj, usa3-adj and usa4-adj
I have a ticket open with the datacentre and will update this post as I get info.
UPDATE: This seems to have been resolved and seems to have been a temporary issue with a supplier outside of the network.
Last night, usa3-pn went down. It was put back up within 30 minutes, however when the server was originally set up, http wasn’t added to autostart when the server boots – this is normally done when a server is set up as it is required for shoutcast streams to start.
As a result, the small number of customers on the server would have experiences a long outage, and we will be offering a partial refund under the service level agreement to all affected customers.
We apologise for any inconvenience caused by the downtime and have fixed the error that caused it to last for longer than necessary.
In relation to the previous post, the outage was caused by power loss that was experienced at BlueSquare datacentre. This affected uk2-pn stream server as well as one of out DNS servers so a small number of people may have experienced outages on the website due to lack of DNS (including websites of customers that use ns2.thisdns.net as one of their nameservers.
All is back to normal now.