There currently seems to be some packet loss on one if the USA value networks. This will affect some customers on usa1-5 vn.
It may be another DDOS attack. Updates to come.
I will be performing an update on the systems used for stopping/starting streams today. All shoutcast streams will need restarting, but this will only cause a few seconds of downtime and then the servers will come back up automatically.
This is intended to resolve any issues with servers randomly stopping as reported by a small number of customers.
The restarts will be performed over the next few hours.
UPDATE: This is now complete on all servers apart from USA3-VN which needs some more work before the change can be made. Updates will be posted here on any maintenance. This will be within the next week.
We are unable to access most servers on the value network from the office. Customers may be experiencing issues streaming on the usa value network (usa1-usa5).
Due to the network being down, we are currently unable to open a ticket with the datacentre, however I can assure you that they are very aware of the problem and will be working hard to fix it.
Update:
All back up now, this lasted about 2 hours.
Message from datacentre:
Our Chicago facility has experienced network issues this evening due to core network hardware failure.
The issue has been resolved and the network should be back to normal now.
The root cause of the problem was a failed 4x10GE blade on one of our core routers. When the traffic failed over to the other core router it has crashed the router thus causing complete network outage.
We have attempted to recover the core setup under existing configuration but once it became apparent that the blade was defective we had to move some uplinks and take the blade offline.
It seems like usa2-vn (value network) is currently experiencing downtime. The server is being rebooted shortly and we expect service to resume.
The vast majority of custmers are not affected by this.
Update: The server is back online and running. This is the first time the server has had a problem for a long time but it will be monitored closely.