Wednesday, November 13, 2013

Network Issue - Status and Summary

Update 12/6/2013
We sincerely regret the network outage and various related issues between 11/11 and 11/14.   Our contingency plans did not include an outage of this nature.  We experienced many additional issues while attempting to bring up the sites on a new temporary network.  Our sites had run in the old address space for more than 10 years!  Various IP addresses were hidden in many different places.  Most everything is stable now, but running a little slow.  If you are still experiencing any issues other than sluggishness, please let us know via email.

Earlier this week, we signed an agreement to move our servers into a state-of-the-art commercial data center.  This facility will sharply reduce any chances of failure or outages.  We are also increasing our bandwidth and upgrading our servers.  We will plan another significant outage between December 21 and December 31 at the time of the move.  It should take is 4-6 hours to move and reestablish all services.

Thank you again for your patience and using eWoFF.com




Update 7:30 PM Central time - 11/14/2013
We are starting to make progress with deploying the sites on these temporary connections.  All messaging will be degraded/disabled for next 1-2 days.  We will continue to work with the logs and address issues as quickly as we can.

If eWoFF.com does not correctly resolve to the new address, please access the site at http://207.131.24.100

We will update you on our long-term plans with network and fault-tolerance.  Thank you for your patience and using eWoFF.com

Update 2:00 PM Central time - 11/14/2013
We are now attempting to get the sites going on a temporary connection.  You may see intermittent errors or slowness.  We hope to have the sites functional later today.  eWoFF.com will be at http://207.131.24.100.  The DNS should start to serve this address during next few hours.  You can click on the address link to access the site.

Update 9:00 AM Central time - 11/14/2013
We continue to struggle with our primary network circuit and the alternate.  We hope to have the alternate connection up with limited functionality later today.  We worked with ATT during the night and they are onsite this morning.  The sites will need to be accessed via a new temporary  address that you will find here.  Please check back later today to access the sites from temporary IP address.

We sincerely regret this.  The actions of our network provide are outside the bounds of our expectations.  Please note that servers and data are healthy and there had never been any risk to them.

Update 10:00PM Central time
Unfortunately, we are not yet successful in making the sites available on a temporary network.  We will work through various options during the night and morning.  We will update status as we make progress.

Update 2:00PM Central time
We are unable to get a timeline from our network provider.  We are seeking alternate ways to have sites available.  We sincerely regret the outage.

Update at 9:15AM Central time
The issue is being worked on by our network provider and the Telco.  We sincerely regret it.  We will try to make the systems accessible as quickly as possible.

We are experiencing a network outage this morning.  We are working with the provider to resolve the issue.  We hope to have the sites available very soon.  There are no issues with the servers and data.

We sincerely regret the inconvenience and thank you for your patience.

eWoFF.com, the Together Network

No comments: