Email Outage Update 1am, ET, December 16th

Could you please give us some idea what's going on? If a security breach didn't cause this, what did?

lets see now, for months i have been getting daemonmailserver spam, that about 3 months ago started using the names of my friends and family, although from different email addresses, but you contend that we were put at risk. i am officially call ya'll a bunch of liars.

Why are we being kept so much in the dark about what is going on, and what's causing this?? Cox owes us some answers!

"we are actively exploring multiple approaches to restore email services"

That means they have no clue or something catastrophic has happened, I take it.

Still exploring multiple approaches to restore email services. They still don't know what's wrong. I wouldn't count on any emails being received till Monday and you might not get any that were sent while the servers were down.

I feel that Cox should give us all a credit toward our bill for this outage of more than 24 hours now!! I,  for one,  get my job assignments via email and I cant get anything right now!!

I called about a credit today and they said they dont issue credits for email problems because email comes "free" with the internet connection....DUH...we are paying for the internet connection, so what is free about it?? There should be some type of credit for OUR troubles!!

I called Cox last night (Friday) and requested a credit on my bill for this e-mail outage, and they were kind enough to issue me a credit of $4.33 for the first full 24 hrs. of e-mail outage.  I would suggest you call them again and demand a credit.  They have no right to turn down your request for a credit.  In fact, it is their duty to automatically issue you a credit for any sort of outage.  After all, we pay quite the premium price for their services!

Sunday night update:  I again called Cox to get another credit and they were very kind and helpful.  They credited me another $5.00 for my second full day of e-mail outage (Saturday).  So I can honestly report that they've been very apologetic and understanding whenever I've called.  While I am very disappointed in this outage, I do understand that sometimes things can go very wrong technically.  And while I do wonder why they didn't have backup servers to handle this type of situation, it is nevertheless a situation in which I don't have all of the details.  So based on the fact that they've given me very good customer service, I choose to forgive them.  And as of about an hour ago, I have begun receiving some e-mails (although I'm sure not all of them).

Dear Frankie,

Have you heard of a DR/BC plan (Disaster Recovery and Business Continuity Plan)? I mean really....When I ran Feist Internet we had full metal backups of our servers as well as standby hardware (that was over 15 years ago).

In this day and age of virtual machines via VmWare, Storage area networks with remote data replication and redundant data centers...something as important as email should never go down.  (I know you say its free....that is BS as much as saying DNS is free...technical yes...but intellectually dishonest). Since Sendmail seems to work..but POP access is down I assume your SAN/Server environment went tango uniform....shame on you. You should have had a replication of the mail spool and redundant servers. If it is an AD problem then you should have had a backup of the system state.

You can BS your normal customers...and some will buy this load of fecal matter...but a 23 year vet of IT is calling this a poor setup.

BTW there really is no excuse .. even a DDos attack is not an excuse as you can take care of that ** at the RDC gateway or the NAP. 

This is very unprofessional. You will lose customers over this.

W

You're guarunteeing that any and all emails attempted to be sent to me over the past two days will be delivered right?  Because you haven't warned me that it's possible I won't get my emails, which I get about 30 a day, all job related, all extremely important.  Correct?  You're guarunteeing that correct?  Great.  Thanks, appreciate that.  

Your lack of communication and utter failure to rectify this problem in a timely manor is completely unacceptable.  You better be offering credit to every single one of your customers that were affected WITHOUT them having to request it.  If you care about your customers satisfaction, that's exactly what you'll do.  

 Dear Santa,

      I am sure you already know that I have been super good this year.  No porn, no stealing of movies off the internet, no hacking and I even paid for my internet with legally made money.  Unlike last year when I paid for it with procedes from selling homemade nuticles, which is what they put into male dogs after they get chopped at the vet!  You let it be known that I was bad by the lump of coal you left me and for that I am sorry.  But this year is different, so can I ask a favor?  Can I have my email back quickly?  Please Santa, make the demons get out of the Cox servers and allow me to track my packages coming from Amazon, oops I mean the North Pole.

      I guess I would be alright with this happening at any other time in the year!  I'm a realist and know issues are going to happen, but IT'S CHRISTMAS!  I have 2 package tracking numbers coming from Bejing, China on my email and have no way of knowing if they will make it in time for that special day!

      Guys/girls, get on the ball!  It's time to leave the Christmas Party (even if you're drunk) and get this problem resolved!  Elvis has left the building on this deal!  After the problem is fixed, go to all accounts affected and issue a credit without us having to call for one.  We pay premium prices for your service and I am happy with it, but from time to time when concerns arise, it is on your behalf to give us some of that money back without us being like a little child asking for a little extra allowance.

     Happy Holidays to all and to all a good night!

Ho,Ho,Ho,Ho

Could be worse.

The server could be physically located in Colorado or Washington.

Nuticles are an abuse of clackers.

As far as Cox losing customers who else could we get for internet service.  Here in Warner Robins, GA they have a monopoly, no competition.  That is why I pay $53.00 a month for internet service only.  So they don't care about all the whining about poor service.  They know your payment is coming next month.  They know once this is fixed we will all shut up.  So they post their little updates hoping to please us.  They know just like us something catastrophic happened and they don't want to give us the full details.  How come the media isn't on to this? 

YES!!! Then they would be drunk from the Christmas Party and stoned!  Awesome thought!

      Just a quick question, please?  When you say that, "we sincerely apologize for the inconvience", does that mean sometimes when you apologize, you're not sincere?  If everyone was sincere everytime they apologize, then there would be no reason to put "sincerely" in front of it.

      Here's one for you: "I sincerely want to pay my bill this month, but I won't be able to!", now that insinuates that I was not sincere when I paid my other bills from the past months.  How about, "I sincerely don't want to switch my internet, phone and television services, hope this doesn't cause you any inconvience!"

     My last crack is this:  I SINCERELY want you to give me a credit this month. 

     You've been sticking me with your COX for many months now and I have been paying you for servicing me, but this month your COX won't work has left me unfulfilled and unsatisfied.  I need you to break me off a little piece to keep me happy.  I admit, I am addicted to your COX but after you're finished with me, I do expect a pat on the head.

Read this for causes of email outages:

http://www.disaster-resource.com/index.php?option=com_content&view=article&id=195%3Aunderstanding-tecnical-and-other-causes-of-email-outages&catid=6%3Ainformation-technology&Itemid=12

We can still login, just get a blank screen.  And test emails sent to my account don't bounce back. So our account ID's are still there, and the application thinks it's still there, there's just no data when we login.

They are looking at multiple solutions, so that points to one of the more difficult issues.  If they had a backup system, it's corrupted too (or inaccessible) otherwise it should have been up by last night, as they originally stated.  Database and SAN failures can cause data corruption to be replicated to backup servers.  This means they have to go back to tapes.  This is p-a-i-n-f-u-l-l-y slow.  If the tapes are corrupted (which can happen), then they have to pull an older second set (if they have one), or try to manually pull data off the tapes, which takes even longer.

If it is a data issues, what they could do is at least restore our ability to get new emails, and then work on the history.  Most of us have our email history stored on at least one device.

- Lisa S., CBCP (Certified Business Continuity Professional) and DR/BCP Specialist