HostSonic.com /
WebSiteCreations.net

Contact Email: support@hostsonic.com
Help Desk: http://www.hostsonic.com/hd/index.php

Current Status: All servers are on main electrical grid power as of June 19th.

On Saturday, June 5th, an explosion and fire damaged the Houston datacenter which houses our hosting web servers. Our web servers were not damaged and no customer data was lost. Service has been restored with main electrical grid power. The following messages are in order with the most recent message at the top.

The datacenter has switched all servers over to main electrical line power. Our servers are no longer on generator power. There was no downtime during the switch over.

We appreciate the patience which our customers showed during the first 24 hours of this ordeal back on the first of June. Though that time was stressful, the datacenter has proven to be resilient and aggressive in how it deals with major tragedies.

In addition to our backup protocols in place before this incident, we have decided to add an additional off-site backup server to our network. Starting on July 1st, 2008, all of our servers will be backed up and that data stored in 3 separate regions of the United States. This will help to ensure the ability to quickly restore all customer data in the event of a major disaster. Of course, while we do provide this backup service as a courtesy to our customers, it will remain your responsibility to also backup your data. Expect the unexpected and be prepared. You may backup your data at any time by visiting your control panel, and clicking on the backup/restore icon.

Thanks again for your patience. This page will be archived on Thursday and the main link will be taken off of the front page of this site at that time.

The servers remain on generator power at this time as they have been for the previous 10 days. The datacenter is nearing completion of the re-wiring of the electrical room which was destroyed in the explosion and fire back on June 5th. The expected completion date for the rewiring is June 19th. On that date, the datacenter will switch all 9000 web servers (in phases) off of generator power and on to regular electrical power. Though the initial report stated that a 4 hour downtime was expected, it is now being reported that there will be zero downtime during the switch over.

We will provide you with a more detailed time frame for the cut over as soon as it becomes available.

The status of our servers and datacenter network remains good. The datacenter remains on backup generator power at this time. All servers remain up and connectivity is good. We have received very few reports from customers who have experienced connectivity problems and each of those initial reports have been resolved.

If you experience any problems, please send an email to support@hostsonic.com or call us at 888.368.7873. Full contact info for our company can be found at www.hostsonic.com under Contact Info.

The servers continue to operate within normal paramaters. The power coming from the generators seems to be consistent and ample. The datacenter CEO informed us today that the earliest we will be off of generator power will be June 14th. That is one week from Saturday. Aside from a few dns problems early on, all servers seem to be operating normally.

In light of the events of this past weekend, we have tested our backup/restore system today and audited our backup files. Everything is in place for a disaster recovery. We have local backups within the Texas datacenter for easy access and we have off-site backup files located in Georgia for disaster recovery purposes. In July, we will have our 2nd off-site backup system in place. At that time, we will have backups located in three (3) states in 3 distinct regions of the country (Texas, Georgia and Oregon). All together, we have over 900 GB of customer and server data backed up and ready to go at a moment's notice. Not that we expect we will need it, but it is there if we do.

Please let us know if you experience any problems or if you have any questions or comments. support@hostsonic.com or 888.368.7873.

The dns resolvers on server1012 were not working properly, so we have updated the resolv.conf file and restarted the dns software. If you are on server#1012 and have been having mail problems, this should resolve the issue. If you use Outlook or a similar program, you may wish to close it down and reopen it or even reboot your pc for best chance at success of reading the new resolvers.

The datacenter is now powering 100% of servers with generator power. The second floor group (where our servers are located) is running off of permanent generator power (that is, the onsite generators which are present 24/7 for power emergencies). Floor #1 is running off of a mobile/portable generator due to the damage to the power wiring in floor #1.

We continue to see occasional "blips" in connectivity. These short outages last just seconds in most cases and most people do not even notice them. We expect that this will continue for at least another week to 10 days as the datacenter must rebuild the electrical room of the generator.

We have also noticed occasional SMTP time out errors. This is due to the dns servers which the datacenter uses which has had some problems due to a brief period of not having been updated. We expect that situation to improve steadily over the coming days.

All things considered, we are satisfied with the performance of our servers and the network at this time. If you notice any problems, please send an emal to support@hostsonic.com or open a ticket at http://www.hostsonic.com/hd/

Further details from the datacenter have cleared up a few hazy areas we were unsure about. The entire datacenter is on backup power at this time. There are two groups of servers in the datacenter, First Floor and Second Floor groups. All of our servers are located in the second floor group.

The second floor group is being powered by the existing (permanent) generators at the datacenter. Because the damage to the first floor wiring was so severe, the datacenter had to bring in a mobile generator for the first floor group. That generator has since had problems and the first floor group has gone dark again.

We feel very fortunate that our servers are all located together in the second floor group which seems to be having fewer issues than the first floor group.

The problems being experienced by the first floor group are a reminder that this issue is still fluid at the datacenter. While we have had continuous uninterrupted power for well over 24 hours, we feel that we are not out of the woods yet. The datacenter still needs to rebuild the electrical room so that permanent power can be restored to the entire datacenter.

We appreciate your patience during this time. If you have any questions, please let us know.

Current status report. All of our servers are online and have been online since approx. 6am EST on Monday. There are some latency issues (slow), but this is expected to begin getting better on Tuesday morning.

The entire datacenter is running on generator power and will continue to run on generator power for the next 10-12 days. The electrical room (which exploded) needs to be rebuilt. Once all of the replacement parts are in, the electrical room will be rebuilt. At that time, there is an expected outage of approximately 4-6 hours which will take place while they hard-wire all of the servers back onto the main site electrical power grid. A probable date for this will be June 14th (a Saturday).

I am not sure if our customers understand the gravity of what happened this weekend. This incident looked extremely grim. There was some amount of time during which we did not know whether the servers had survived the fire. The explosion completely destroyed the electrical room wiring under the floor of the datacenter. But still, the datacenter was able to get all sites back up and operating on generator power within 48 hours. Considering the scope of the damage from the explosion and fire, this was truly an amazing feat.

A customer asked today if we have off-site backup data. The answer is "yes", we do keep backups of all customer web site and sql data off site for disaster purposes. The servers are in Texas and the backup data is in Georgia. To play the "what if" game, if our servers had been destroyed in the fire this weekend, we could have simply requisitioned a new server, and then upload all customer site & sql backups from off site storage. Fortunately, we did not need to restore any backups at all as our servers were not damaged. When the datacenter powered up Monday morning, our servers came to life and have been doing great since then.

Expect the pace of updates to slow now that things are at least partially back to normal. If you have any issues, questions or comments, please send them via email to support@hostsonic.com or call us at 888.368.7873.

Thank You.

Monday, 7:51 PM EDT

From the data center...

-----------------------
Datacenter bulletin
Late last night, I told you we hoped to have power to the 6,000 servers in Phase 2 of our H1 data center by midnight, with all servers up by early morning. I am glad to say we came close, just a few hours after sunrise. At this time, 100% of our servers in Phase 2 have power, and our technicians are working with customers on any remaining server issues. We are confident all remaining issues will be resolved shortly.

I also explained the significant challenge we faced in the other phase where the actual explosion occurred. Our team came up with a creative way to restore power quicker than the 4-5 day outage. We decided not to wait for equipment for the electrical room completely, opting instead for a temporary solution to get power to the 3,000 servers. That solution involves using generator power for the next 10 -12 days until all the new equipment arrives to rebuild the electrical room for Phase 1.

I explained that we expected to have a temporary solution in place by midnight tonight, with servers powered up tomorrow. The good news is that as you read this letter, the power is restored, and the temporary solution is in effect. Within the next two hours, the remaining 3,000 servers have power. We have overstaffed our data centers again to help during this initial power up. This now leaves us facing step two of this process, which requires getting all of the equipment delivered and then rebuilding the electrical room to its original standard.

To make the cutover to the rebuilt electrical room, the operations group believed it would take a maintenance outage of 24-48 hours. I have good news on that front. It's not perfect, but at present we now believe the maintenance window will be just 4-6 hours. That's still too long, and we will continue this week to find ways to reduce the time. Given that there will be some outage for the cutover, we will execute this step at midnight on a Saturday, either June 7 or June 14. We want to pick the most appropriate time to minimize impact to you.

I must admit that I am amazed. We are almost 18 hours ahead of schedule with this phase, thanks to our great suppliers and of course the great folks working here. There is still more work to do, but the progress is terrific. We will continue to work any and all customer issues, and we face the challenge of putting the permanent power fix in place for Phase 1. Nonetheless, there is still good news based on what I told you last night. As each hour passes, we learn more and more. Please give us the time to continue our planning. We will provide you with information as we have it.

----------------------------------------

Monday, 4:06 PM EDT

The datacenter has begun to restore power to the first floor servers. Of course, all of our servers had power restored early this morning, so this phase doesn't really affect our customers. It is important to note because it is clear that the data center has exceeded its estimates time-wise in restoration of generator power to H1. This is a good sign and bodes well for us all.

We continue to operate off of generator power. Aside from some minor connection issues earlier today, things seem to be running smoothly at this time.

After the first floor servers are powered up, we expect that the data center will put all of its energy into restoring electrical power to the building and turning off the generator power. Considering that the generators run off of diesel fuel and diesel fuel is $4.25 a gallon, I'm sure restoring electrical power to H1 will be their top priority.

Monday, 1:22 PM EDT

Things appear to be stable at this time. We have noticed some latency on the network (slow down) and some reports of momentary downtime lasting less than 4 minutes. It turns out that some amount of fortune was on our side. It turns out that all of our servers were located on floor #2 which sustained virtually no power wiring damage. Thus, all of our servers are back online. There are many web hosting companies whose servers are located on floor #1 which sustained massive wiring damage. Those web host servers are still down at this time and are not expected to come online until late tonight.

There is a dns issue with the datacenter's dns servers. To counter this, we have changed our server /etc/resolv.conf file to reflect an additional set of dns servers outside of the datacenter network. We have received no reports from customers about any connection issues, so we believe the new resolver file settings is working well.

We are continuing to monitor our servers, including hardware such as the hard drives, RAM and cooling fans. Our initial reports show that everything is working well.

If you are experiencing any problems, please contact us at support@hostsonic.com or call us at 888.368.7873 .

Thank you for your patience.

Monday, 11:56 AM EDT -- support@hostsonic.com

It appears that our company email is again resolving and working fine. You may begin contacting us again at

support@hostsonic.com

If you do receive an error message, revert to our off-network email which is onlinesupport@email.com .

Our servers remain on generator power so I caution everyone to expect a few intermittent outages as the datacenter works to restore permanent power to the H1 datacenter.

Thank you all for your patience.

Monday, 10:38 AM EDT -- Email

Because datacenter DNS servers were affected by this downtime, you may experience problems connecting to your email account(s) if you use Outlook or other similar email programs. If you believe this is the case, then you can temporarily use our web mail system as we have verified that the web mail system is catching and sending emails fine.

FAQ How to access your web mail system
http://www.hostsonic.com/faq/index.php?op=view&t=105

Monday, 9:28 AM EDT

All of our servers are back online and operating. Our email @hostsonic.com continues to be troublesome but this is due to our dns switchover for the hostsonic.com domain on Saturday in order to have our site off network temporarily. If you need to contact us, call us at 888.368.7873 or email us at onlinesupport@email.com .

While progress has been made, please understand that all of our servers are currently on generator power at the datacenter. Expect intermittent downtime for the next few days until the datacenter can restore power completely. Additonally, there may be dns resolver issues so you may experience odd behavior with outgoing and incoming email until all dns issues have been resolved.

Please report any issues via email to "onlinesupport@email.com or call us at 888.368.7873 .

Monday, 8:50 AM EDT

From the datacenter...

---------------------------
Datacenter Bulletin
As we continue to restore power to customer servers on the second floor, several customers have reported intermittent losses of connectivity. This connectivity loss is due to the balancing of network gear in the data center and is unrelated to power. This should improve soon.

Monday, 8:27 AM EDT

All of our servers seem to be responding fine at this time. However, you may notice continuing intermittent outages throughout the day and coming days. It is still early, but we are happy that significant progress has been made. Below is the latest from the datacenter.

---------------------------
Datacenter Bulletin
We've made significant progress in restoring customer servers on the second floor (phase 2) of H1. The data center staff is still in the process of verifying that servers booted appropriately and are troubleshooting any that have not yet come online. Per Doug's earlier message, we are still on target to restore service to the first floor (phase 1) by this evening.

The datacenter has just reported the following news. It sounds like progress. Good news. Still much hard work to go. Still several hours (minimum) away from the first servers coming online. But progress none-the-less.

-----------------------------
Datacenter bulletin
After the fire marshall inspected the H1 location, we were given the green light to bring power back to the facility. The generators have been turned on, and we are receiving power on the second floor. The generator power restoration is the first step in the full restoration of service to the data center. From here, we will begin the process of cooling the DC floor, which could take a few hours. As soon as the power integrity is confirmed and the DC floor is ready for operation, we will be restoring power and checking server hardware on a rack-by-rack basis.

There is some amount of good news. All but one of our servers is located in Phase 2 which means they are in the part of the datacenter which is least affected by the fire and will be among the first servers to come back online. If you are not sure what I am talking about, then please read the next post down from 12:27 AM EDT for full details. The single server which is in Phase 1 will come online last and will be on generator power for about a week once it comes back online.

Monday, 12:27 AM EDT

The datacenter has finally released new information. There is some good news and some bad news. We are not sure whether our servers are located in Phase 1 (downstairs) or Phase 2 (upstairs). If we are in Phase 2, then our servers will be among the first to come back online as early as early morning Monday. I stress, that is just a guess as this situation is very fluid at this time. If we are in Phase 1 (downstairs) where the most damage was done, then those servers will not come online any earlier than Monday Evening. Please read the full comments below which are from the CEO of the data center.

---------------------------
Datacenter Bulletin

As previously committed, I would like to provide an update on where we stand following yesterday's explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours. We are acutely aware that uptime is critical to your business, and you have my personal commitment that our team will continue to work around the clock to restore your service.

As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed. There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running. We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

Douglas J. Erwin
Chairman & Chief Executive Officer
------------------------------------------------------

One of the affected web hosts took some pictures outside of the Houston datacenter today and then shared them with the affected community. No pictures of the damaged inside of the building have yet been released. Click on the photos below to view full size.

Sunday, 9:15 PM EDT

The last communication from the datacenter has shed no light on this issue. As we suspected, this looks like an extended outage. Because we have not been provided any actual details in the previous 8 hours, we can only assume that the status of the datacenter powerup is as it was before, still not completed.

We are working to get more details. We are very grateful to you for your patience.

Sunday, 8:21 PM EDT

Communitcation from the datacenter has been sparse for several hours. We are awaiting their latest news concerning the powerup and testing. It still appears to us that the situation remains unchanged without a set time for the servers to go back online. We will continue to keep you posted as soon as any new info becomes available.

Sunday, 5:46 PM EDT

We are approaching the time (5:00 CDT) of the initial power test in the affected data center. We are awaiting communitcation from the datacenter about the status of the test which we expect sometime after 6:00 PM EDT.

Our last communication with the datacenter shows that they are continuing to target the 5pm CDT as the first test of the power. If that goes well, then testing will ensue which will last for approx. 4 hours. If testing goes well, the earliest server powerups will begin directly after.

We thank you for your patience.

Latest note from the datacenter.

Sunday, 9:09 AM EDT

Latest note from the datacenter.

Sunday, 2:29 AM EDT

The datacenter has told us not to expect any new information until tomorrow, mid-morning. They have assured us that they are already hard at work trying to repair the components that were damaged in the explosion/fire. I remind you that if you would like to ask us any questions, please send an email to our off-network email which is onlinesupport@email.com . Due to a high volume of emails, we may not be able to respond to your email quickly, but rest assured that we will respond asap.

Please understand that this is an extended outage. If you have not yet done so, please read the previous posts below for full details.

We appreciate your patience.

Thank You,

Bill Micke

Sunday, 2:03 AM EDT

The datacenter's early projection is that "some" servers may begin coming online mid-afternoon on Sunday. Please understand that this is an "initial" assessment and may not prove to be accurate. The truth is that the incident sounds bad... very bad. The power to the entire datacenter was knocked out due to an explosion and fire. In addition, the fire department will not allow the datacenter go to backup generators at this time. The good news, however, is it seems that none of the several thousand servers in the datacenter were affected by the fire/explosion. So, as you can imagine, though this is a trying time for all involved, things could have been much worse.

The data center which houses our company servers is a multi-million dollar company. They have the resources and clout to get the job done in regard to restoring service to its thousands of customers (each of their customers is a company like ours which hosts thousands of individual web sites. Literally, hundreds of thousands of customer web sites are down due to this terrible tragedy). On previous issues which were much smaller in scale than this one, we found the staff and management of the EV1 datacenter to be exemplary. We expect only the best out of them during this terrible time also.

Please understand that this may not be a short outage. Though the initial estimate is mid-afternoon on Sunday for the first servers to come back online, it may extend beyond that. We just don't know at this time. Though we remain hopeful, early estimates may prove to be wrong. We will keep you posted as soon as we have more info.

-- end

Sunday, 1:25 AM EDT

Another note from the data center.

-----------------------------
Datacenter Bulletin
As you know, we have vendors onsite at the H1 data center. With their help, we’ve created a list of equipment that will be required, and we’re already dealing with those manufacturers to find the gear. Since it’s Saturday night, we do have a few challenges. We are prioritizing issues as follows:

1. Getting the network up at H1 is first and foremost. We’re pulling components from our five other data centers – including Dallas – which will be an all-night effort.

2. Getting power back to the data center is key, though it is too early to establish success there.

3. Because ServerCommand is in H1, our legacy EV1 customers are blinded about this incident. We are in the process of moving the ServerCommand servers to other Houston data centers so that we’re able to loop them into communications.
---------------------------------

Just a note, we are a legacy EV1 customer and thus, so are you. When they have finished moving their servercommand servers, then we will hopefully have at least established a better communication between the datacenter and our company. As it is, the datacenter is using a third party web site to communicate to its affected customers.

This is a major outage and resolution will not be quick. Please understand and pass that information down to your customers as you feel neccessary.

More coming as soon as we have new info.

Thanks,
Bill

Dear Customer,

The data center just sent out this message to all of its customers.

----------------------------
Datacenter Bulletin
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
------------------------------

We will keep you informed as soon as we know something more. We are very grateful for the extreme patience that you have shown.

Dear HostSonic/WebSite Creations Customers,

As you may have noticed, our main site is down and your site may also be down also. We have performed an emergency re-route of our DNS to a different web server so that we can post this message at our company web site address.

W-e have just been notifed by our data center in Houston, Texas that there was a fire in the data center that hosts several of our web servers. This data center also hosts hundreds of other servers for other web hosts.

The first post-emergency assessment from the data center reveals some good news. First, no one in the data center was injured by the fire. Second, the fire was small and it was contained quickly. Also important is that no web servers appear to have been damaged by the fire. However, power remains out at the data center at this time and they are assessing the power situation at this time. The data center does have massive backup generators, but we have had no word on the viability of the backup generator system at this time.

Because our ticketing system and email system were located on the servers located in the damaged data-center, we will use this page to communicate with you until this crisis is over. We will continue to update you throughout this crisis.

If you need to email us, we have an off network email you may send to.

onlinesupport@email.com

With Kind Regards,
Bill Micke
www.hostsonic.com