T-Mobile, Sidekick & Microsoft = Danger – Updated 2x

Let me start with a note from T-Mobile:

“Sidekick customers, during this service disruption, please DO NOT remove your battery, reset your Sidekick, or allow it to lose power.”

If you haven’t heard, there’s trouble in Sidekick City. The data for T-Mobile’s Sidekick users is stored on the cloud servers of a company called Danger, or I should say was stored there. This weekend, a server glitch (we’ll get to that in a minute) caused an almost certain loss of Sidekick users personal data. That means that contacts, calendar entries, photos and to-do list entries for a lot of people are gone. Did anyone at T-Mobile look at the company’s name? I mean storing important customer data on a company called SOL would not be allowed, right? But Danger is OK simply because it is Microsoft/Danger?

Microsoft acquired Danger in 2008 and, according to PCWorld,  their position is that it was Danger’s technology that failed and not Microsoft Technology. I don’t think there are a lot of Sidekick users thinking that makes it all OK. Most companies have backups, but it appears the company called ‘Danger’ did not. The T-Mobile note continues:

“Regrettably, based on Microsoft/Danger’s latest recovery assessment of their systems, we must now inform you that personal information stored on your device – such as contacts, calendar entries, to-do lists or photos – that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. That said, our teams continue to work around-the-clock in hopes of discovering some way to recover this information. However, the likelihood of a successful outcome is extremely low.”

Someone at Microsoft must have wondered “could you have just said Danger instead of Microsoft/Danger”. Not good.

So how could this ‘glitch’ happen?

A failure of basic network standards. Hiptop3.com (via PCWorld) has an interesting post that suggests a SAN (Storage Area Network) upgrade went wrong. Yep, that would do it, but that’s what you have backups for (unless you are Microsoft/Danger)! Most reports suggest Microsoft failed to do a good backup prior to messing around with the SAN. I don’t know if that is the case, but what about archival copies? Are you telling me there was NO backup? I understand there was no backup specific to this project, but what about past incremental, differential or full backups? Tapes? An autoloader in a closet? A frigging CD?

I am not surprised someone thought this project was safe, and that a pre-upgrade backup was not worth the time. I have seen people take that risk because they figured they had recent backups in the event of an emergency. What is the most shocking to me is the fact they are saying it is gone for ever. That says there are NO backups in existence. No daily, weekly, monthly, quarterly or yearly backups at all. How the heck is that even possible?

There is one benefit from this mishap. Really makes you think about this cloud thing. We will never really know how well these networks are being managed and maybe it will teach end users to backup. Those Sidekick users angry about this might just start to think about the photos and email on their computer. What if any other cloud based service you use lost data? If Gmail died, would you be OK? And what about your local stuff?

Please backup. It does not matter if it is to a CD, DVD, Tape, Disk-to-Disk, or an online backup service, but backup. A local backup helps for quick restores and offsite backup for disasters.

For the Sidekick users, your current backup is on the phone. So, like they said, DO NOT let it power down because it is the only copy.

Update:

On October 12th T-Mobile released an update. They are now saying that:

“…recovering some lost content may now be possible.”

This part is funny:

“In the event certain customers have experienced a significant and permanent loss of personal content, T-Mobile will be sending these customers a $100 customer appreciation card. This will be in addition to the free month of data service that already went to Sidekick data customers. This card can be used towards T-Mobile products and services, or a customer’s T-Mobile bill.”

$100 card for use on T-Mobile products and services and a free month of data service? Ok, for what data! If they do get your data back, then the $100 may not apply to you. But you do get that free month of data! My question is who starts with the worst scenario and lets it get better? Why would ever say it may be gone for good and later say ‘well, maybe we can get it back’. Don’t announce failure until you fail. If I were a Sidekick user, which I am not, I would be gone. I would want out of any contract and I would find another good messaging device.

Update 2:

It appears that now Microsoft is saying that they may recover “…most, if not all, customer data…”. Amazing. So, some employee had to go through the pain of telling customers data was gone forever, when it was not? Tech support had to field angry calls and explain the data loss was permanent was it was not? Gotta be a better way people.

I heard Leo Laporte make a comment about this and he made a great point. He recognized that the T-Mobile and Microsoft stepped up to deal with the issue. Of course, he also pointed out they had little choice since the Sidekick world was at their door. My problem is the path they took. Data loss happens, hardware failure happens and human error happens. Shit happens. But you don’t tell customers they are screwed until they are. Admit the failure and reserve predictions until you know the real facts. An enterprise backup system is supposed to include disaster recovery and continuity planning. In this case they should have been able to say, from day one, that the data could be restored. Maybe over a long period of time, but restore should have been the first assumption.

One Response to “T-Mobile, Sidekick & Microsoft = Danger – Updated 2x”

  1. Mark, Sr. says:

    From the rumor mill:
    The name of the company that Microsoft/Danger is using to recover the files? . . .

    FailDisk™!