Category: Microsoft

My obligatory Bill Gates post.

As everyone has already jumped the gun and spent all week posting about Bill Gates retiring from his role at Microsoft I’ll jump in too. I was going to post this on Friday but hey whaddya gonna do.

I joined Microsoft as a contractor in April of 1994, and was hired full time by the company in January of 1995. So I’m of the Microsoft era where Bill really tipped from being the 60% technological/40% business, to 60% business/40% technological as the company ramped up and exploded in all directions with products and technologies. Oh and went from a small company to a ginormous one. (it’s a word!)

Back in those days Microsoft had about 12,000 employees, but even then the presence of Bill was often felt in everything you did. Even when you’re an entry level support engineer in a satellite site (we weren’t even really called a "Campus" then because we leased two buildings). Bill was famous in the company for his unforgiving nature of mistakes.

Product support had a really outsized role in the shipping of products for Microsoft. Back then, the processes and procedures for shipping software meant that the developer team handed off a release candidate to support, and support would "Sign off" on the build. That "Sign Off" was an actual formal document where support said "We’ve documented the known issues and we agree that this product is supportable and we will be on the hook to support it and the associated costs". So after getting an RC build, support would then spend days "bug bashing it". Anything we felt was a bug that would impact customer satisfaction or dramatically increase support costs was filed and support would withhold sign off (meaning release could not happen) until it was fixed. In the end a contentious sign off process always got escalated to the development and support VP’s to reconcile if development thought support was being over cautious or support thought development was being blinded by ship fever.

Well, a bug was found during testing of a side technology of a product which in this story shall remain nameless.

Development balked as it felt the product issue was an edge case, it got escalated, and the product ended up shipping, but not before we in support studiously documented our projections on impact and our rationale for getting it fixed.

That particular product experienced a record number of support costs due to issues involving this particular technology, way beyond projections. Of course, that type of thing reaches Bill and he called a meeting of the support and development people.

With laser like intensity Bill lit into the support people. How could this happen? Who was responsible? What kind of way was this to run a support business? The support folk, well armed with the documentation from the sign off process, handed it over to Bill. Bill looked it over.

He then pivoted his chair over to the other side of the table and lit into the development people. "How could this happen? Who was responsible? What kind of way was this to run a development process?" Needless to say the dev people had no documentation, and no defense. The support people felt good after that meeting. The Devs, not so much.

Not many people have had practical experience with just how overwhelmingly smart Bill is. He can take a completely new technical situation and dive to the deepest part of it quickly to find out things you never thought of.

When the vulnerability that resulted in MSRC security bulletin MS03-026 was reported (the vulnerability that criminals exploited with the Blaster worm) we determined this was one of those we wanted to make sure the executives were fully briefed on. Mike Nash, then Vice President of the Security Business Unit, wrote up an email summarizing the vulnerability, fix plan, communications plan, etc. to Bill and the executive leadership. That mail was sent on a Sunday afternoon and I was on the cc line as the technical contact/MSRC owner for the bug.

45 minutes later Bill replied to the email (I confess I let out a geek squeal of excitement. Bill replied to an email that I AM ON!). The jist of the email was "I’ve read the technical breakdown and understand it. One question: What happens if [detailed technical scenario involving attack that none of us had considered]"

Bill had, just in a few minutes of reading some high level exec summary stuff, uncovered a deeply technical edge case we missed.

I kind of turned pale(r). My cell phone rang, it was Mike Nash. "He’s right isn’t he?" Mike said. "Yeah." "But as I understand the scenario, the fix still blocks it." Mike said. "Yeah, it does. But I wish I’d thought of it in the vectors section of the mail." I replied.

Mike laughed, "Do you want to reply?" "Uh no Mike, I’ll let you tell him."

So Mike replied and told Bill he was right but the fix addressed that and the scenario didn’t affect our timelines to get this out ASAP. Rochelle came into my office cause she knew I was working this thing and I just sat there gobsmacked. And tried to describe to her that someone eight levels above me was engaged enough at the technical level to turn a problem inside out like that so fast.

You hear Bill is smart, but rarely do you realize the level at which he’s smarter than almost everyone who works for him, at almost every level in the company.

The last thing I will say is that when you’ve been a long time Microsoft employee from the pre-Steve Ballmer presidency, you kind of take for granted at how approachable Bill is. People are often shocked by it. Couple of years ago I was looking into getting a SPOT watch. I liked the idea, and the screen, but it just seemed so clunky and heavy to have on the wrist. Then it hit me late one night reading about them, hey what if you made a SPOT watch that was a pocket watch? The form factor works much better, and the screen could be a bit larger and perhaps you could even integrate touch to it. So knowing Bill was a huge booster of SPOT watches I sent him an email. "Have we looked at doing SPOT watches in the ‘Pocket Watch’ form factor?"

People kind of freak out when they hear that, "you just what, emailed the CEO? Out of the blue?"

Yeah.

You could do that. Not some long winded email about convoluted topics, but a quick idea involving something of interest to the company.

Bill replied almost immediately (at night!) saying yes indeed they looked at the idea, but not enough partners wanted to make them, pocket watches being a niche market, and he cc’d the SPOT watch market research executive to give me more info if I wanted it.

And that was Bill, first and foremost he was a lover of technology. He loves interesting ideas and never wants to wall himself off from them.

I could say I’m going to miss his influence at Microsoft, but that would be remarkably shortsighted. His work at the Bill and Melinda Gates foundation is going to end up changing the world in far better ways than his work here did.

Protip: How to be an effective spokesperson

This is a post I’ve been writing for a while, and finishing up the night at the Washington State Democratic Convention on the eve of the death of someone as good a journalist as Tim Russert was, I thought I might just go ahead and post it.

So, for a long time I was the spokesperson for Microsoft during security response situations, either routine or crisis. This post isn’t really about the particulars about that. Nothing of the below really is meant to describe that time, it’s just meant to show what I learned from my own experience and watching others in similar roles in various venues both public and private.

In short, this is about how to be an effective spokesperson.

Not a good spokesperson, that requires a lot of things I would not presume to say I have or could pass on. I just mean how to effectively communicate what needs to be communicated to express transparency, as well as what needs to be fully understood that WONT be communicated so that you have the full context such that you won’t lie. Intentionally or otherwise.

First and foremost, dive into your beat reporters. Know them well. I dont mean their tone, I don’t mean what a PR agency report tells you about their tendencies. I mean know them as people. Do they have families? What’s important to them? And find this out from THEM. Talk to them, never be afraid to make the time to know them ahead, during, after the interview. I don’t mean waste their time of course, learn this over several encounters. But know it and truly internalize it. Then, be sure you share the same.

Never use that information as leverage, the point is that communications is never without human context. I think Tim Russert knew that very well, which is why anyone he ever covered today is talking about his family and his dad and his life in positive terms. The journalist is not your enemy, nor your friend. They are a human worthy of respect, and the understanding of the context they bring to the topic you are representing is crucial. Know that. Respect that. For any journalist I have spoken to officially to this day I can tell you the saliant facts about them in a positive and respectful manner. Even if I didn’t like their coverage, I can say what makes them a three dimensional person with character.

Crisis communications, in and of itself, is easy. Tell everything you know, and tell it yourself, first.

Such a simple rule. But often it becomes so caught up in "but if we did that we would get sued!" or "if we did that the terrorists would win!" or "if we did that [insert excuse]"

Regardless of the mechanics of the rule, the most important part about having a communications expert is to trust them. They have to know everything. Every decision. Every reason behind it.

I’ve been locked out, and seen others locked out, on the premise (bought into by the decision makers) that "we can’t tell the communications folks yet…they talk to the press!"

If you’re in that position as a spokesperson, immediately communicate that the person who has hired you has just expressed a "no confidence" vote in you. The worst case danger here is that the people who hired you are not interested in true PR and respect for journalists, they are interested in not being transparent (for whatever reason) and making you the mechanism for that lack of transparency. You have to point out to them that you’re the most trusted individual in the process, not the least. Otherwise, you have to eventually quit and your only recourse is a book. And then you just become a regretful whistleblower for profit.

Good communications people and spokespeople insist upon being in the room when the decisions are made that they will have to communicate. And they point out to their clients that they were hired to communicate effectively and with integrity when they might be presented with resistance.

Journalists will challenge you up to a point at the outset, but the Internet and speed of news cycles today means that you could possibly get away with a deception (intentional or not). But for only so long if the stakes are high enough, or the long tail of the topic is long enough.

Here are the warning signs to the fact you, independent of skill, cannot be an effective spokesperson: Crucial meetings on critical public topics you find out about later that didn’t involve you. Messaging you didn’t write and had no hand in creating suddenly appearing as "we need this communicated broadly ASAP", when you push back you are told "don’t worry, this is all lawyer approved, we didn’t want to involve you" or worse: "you are legally bound to say this and only this" when you were never in any discussion with a lawyer.

To be an effective spokesperson, never let anyone that you represent communications-wise cut you out. Period. There is no "letter of exoneration" where you can claim "hey they MADE me say this!" You’re either a communicator who trades on integrity to only communicate for people you trust, or you’re the sucker at the poker table, where you thought it was all fun and games until everyone else figured out you had a weak hand and some money.

Don’t get Intellectually Lazy

Every once and a while any good security professional gets reminded of just how pervasive you have to let your security paranoia be when designing tools and systems.

Case in point, we discovered a minor issue in the tool we created to do Xbox LIVE complaint enforcements. It caused the tool UI to hang when displaying profile field complaints. Gamertags and everything else was humming along just fine. We were kind of scratching our heads over it.

Without going into too much detail on architecture, the tool has a UI component that renders complaint data to our agents so they can review it, and an enforcement component that enforces the decisions. I designed these to be separate since the UI component for profiles and gamertags is basically taking in a complaint stream consisting of a large amount of user created data.

Granted, that data is text only, but my theory was that if anything happened on the UI side, it would be isolated to that side only. The enforcement engine is on a completely different machine and only accepts set limited non-variable input. The UI is rendered for low rights IE 7 and the agents process the complaints on restricted user accounts on Vista. The entire toolset is also isolated from the LIVE service itself.

Yay for me.

Until someone put script in their Bio field and someone else complained about it. The UI design hit script where it didn’t expect to see it and halted. [EDIT: since I got asked, no the script was not malicious in nature, it was just a simple display of a bad word]

Thankfully it only caused our UI for profiles to hang until we (quickly) figured it out. This was a good "Fail safe". It didn’t interrupt enforcements or represent any threat at all to the service since the enforcement tools are totally isolated from the LIVE service, but when we discovered the problem boy was my face red. How many times have I dealt with issues involving trusting input to let something like this happen. Michael Howard would point at me right now and laugh.

So here’s a case where overall design took into account best practices (I knew to isolate variable user generated input) but I only trusted myself to think about the threat model and didn’t define specifically what bad input would consist of. I got a little intellectually lazy, but thankfully thanks to design this wasn’t a threat of any type other than an annoyance. Let that be the lesson I was reminded of so you don’t have to be!

(and yes we now vet the data stream for a variety of nasty text bits)

An open letter to Sven Jaschan

I’d like to say the following to Sven Jaschan:

The attack you released to the Internet disrupted the lives of computer users and administrators all over the world. I know that people have told you all about the large businesses that were disrupted and the amount of money in damage you caused. I can speak first hand to that, helping customers during the outbreak. But here’s how Sasser affected me in my personal life.

In addition to the fact I was working non-stop and practically sleeping in the office trying to protect customers from the attack, Sasser occurred in a time when my mother and step-father were going through a painful divorce. It took time away from my ability to be there for both of them. My wife’s birthday was May 6th and I spent it working instead of the dinner and the night out we had planned. Finally, Sasser caused me to miss what would turn out to be the last few weeks that my first golden retriever, Hennessy, would be alive before we found out she had terminal cancer and had to put her down. Between dealing with Sasser and my wife’s work schedule it was difficult for us to get her to see the specialist she needed to see. And the third week in May we finally were able to do it, only to find out the cancer had already spread too far long before Sasser even happened. I hate the fact that the last few weeks she was alive I was busy dealing with what you had unleashed instead of doing the things she loved to do like going to the park in the time she had left.

I don’t say any of this to tell you I’m angry at you. I’m not angry at you Sven, because people make choices all the time, and sometimes those choices have unintended consequences. But I don’t think it’s wrong to put a human face on those unintended consequences.

I believe you had no idea any of this would happen. And what was going on in my life would have happened anyway, I just would have been able to be there for it.

But other members of my team were impacted, including one who nearly missed the birth of his daughter. Millions of people had things going on in their lives that had to be put on hold while they dealt with Sasser.

So I have some advice, I do think I’m entitled to that much, as is anyone impacted.

You’ve been given a second chance. I hope very sincerely that you use it to serve as a cautionary example, and speak out against the creation of these types of attacks. I know you just want to put all this behind you. But you have the opportunity to do a tremendous amount of good. Please use it, don’t squander it. You’ve taken the first step by confessing and being honest about what you did. I respect that. Please don’t waste the opportunity you have potentially been given to help some kid right now avoid your mistake and think twice before taking what seems to be a simple action, but that can impact so many people.