“I’m probably pretty typical for someone of my generation who went into technology”, begins Lipner, as I ask him to talk me through his career, starting at the very beginning. Typical in education, perhaps, but certainly not typical in terms of talent – or CV. Steve Lipner is the partner director of program management in Trustworthy Computing Security at Microsoft, which certainly can’t be defined as ‘typical’.
“I grew up in the Southwest of the United States – Texas and Arizona – and went to public schools”, Lipner recalls. He tells me that he considers his role as an amateur radio operator in high school, combined with his uncle’s job as a chemical engineer, to blame – or to thank – for his initial interest in technology.
Lipner majored in civil engineering at MIT in Boston, in a department that was doing innovative work in the application of computers to engineering products. “By the time I was in graduate school, I was managing a small software development group”, Lipner remembers. “It introduced me to building software beyond what one person could do, and to the challenges of managing a small group.”
When he left college, Lipner joined a US defense contractor – the MITRE Corporation – that worked mostly for the Air Force. “After a fairly short period working for them, my manager asked me would I get involved and lead their efforts in trying to understand how to make computer systems protect the security of classified information?” At the time, Lipner was nervous that he did not have the right expertise, so he agreed to take on the project only “until they could hire the right person”. As it so happens, Lipner was the right person. “I started that assignment more than 41 years ago, and I’m still waiting for my relief to come”, he laughs.
Looking for Something More
Only recently, Lipner emailed his department a scanned copy of the first paper he ever wrote on cybersecurity. “We looked at mathematical models of security, some of which are still in use, and we looked at using virtual machine systems for protecting sensitive information, which is a concept that you still run into a lot when people are trying to build high-security systems”, he said of the content, which is still relevant today.
The main difference between then and now, he considers, is knowledge. “In 1971, we understood less about what we didn’t know, and we were building systems that were much simpler”, he notes. “They were more primitive and less usable.”
In 1981, Lipner decided that non-profit defense contractor environments were not for him, and for the next eighteen years, Lipner worked for Digital Equipment Corporation – “the big mini-computer maker of the Seventies and Eighties” – and a company called Trusted Information Systems. At the latter, Lipner “worked on building their firewall business, which was sort of fun, because we were actually able to build a product that people wanted to buy”. Lipner considers the product both a business success and a “pretty good security product for its era”.
During a stint in the late Nineties, when Lipner worked briefly for a government contractor while he “looked for something more interesting to do”, he received a call from a friend who worked for Microsoft. There was an opening for a Microsoft security response job. At this point, Lipner concluded that he wanted to stay in security, and had “grown tired of waiting for my replacement”.
Completely Naive Approach
Lipner was hired to take over the Microsoft security response team, which, back then, was him and one other person. “We changed the name to the Security Response Center in 2000”, he recounts. “We started to add more people, and build more mature processes.”
By the summer of 2000, the security research center was made up of five or six staff, “and we started to build some of the processes that you’re now familiar with – releasing updates on a regular cycle; having a severity rating system; and trying to build relationships with the security researchers who found vulnerabilities in our products.
“In 2001, a security program management position opened up. It came with responsibility for some of the Windows security features, and for what we called the Secure Windows Initiative.” Lipner was offered the position, and accepted. The Secure Windows Initiative was formed with two or three people, and was supposed to review all the code in Windows, find security bugs, and then file bugs so people would fix them.
“Of course, that was a completely naive approach”, Lipner admits. “So we started to change and focus on training and developing processes that power groups could apply.”
Despite taking on his new role, Lipner retained responsibility for the Security Response Center – managing the response for Code Red, and Nimda, which he describes as “pretty awful and chaotic”. A management team meeting to determine how to put Microsoft on a path to fix this took place later that year. This meeting, Lipner recalls, was a game-changer.
“One of the other product groups – the Dotnet framework group – had security objectives, a managed code framework. They decided to do nothing but security reviews and security testing until the product looked good enough to ship”. It was decided, Lipner remembers, that the version of Windows under deployment at the time – Windows Server 2003 – may benefit from the same game play. “It was an enormous development, but we couldn’t think of anything better”, he says.
After taking this idea to his director, Lipner soon found himself pitching to his boss and the vice president. “By that time, nobody had ever said yes, but nobody had ever said no, and we’re suddenly shutting down Windows and scheduling sessions to train everybody in the division.”
We Made Some Mistakes
This fundamental change in direction was coincident with Craig Mundie working with Bill Gates on the Trustworthy Computing commitment. “So we organized the Windows security approach. We shut down the Windows division for February and March of 2002. The thing people don’t talk about is how chaotic it was”, Lipner remembers. “We were one step ahead of the developers at every turn. We had threat modeling practices, but they were immature and pretty awful. We had tools and we had some testing processes. We made some mistakes; we did some things right.”
By the autumn of that year, “we had made changes to the product and it was more secure”, Lipner asserts. “We’d found and fixed a lot of bugs, and that set us on the path to making the security piece of the computing promise real.”
A meeting with the company’s senior leadership team in 2004 led to the conceptual creation of a security development lifecycle. It was decided, recalls Lipner, that we would “make it mandatory for essentially everything that the company did, and would do forever. It was a meeting that took less time than was allotted on the agenda, and Steve Ballmer said ‘we’re going to do this, and we’re not going to discuss it anymore’”.
| "If I have a good day, I can make life a little better and a little more secure for hundreds of millions of Windows users" |
Lipner remembers walking out of that meeting saying: “well, I think we just got what we wanted”. It went live as scheduled, July 1, 2004, first with a six-month update cycle, later changing to an annual cycle. “We’re currently on the Secure Development Lifecycle (SDL) version 5.2, and working on SDL version 6”, Lipner advises. “We’ve made the processes and the tools more mature, and we’ve made the training better”.
Lipner is, quite justifiably, immensely proud of the SDL, and this is apparent in the way he talks of its creation and evolution. I later ask him what his greatest achievement is to date, and he answers “without question, SDL and the impact it has on Microsoft, and the whole industry”.
It may go down as his greatest achievement, but Lipner does not overlook the other things that he and the security group at Microsoft have done along the way. “I’m proud of what we’ve been able to do internally, but I’m also very pleased that we’ve taken what we have and made it available externally for customers and partners, and anybody else who wants to do secure development.”
Lipner says he has colleagues who think he has “sold out” because he works on “making these rich, complex products secure”. He counteracts this, saying “If you build something that’s so simple and primitive that nobody will use it, and you make it secure, it’s not clear you’ve done anything for anybody.”
Far from feeling that he has “sold out”, Lipner says, “If I have a good day, I can make life a little better and a little more secure for hundreds of millions of Windows users. That’ll keep me coming to work in the morning.”
Changing Perceptions
Of all the interviews I’ve had with Microsoft employees, Lipner is the first to speak honestly and openly about the history of bad press Microsoft has when it comes to security. “People tend to look back, and it takes a while to overcome a negative history and negative reputation”, Lipner admits. “It’s certainly true [that there was] not a lot in the way of security consideration in initial versions of Windows. People remember back to those days, and it just takes a long time to change perceptions.”
“There was a lot of embarrassment and a lot of customer upset”, Lipner recalls, referring to Code Red, Nimda, and later, Blaster and Slammer. “People at Microsoft take a lot of pride in what they do, and are very business-oriented”, he says. “It was not acceptable to not address the problem.”
Lipner confirms that it certainly wasn’t coincidence that Microsoft’s different security initiatives and products were born simultaneously at the beginning of the twenty-first century. “There was a small group reporting up to Craig Mundie that became Trustworthy Computing, who we worked pretty closely with. We all knew about September 11th, and people’s concerns about security, including more cyber-attacks”.
Despite having different objectives and different responsibilities, the Windows group worked in parallel with the Trustworthy Computing group, “proceeding towards a common end and a common strategy”.
Of course, the road hasn’t always been smooth, and Lipner is the first to admit that finding approaches that would work took time, and that mistakes were made along the way. “A lot of what we’ve done has been about trying things, seeing what works and building on that”, he relays.
| "I wanted to be a pilot in the air force when I was a kid" |
Looking back at the Windows Security push, Lipner considers trying to train people to be penetration testers as one of their biggest mistakes. “Penetration testing is good, but you don’t want every developer to be a penetration tester, because most of them won’t be good at it”, he says, honestly.
Lipner references the “fuzzing of Windows Office” by researcher Dan Kaminsky as a measure of the progress Microsoft has made. “The number of bugs that ran to almost a hundred in Office 2003 was down to seven in Office 2010. We look at those things internally, and say, ‘yeah – we’re making progress’”.
Making a Difference
Despite his already significant dedication to Microsoft, Lipner still considers that he has work to do. “I’m still focused on SDL and making it even more effective for cloud online services and devices”, he says, looking forward. “I’d like to develop some prescriptive things that enable us to do a better job building software that’s more resilient to social engineering and where the security is more usable.”
He certainly has enough on his plate to keep him busy and challenged, then. That’s on top of his daily responsibilities, which he describes as “like any other manager of a group of that size”, listing reviewing documents, hiring loops, and conference calls as a regular part of his day-to-day duties.
Extra curricular interests see Lipner as chairman of the board of Safecode – an industry association of vendors devoted to doing a better job of building secure software. Microsoft is also sponsoring a Security Development Conference in Washington in May. “It will be an industry event, focused on secure development practices, to bring folks in the industry together to share best practices”.
When Lipner eventually moves on, his will be very big shoes to fill, and I ask him what advice he’d give to his successor. His advice is specific: “Be very rigorous about insisting that the things put into the SDL are actually going to be effective. Microsoft has upwards of 30,000 people in development roles, so that means if you tell all the developers to do something, you’re putting a lot of effort into it. So it had better make a difference”.
Lipner is so passionate about SDL and his crucial role at Microsoft, that I can’t possibly imagine him dedicating his career to anything else. Still, I ask him what path he may have chosen had he not gone into IT and security. His answer? A pilot. “My eyes wouldn’t have made it, but I wanted to be a pilot in the Air Force when I was a kid”, he says with a hint of nostalgia.
Having spent an hour with Lipner, though, it is clear that this role – for now at least – is exactly where he belongs.