I’m late to the party with this post–it’s kind of funny that I’m late, or maybe just sad, as you’ll see–and anyone who cares has already seen the information in a dozen other places, but today we released the Entity Framework Beta 3 and the Entity Framework Tools CTP 2. You might expect to hear about the new features, but Danny Simmons has already done a far better job than I could. Maybe I could tell you about breaking changes since Beta 2. But Jeff Derstadt gets the honors there. Maybe there’s a big picture I could paint, but the press release takes care of that.

Since I’m not breaking any news, and I have nothing obvious to add, you might wonder why I bother. I should have something better for those dedicated handful of you who are still here after all this time. So I’ll tell you a story about how we take a finished software product and get it up on the Web for people to download.

I said that we released the Entity Framework Beta 3–and I’m going to talk about just the runtime since I am in the Data Programmability Runtime team and it makes the story easier to tell–but, to take a very narrow view of what it means to “release the software,” we didn’t do anything: I did. I pushed the button that made the Entity Framework live on the Microsoft.com Download Center.

This release process started a few weeks ago before all the bugs were closed and all the tests were finished. Everything that we release to the public has to meet a bunch of criteria that have nothing to do with the functionality of the product. Some of the requirements are driven by Microsoft’s legal division, LCA. They require things like geopolitical approval (this focuses on the text and graphics in the product–it’s tougher than you think since something that’s innocuous in one country is sometimes illegal in another), an appropriate EULA, and proper diligence around any third-party code, among a lot of other things. There are a set of tools that must be run against the source code and the compiled binaries. Some of these are part of a final security check and others validate compliance with Microsoft policies.

There are a lot of tools to run, a lot of boxes to check, and a lot of people to round up to sign off on things. And, remember, none of this has anything to do with the testing and bug fixing that goes into the product itself.

There is an internal tool called the Download Management Tool that allows you to build the “download details page”: the page you see when I point to the Entity Framework Runtime Beta 3 that has the overview, the requirements, the instructions, and the actual link to the setup programs themselves. The DMT is a pretty slick tool, but there are a lot of boxes to check and fields to fill out. And, of course, someone has to write the text that goes into that overview, the requirements, and the installation instructions, and verifies the links in the text and a number of other things that I’m forgetting right now.

When you first set up a release in the DMT, it reserves a destination URL on the live site for that release. This helps people who need to prepare documentation, press releases, blog posts, email announcement messages, and even magazine articles. But those Download Center URLs are long and we might need to do something that would change the URL, so there’s another tool called the Forward Link tool, or FWLink. If you have spent much time with the Microsoft Web sites, I’m sure you have seen them, They look like http://go.microsoft.com/fwlink/?LinkId=104981. They exist to redirect requests. So you can reserve these FWLinks early in the process before their destinations even exist, then use them in all the documentation and only point them to the correct location once that page or that file is online. We have them not just for the download, but for the readme, the documentation, the samples, and even the ADO.NET forum. Among others. When there are so many, it’s tough to keep track of all of them, so that’s one more checklist I have to maintain.

Once all the policies have been met, all the FWlinks are set up, the download details pages written, and the readme compiled, written, edited, and reviewed, it’s time to start the final packaging of the product. Of course, that’s after all the tests are passing and the team has signed off on the product.

The final packaging starts with digital signing. Digital signatures are a security measure so that customers can be sure the code they’re running actually came from Microsoft and hasn’t been changed somewhere along the way. We have a two-pass process: we first sign the binaries that make up the product itself, then we insert those signed binaries into the setup program and we sign that. Microsoft takes its digital certificates seriously and any signing requires at least three people with the appropriate permissions: one person to submit the “job” and two people to approve it. We can do it all over the network now using our smart cards. A few years ago we had to take our product code to another building on CDs or floppies where release services would sign the bits and write them back to removable media. It’s definitely better now.

Once things are signed, the team does a final verification to be sure that there were no mistakes in the act of reassembling and signing the setup. I then take the signed, verified package and post it to the Download Center. This is done in “undiscoverable” mode: there is no details page, just the executable available at a hidden URL. The test team does one more verification from this URL and, once they sign off, the release is ready to go live.

There is a whole host of other activity that happens in parallel and involves marketing and public relations who put together material for press releases, coordinate with partners, with the media, and a whole host of other activities. The UE team has to post the documentation. The MSDN Web site team has to update the developer centers. We all agree on a day and time when we will make everything available to the public and that’s when I press the button and release the product.

P.S. I forgot to tell you about gathering the samples and posting them on CodePlex. Maybe some other time.

Happy day today: the ADO.NET vNext CTP is available. We have worked hard to get this out the door and we’re anxious to know what you think. You can discuss it at the ADO.NET Technology Preview forum on MSDN.

Alex has been bugging me to write about the experience of pulling this Community Technology Preview together to provide an in-the-trenches view of things. I’m still too excited about the release to know what I would write but I’m sure there will be something in the next few days.

In the meantime, enjoy!

Update: Sanjay Nagamangalam has made a screencast showing some of the Visual Studio tools in action.

Tags: ADO.NET, work, Microsoft, database, programming, CTP

Kathy Sierra says it’s important to blow your mind on a regular basis. Well, she just blew mine with a post about neurogenesis and the discoveries that learning heals the brain.

You see, this month, as I think back on the DRI work, I can tell that my mind has healed. Really. I can almost feel it. I couldn’t put it into words until I read her post, but that is what I gained this month: a healthy mind.

In January, I thought I had just gotten tired of software, or Microsoft, or something. But I couldn’t articulate anything else I would rather do. I know that the first step in achieving something is to know what you want, but I had no goals, no dreams, and no visions of what I might want to do. My mind was dull and lifeless. I had never experienced this emptiness before.

What had happened to me?

I worked hard this month and I learned while I did it. That combination of duty and learning sparked some kind of renewal inside me. The flurry of ideas and projects and possibilities that have always been my constant companions are back in full force and I’m acting on them.

For something that I dreaded so much, the DRI work has been an absolute blessing to me. Hawai`i was transformative in one way, but I wasn’t ready to learn what it had to teach me: the lesson was incomplete. But I’m ready now. As I re-evaluate that experience, I’ll be writing about it here.

Tags: learning, healing, mind, work, duty
del.icio.us tags: learning, healing, mind, work, duty

Kathy Sierra and 37 Signals aren’t the only ones who understand the value of enjoying your software — and that enjoyment may be more important than what the software does.

Though some people wouldn’t believe it, there are people at Microsoft who understand that, too.

Like Chris Pratley and the OneNote team.

At first glance, spending only 1% of the development budget on adding “soul” to the product seems small. But the people who support the philosophy to implement those 1% features are also working on the other 99% of the product, and that philosophy shows.

OneNote is a fantastic tool. Sometimes I use it just because it feels good to work in. Really. I’m amazed at the some of the ideas I develop after noodling around with them inside OneNote for a few minutes.

Tags: OneNote, software, philosophy, passion
del.icio.us tags: OneNote, software, philosophy, passion

And thankfully nothing remains quite the same.

This month, I have been the DRI — the Designated Responsible Individual — at work. Fancy title. So what? I’m responsible for keeping a team-specific branch in our source control system in sync with the main branch. Don’t worry if you don’t understand what that means. I’ll explain it a bit.

Every piece of software that you use, from your Web browser to your favorite game to your operating system, starts out as source code: a set of instructions that tells the computer what to do. We’re all familiar with the idea that computers only understand binary code – a bunch of 1s and 0s. But it’s terribly hard for humans to write a meaningful program in binary. So we have programming languages: C++, Java, C#, PHP, Smalltalk, Python, Ruby, Scheme, and many more. These let programmers write things like printf(”hello world”); rather than a long string of ones and zeroes. In order for the computer to understand printf(”hello world”); the source code needs to be translated for the computer. That translation from source code to something that the computer can understand is called compiling.

Developers need to keep track of the changes they make to source code. Sometimes they make a mistake and need to go back to an earlier version. So they use a source control system that remembers all the changes and lets them get to any version they want. Source control systems are important when there is more than one developer writing a program. The source control system lets the developers share code with each other in a reliable and automatic way. In a typical design, there is a central server that stores the master copy of the source code, and then each developer makes a copy of the source code to his computer. This is checking out the source. Once a developer has made the changes she needs to make, she sends the changes back to the server — she checks in her source code.

Even simple programs may have dozens of source files and big programs like Microsoft Office or Windows have hundreds of thousands, maybe millions, of files, each with hundreds of lines of code. Add hundreds of people making changes to this source tree and you need a lot of machinery to keep things from devolving into chaos.

One of these mechanisms is called branching. It’s a way to segment the changes so that only a few dozen developers, say, are making changes that directly affect each other. Start with the source tree and make a copy of it on the server. Now there are two copies of the code. One team works in one branch, and another team works in the other branch, each making changes. At certain times, the teams need to synchronize their changes so that they’re working from a common base again. This synchronization is called integration. When the changes move from the primary, or main, branch, to the secondary branch, it’s called a forward integration. When changes move in the other direction, it’s called a reverse integration.

There are a whole set of ways to verify that the changes in one branch are ready to be moved into the other branch, and to verify that the integration is successful. The first test, though, is whether the code will build. You can think of building as compiling the source code, although it’s more complicated than that. There are many pieces of source code, and those pieces depend on other pieces in the system. So the build system has to compile the source code in the right order. In some cases, developers haven’t written the actual source code, but have instead written source code that will generate the actual source that gets compiled. The build system needs to manage this. There are steps needed to link all the compiled files into an executable that the operating system knows how to run. And a whole lot more. If the build system can successfully do its job, one step is out of the way.

But simply building doesn’t mean that the program actually does the right thing. You can compile a program with instructions that tell the computer to delete every file on the hard drive. Unless you’re a malware author, that is not a feature you want in your program. So there’s a second step: verify the build. This is done with a set of tests called build verification tests (BVTs) that make sure all the features of the software program work. There’s a lot more involved in testing, from automated functional testing, to stress testing, to performance testing, to ad-hoc testing, but that’s for another time.

Now, finally, I can explain what it is I do as the DRI for the month of February.

Every night after work, I spend a few hours integrating changes between the main branch and our team’s branch. I merge the changes from one branch into the other, build the source tree, and run BVTs. If all is successful, I check in, then turn around and pull changes in the other direction — a nightly reverse integration and forward integration.

I’m not constantly working during those hours since much of the time I just start a program and waiting for it to finish, but it requires enough of my attention that I can’t go out to a movie, for example (unless I want to stay up very late).

That’s on a night when things go well. If, after the integration, the build doesn’t complete or the BVTs don’t pass, I can’t check in the changes. Instead, I have to track down the owner of the tests and/or the code that’s at fault (I do a bit of investigation to narrow down the problem), give them a pointer to the build (that’s the result of compiling the code) and the enlistment (that’s the copy of the merged source used in the build) so that they can investigate the problem and figure out a fix.

Sometimes, there are big changes that need extra work — a more involved build than normal, for example — and thus extra attention to the process. Sometimes, due to the timing, a change happens in one branch and another change happens in the other branch, and, though each branch builds fine on its own, when they’re combined, the result won’t build. It’s why they have a person doing this work rather than automating the whole thing. We have a lot of developers working on a lot of code.

So — finally — what’s the attitude and how did it change?

Before February, I couldn’t help but resent the responsibility. Not only did I have to do my normal daily work, but then I had to do this DRI stuff, too. Never mind that every developer in our team does it at least once. I tried to convince myself that I would learn from this, but the resentment ate away at that approach.

On my first night, I was terrified. There are hundreds of people who depend on this source code — if I make a mistake, they’re all blocked. I had never done this before. What if the directions are incomplete? What if I discover something that isn’t documented? What if I’m just not able to do it?

After a few nights of success, I wasn’t as worried any more. I even relaxed a bit and started to write down the things I was learning as part of the process. The variety surprised me: from working with test automation tools, to the configuration and behavior of the autobuilders, to a way to write subroutines in a .cmd file. And, of course, I developed a confidence in our complicated build system and — for the first time — a sense of the overall shape of our source tree.

We’ve had problems. When I started, our development manager told me I had drawn a tough month. I thought it wouldn’t be so bad since I had gotten the shortest month of the year. But he was right. There were breaks after merging the branches, and there were some big changes that required a lot of effort, patience, and painstaking attention to detail. I spent time on weekends and got behind on my “real” work while trying to solve the DRI problems.

But it’s those problems that have washed away the resentment and let me grow. If the month had been normal, I wouldn’t have learned half what I did. I am more confident and enormously more useful to the team after doing this.

And I’m so much happier.

I will be relieved on the first day of March, but I’ll be a little sad, too. Even though I feel a heavy pressure when things aren’t working, and I worry every night until the tests finish, the joy of completing a complicated integration and the enormous satisfaction of doing it well make it worth it.

Dawn tells me that I thrive on challenges. And yet in the past couple of years, I have tried to create processes to smooth out the bumps in my environment. I’ve tried to create predictability and a regular routine. Somehow I got onto the mistaken idea that I would be happier if work were predictable and I could avoid surprises. But I don’t learn in those situations. I don’t build an emotional connection to the team, to the work, if I already know what to do before I need to do it. With unexpected problems, the need to figure things out that I’ve never seen before, and pushing hard to meet a deadline, I build a commitment to the people and the product, I engage my mind, and I learn.

And I’m well and truly happy when I am learning.

Tags: challenges, growth, software, development
del.icio.us tags: challenges, growth, software, development