The only thing I'm worried about the 2012 games is:
Will NYC build everything it has to in time?
The three most significant pieces of the construction puzzle are the Olympic Stadium, the #7 extension, and the Olympic Village in Queens. There are other construction projects, but these three stand out.
The Jets want the stadium to open their 2008 season, because their Meadowlands lease expires in 2007. The new stadium has to be under construction by sometime next year. If not, New York probably isn't getting the bid anyway.
The #7 extension is two stations. I think the MTA can build two stations in less than eight years. This is another piece that realistically needs to break ground before the IOC vote, simply to prove that the bid is serious.
The Olympic Village would be converted into apartments after the Games, and there is always a demand for apartments, especially with river views. I wouldn't worry about the Olympic Village getting built.