At the 2004 Olympic Games in Athens, Patterson competed as an all-around athlete and delivered outstanding performances in the preliminary competition, finishing first overall, qualifying for the all-around and balance beam finals. In the team finals, Carly did not perform as well she did in the preliminaries. She under-rotated her vault, stubbed the low bar with her foot, and her beam routine was marred by wobbles and a lunge forward on the dismount: only on floor did Carly perform as well as she was capable of. She later admitted that she was more nervous than she had previously been in the competition. The U.S. team made other mistakes, such as Courtney Kupets costly missed turn on floor, and came away with the silver medal after having been favorites for gold.

In the Individual All-Around, Carly was back to her best and the competition proved to be the much-anticipated battle between Carly and legend Svetlana Khorkina. After scoring lower than usual on the vault (9.375), C