Brody was born in Woodhaven, Queens, New York, the son of Sylvia Plachy, a photojournalist, and Elliot Brody, a retired history professor and painter. Brody's father is Jewish and Brody's mother was born in Budapest, Hungary to a Catholic father and Jewish mother. As a child, Brody performed magic shows at children's birthday parties as "The Amazing Adrien". Brody attended New York's Fiorello H. LaGuardia High School of Music & Art and Performing Arts (famous as the inspiration for television's Fame). His parents enrolled him in acting classes to distance him from the dangerous kids with whom he associated.

In 1992, Brody was seriously hurt in a motorcycle accident in which he flew over a car and crashed head-first into a crosswalk. He spent months recuperating. An avid daredevil, he has broken his nose three times doing off-the-wall stunts. His nose was also broken once again during the filming of Summer of Sam.