Pantoliano was born in Hoboken, New Jersey, to first-generation Italian American parents Mary, a bookie and seamstress, and Dominic "Monk" Pantoliano, a hearse driver and factory foreman. He attended HB Studio, but studied extensively with John Lehne and lives with his wife, former model Nancy Sheppard, and their four children.

On October 9, 2007 Pantoliano announced on the National Alliance on Mental Illness blog that he has been suffering from clinical depression for the last decade, although he was only formally diagnosed recently. He claims that his recent film Canvas was what helped him come to terms with his depression. Rather than hide his struggle from the public, he has chosen to speak out about it to remove some of the stigmas that are commonly associated with mental illness. He founded a non-profit organization, No Kidding, Me Too!, to unite members of the entertainment industry in educating the public about mental illness. He also suffers from dyslexia.