On August 9, 1973, Perkins married photographer Berry Berenson. They had two sons: actor Oz Perkins (b. February 2, 1974), and musician Elvis Perkins (b. February 9, 1976).
He once said he felt too nervous around women, and resisted actresses Jane Fonda and Brigitte Bardot, who had tried to seduce him during his youth. He was a very shy actor, especially in women’s company. He reportedly had affairs with actors Rock Hudson and Tab Hunter, dancer Rudolf Nureyev, composer/lyricist Stephen Sondheim and dancer-choreographer Grover Daleprior to marrying Berenson. He had his first intimate heterosexual experience at the age of 39 while working on the 1971 film The Life and Times of Judge Roy Bean with an actress who also appeared in the film. According to an interview with People magazine Perkins declined to identify the actress, but “other sources” have identified her as Victoria Principal.