In one of his most candid interviews, the Bright Eyes singer tells Britain's Sunday Telegraph he initially quit Simon & Garfunkel and turned his back on showbusiness to become a teacher because he could no longer work with his sidekick Simon.
He says, "I don’t want to say any anti Paul Simon things, but it seems very perverse to not enjoy the glory and walk away from it instead. Crazy... (But) he was getting on my nerves. The jokes had run dry.
"A rest of a year was all I needed. I said, 'I’m not married yet. I want to jump on a BMW motorbike and tour round Europe chasing ladies'."
But it's his candid remarks about the start of his friendship with Simon, which led the duo to a recording deal as teenage act Tom & Jerry, that will surprise fans the most.
He admits he felt sorry for the singer/songwriter because he was so short, and adds, "That compensation gesture has created a monster.
"George (Harrison) came up to me at a party once and said, 'My Paul is to me what your Paul is to you'. He meant that, psychologically, they had the same effect on us - the Pauls sidelined us. I think George felt suppressed by Paul (MCCartney) and I think that’s what he saw with me and my Paul.
"Here’s the truth: MCCartney was a helluva music man who gave the band its energy, but he also ran away with a lot of the glory."
During his interview with the Telegraph, Garfunkel also calls Simon a "jerk" and agrees with the interviewer that the You Can Call Me Al singer has a Napoleon complex, linked to short people hooked on power.
Despite his comments, Garfunkel, who last toured with Simon in 2010, is open to another Simon & Garfunkel tour: "Will I do another tour with Paul? Well, that’s quite do-able. As far as this half is concerned, why not?"