Weller wrote the anti-establishment tune in the early 1980s after watching activists on the Right to Work march in England as they walked past the toffs at Eton College in Berkshire.
In new DVD documentary The Jam: About the Young Idea, the singer/songwriter, who became a leading light in the anti-Conservative government organisation Red Wedge, recalls feeling disgust as the Eton students mocked the protesters, who were marching for employment rights.
Cameron, who was a student at the time, recently admitted he was a big fan of the song, prompting Weller to suggest he didn't read the lyrics.
The A Town Called Malice singer says, "David Cameron said that Eton Rifles was one of his favourite songs when he was at school... Not everyone reads the lyrics! Sometimes, it's just a nice sound in the background."
Weller isn't the only British rocker confused by some of the Conservative Party leader's favourite tunes - former The Smiths guitarist Johnny Marr provoked a storm of debate in 2010 when he "forbid" the Prime Minister from liking his music.
The statement was a reaction to Cameron's announcement he was a fan of the cult 1980s group, and in an interview with music website AskMeAskMeAskMe.com, he said, "It just came to my mind. I wasn't too bothered about David Cameron. I thought it was silly what he was saying about being a fan of The Smiths, so I just said it (on Twitter) and then I went to bed.
"I woke up about noon, phone ringing like crazy. My manager was asking me what he should do about all of these requests from TV news programs (sic). I looked at my iPhone and I had about 20,000 more followers (on Twitter) in five hours. I didn't know what was going on."