The British musician - who has recently been playing the group's 'Unknown Pleasures' album live in its entirety - is remaking the tracks with former Happy Mondays backing singer Rowetta Idah, who sings with him when they perform the album live.
Peter told website Slicinuipeyeballs.com: "She wants us to record an EP of the four Joy Division songs that she sings, which are 'Insight', 'New Dawn Fades', 'Atmosphere' and a track called 'Pictures' that I finished up, which is an old Joy Division track. So we're going to do an EP of that, just because the versions of the songs with Rowetta singing have a really nice quality."
The track 'Pictures' is taken from a demo which singer Ian Curtis - who hanged himself, thus ending the group in 1980 - recorded but never completed.
Peter - who formed the band New Order with the other members of Joy Division, Bernard Sumner and Steve Morris after Ian's death, and is now in new band The Light - also said he's surprised at how good playing the 1979 album still sounds.
He added: "The test of good music is when it sounds as good when it was made as it does now, and I think Joy Division is one of those groups that sounds just as good now as it did 30 years ago, and I think that's a great compliment to our songwriting skills - all of us, Bernard, Steve, Ian and me.
"When we play it, a lot of people come to me and say, 'Wow, you look like you're really enjoying it,' and I'm thinking, 'I am.'"
Peter is playing 'Unknown Pleasures' live in the US in December.