The I Want Candy singer recently opened his heart about his feelings for first love Duff shortly after she announced that she'd split from husband Mike Comrie.
Carter said, "I'm gonna do everything in my power to fix those (broken) relationships in my life, just like I want to fix the relationship I had with the love of my life. I'm not gonna give up on Hilary. Ever."
But the So Yesterday hitmaker didn't appear to be impressed by her ex's comments, telling BuzzFeed.com, "I don't know how I feel (about that). I mean, that was so long ago, and obviously I'm still married, and I have a baby and we kind of just... don't know each other."
Now, Carter tells Wenn he understands Duff's annoyance, saying, "If I'm too open about how I feel then people wanna nag and pick and poke at me. I don't know Hilary either. She don't know me and I don't know her. I just need to shut up now about it. I think it's time...
"She's got a kid and she's married and I'm not trying to be that dude. That's not my intention. Hilary will always hold a very special place in my heart."