Jelly Roll knows his journey to the top wasn’t without its setbacks — and while he understands that not everyone from his past will be thrilled by his accomplishments, he’s working on making amends.

The country star, 39, opens up to PEOPLE in this week’s issue about reconciling with his early days, during which he was in and out of jail some 40 times over 10 years, starting when he was just 14.

“I’m rounding third on my amends list, and I think when I get there, I’ll feel a little better,” he says. “I was hitting some stone walls with people that wouldn’t forgive me and I was like, ‘God, maybe this has something to do with the fact [that] I haven’t forgiven myself either.”

Jelly says that while talking about his past in interviews, he’s often forced to “brush over 10, 12, 15 years of living an extremely crazy, narcissistic, selfish lifestyle” to condense it into a quick soundbite. But he knows the reality goes much deeper than that.

“The truth is, there was a lot of pain in there. I hurt a lot of people,” he says. “No matter how much I’ve changed, they still watch and are mad that I’m successful. I understand that. But I’m doing better at letting go of the past and realizing that I don’t owe that part of me anything.”

Jelly has previously been open about being “embarrassed” by his behavior during his younger years, saying on a recent podcast that he’d had an inflated sense of “entitlement” in his youth, and hoped that he could make amends with the victim of his robbery.