But one moment changed his life, and put him on the path to helping other young people from deprived backgrounds.
He has gone on to become a role model for youngsters in his community and developed a truly innovative way of teaching children maths that saw him rapping in the House of Commons.
Clifford, who lives in Birmingham, had always found school tough.
Suffering from dyslexia, he struggled to keep up with his classmates and failed most of his exams.
He remembers: "I didn't know what I wanted to do. I grew up in a very strict household, and when I left school with no qualifications or job prospects I rebelled. I was a bit like a caged animal breaking free and didn't care that I didn't have a job."
As he approached the end of his teens, Clifford's grandparents and mother sadly died in quick succession.
"That's when things became really bad," he says. "I wasn't interested in bettering myself and remained out of work for nine years. I was causing trouble, getting involved in petty crime, drinking, loitering and smoking."
But one day his benefits failed to come through - and that is when his life changed forever. He says: "It really upset me. But then I thought, what am I doing waiting for this money when I could get a job instead?"
Clifford managed to secure a cleaning job, and - inspired by the big offices he was working in - started to think about his future.
He heard about The Prince's Trust and made the decision to approach the youth charity for help setting up his own business, which started out as a children's party company.
But he decided he could do more than better himself. He became determined to make a real difference in his community too.
Now he volunteers as a youth worker with vulnerable young people. He's also worked with children with disabilities, helped out at after-school clubs and coached local football teams.
"It is so important to give back," he says.
And the most remarkable way Clifford gives back is by helping children learn maths - having developed a way of teaching them their times tables using songs and raps.
His hugely interactive Tunes Times Tables method has proved popular with schools, and Clifford, 36, now employs two people to help him spread the word. He was recently invited to the Houses of Parliament to demonstrate his innovative teaching methods to a group of MPs.
To reach children from all walks of life, Clifford now organises regular community events to give children from deprived areas the opportunity to have fun and forget about their problems.
He says: "My work in schools means I have met children from extremely poor backgrounds. I am so driven to help out where I can. And I want to change the negative view of rapping.
"I didn't believe in education as a child - but now I know it's the way forward."
Clifford turned his life around through his own drive and initiative. His teaching method is infectious, and he's a great male role model in an area where lots of children are growing up without a father figure.The Judges