The ex-Celtic captain criticised Grant, who was assistant to Tony Mowbray, for his man management skills.
Brown revealed he would have left Glasgow for a move south if Grant had remained on the coaching staff.
While at Celtic, Brown received offers from Spurs and Newcastle but turned down the chance to move to the Premier League.
However, he revealed he had no time for the former Scotland coach, who would leave the club as Mowbray was dismissed as Neil Lennon took over as boss.
Speaking on Open Goal, he said: “Lenny came in and we started winning games and the momentum changed from when Tony was there.
“I think if Tony and Peter Grant were still there, I’d have been away straight away.
“I had no time for Peter Grant whatsoever.”
When asked to elaborate on his issues with Grant, Brown said: “Man management. Personal. Personality? Zero.
“Tony, I got on alright. But Peter Grant, no.”
