According to his agency, the original cast member is leaving because he's been unable to keep up with the physical deman