"The Late Late Show" host was briefly banned from the famed New York City restaurant Balthazar over his alleged "abusive