The International Paralympic Committee says Paralympic growth could be "jeopardised" by any merger with the Olympics.