Government targets are at risk if infrastructure spending is cut too far, a top adviser warns.