Some at the Bank of England suggest less competition from EU firms is part of the picture.