Investors are fleeing the stock market but will they end up poorer?