A man who might seem rather snooty gradually reveals himself as cultured and restrained, rather like Moore's art