The actor related an incident where his family walked away from him when he was a child and hid to see how he would reac