Improve the bracketed part of the sentence.
Grandfather has said so out of affection, do not take (to) heart.