they say that because you are infected with some sort of disease
see this guide
go to temple or drink cure disase potion to get rid of the disease
then people wont tell you anymore that you look sick