Do you ever feel bad you couldn't be there even if you couldn't help it?

And maybe the asker was being unreasonable. I'm not sure if it's just me, I'm too sensitive, or what, but when somebody asks something of me I always want to help, and when I give the dreaded 'no, sorry' I feel anxious and remorseful for hours after. D;