Whenever I decide to drop some weight, and therefore begin turning down donuts, cakes, etc, people invariably inquire if I'm "dieting". Once I admit that I am indeed attempting to shed a few pounds, I always get the comment, "But you're not fat."
I always smile and thank the commenter. If they're actually interested, and not just making conversation, I usually then go on to explain that I probably weigh more than they think I do. People were guessing my weight at 230 or so when I weighed 285, so I know that folks tend to underestimate my size. I'm right at 6'2" and I'm built like a linebacker (or a gorilla if you ask my pal Brian) and this tends to let me carry more weight without looking overweight.
See, one of the things that makes people, and men in particular, look fat is the shoulder to hip ratio. The wider your hips look in comparison to your shoulders, the more you look like a pear. So the broader your shoulders are, the bigger your waist can be without you looking rotund. You still have something of a V shape.
The other thing is that I don't have a gut. I don't tend to gain weight in my stomach, but rather across my entire body. I refer to this as 'the nesting doll effect.' It's like there are extra layers of me. I lose weight in the same way, shedding an outer layer or two as the weight drops off. This also means that I have to lose a whole lot before people actually notice. I think I had lost about 30 pounds last year before anyone realized I was getting smaller.
The other comment I tend to get is something like, "Yeah, but you're a weightlifter and muscle weighs more than fat." Yes it does, and while I have a lot of muscle mass, it is surrounded by an ugly layer of fat, thanks.
Anyway, it's not that I don't appreciate the folks who think I look fine at my current weight. I really do. But I can feel that extra fat on the sides of my body when I bend or sit, and I don't like the way that feels. I don't like the fact that my pants are a little snug or that I've had to loosen my belt one notch. It's a lot easier to just fix the problem while it's not that noticeable to anyone but me.