If only I could count the number of times that I've been called a fitness fanatic or an exercise nazi (or some variation on that theme). It doesn't bother me though. I've always had desire to do what I wanted and disregarded doing things that would make people approve of me. Besides, few will complain or criticize the results of my training even if they ridicule the way that I got it.
I find that comical. People with inferior builds telling those who are physically conditioned that the way they got that body it is too strict, rigorous, or fanatical make me laugh. If what they did yielded good results, then why are they never getting to where they want to be physically?
Some philosopher once described virtue as being the middle ground between too much and too little. For example, courage would be the middle ground between being a coward and being stupid. Now, the middle ground will change from time to time as the social definitions of excess vary.
That is the problem right now with society as we know it. The virtue of how much exercise is enough to ensure a healthy, strong and good looking physique is lost in modern society's interpretation of excess and deficiency. Any sort of daily, regular physical training defines you as being extreme. Daily physical training and a clean diet is now considered excessive. We don't often take the time to realize how much modern society skews the lines. Still, it works and its critical for succeeding in getting a fit, strong body.
This is where having a thick skin comes into play. You have to be able to ignore this faulty thinking that society accepts as truth. Manipulating your body, be it losing fat or gaining muscle, requires diligence, focus, determination that the flab-ulous in society can't or won't comprehend. Don't let them talk you out of it. If what they did worked so damned well, then why are they in worse shape than you? Keep that in mind before you allow yourself to be disuaded from healthy living.