Working in a male dominated industry, I know just what kind of behavior you're talking about and its worse when the man thinks he's God's gift to the world. Most of them are harmless but you can't always be sure who is harmless and who is going to turn in to a stalker dude on you.
Bottom line is that this fellow makes you uncomfortable which is cause for concern. I don't know what your relationship is with your supervisor, but when I get those feelings about people I have to deal with, I mention to my boss that "I may be overreacting, but Jim is spending a lot of time in my office socializing and it is making me a bit uneasy. If he were there about work, it probably wouldn't bother me as much, but he is increasingly visiting just to chat which is becoming a distraction. What do you think I should do?"
Other than that, I can only suggest that you continue to be professional and polite to this gentlman in your dealings with him. You're not encouraging him now, so you're doing the right thing. By mentioning it to your boss, if his behavior does start to cross the line, you won't be criticized for waiting to long and he/she will at least be aware of a possible problem.
Sorry I can't be more help.
Michelle
