Finding yourself attracted to another person means you're a breathing human. Self control, valuing your principles, not wanting to bring harm to your H/family/friends/other people's families/friends, not wanting to suffer the heavy cost of infidelity, etc. keeps you from acting on attraction.
After many years of my W's frigidity, I was terrified of even entertaining the idea of allowing a female to touch me. If a female flirted with me I intentionally made it awkward for her (not rudely, just a proper measure of intentional aloofness) to discourage her doing so again and I never gave another woman a hint of interest from me. All because I was keenly aware of a weakness I had due to intimacy starvation. While I really find making love extremely pleasurable, it was the slippery slope of person-to-person emotional intimacy that would trip me up and no doubt lead to a PA as well. I knew once I began down that path it would be like a drug for me and I'd be quickly living a double life.
Quite frankly it's a good thing that you are having an internal dialog of wanting to be cautious. I was and am too. I am as capable of marital transgression as my WW was or any WS. It's my hope that being aware of and facing my weaknesses, bluntly and without a filter, limits my acting on feelings that will cause personal and familial carnage. My weaknesses don't make me a bad person. Your weaknesses do not make you a bad person either.
Maybe it'd be a more balanced view for you to acknowledge that self-awareness but to not judge yourself as deserving of guilt because it's present. Use that knowledge as a weapon against your weaknesses.
One more thing to possibly try for you. Have you considered sharing all of what you shared with us with your H? No doubt, true intimacy is not just sexy time, smiles, hugs and kind words. As difficult as it may be to expose the real you by expressing how you feel and the risk you perceive with the man at work with your H, it'd open the opportunity for a deeper intimacy for each of you.
[This message edited by NotMyFirstRodeo at 4:57 PM, Thursday, October 28th]