Damn, he's cruel. Does he usually throw this relationship in your face or was it just this one time because you asked?
An emotional affair requires secrecy, intimacy and sexual attraction.
One thing you haven't mentioned is if you asked him why he hasn't been physical with her yet.
It does sound like he is no longer committed to you. If he is committed, he will make drastic changes in his behavior without complaint.
These are examples of risky behavior between oppo sex friends and behavior that should be prevented if the wayward wants to preserve the M/R.
This paraphrased list is from Dave Carder in his book, Close Calls. What ADULTERERS Want You to Know About Protecting Your Marriage.
1. Do you save topics of conversation for your friend?
2.Have you shared spousal difficulties with your friend? (Complaining, playing therapist, my husband/wife never...)
3.Has your friend shared difficulties they have with relationships? (Complaining, playing therapist, my SO always...)
4.Do you look forward to seeing your friend more than you look forward to seeing your spouse?
5.Do you compare your spouse to your friend (why can't my husband/wife be more like...)?
6.Do you provide special and thoughtful treats, food, gifts and cards to your friend?
7.Are you more concerned for your friend's welfare than your spouse's?
8.Do you fantasize about being married to this friend?
9.Do you spend more alone time with your friend than you spend with your spouse?
10.Are you comfortable letting your spouse review all your phone interactions, all texts, emails and voice mails?
11.Do you spend money for things like lunches, gifts and coffee on your friend?
12. Are you and your spouse in conflict over this relationship?
13.Do you lie or manipulate the truth in order to spend more time with your friend?
14.Do you hide receipts, phone bills, mail, gifts and time spent with your friend from your spouse?
15.Do you get angry or accuse your spouse of jealousy when talking about your friend?
16.Have you and your friend developed special rituals and/or places or any experience that you and your friend anticipate?
17.Do you experience a shiver when your friend admits feelings for you, or touches you accidently or purposely?
18.Does your conversation with your friend have sexual or erotic content?
19.Do you or your friend ever use work to date? (Including travel, entertainment, drinks, expensive meals, staying at same hotel.)
If your H has any desire to preserve the M he must cut out all this behavior.
If he doesn't want to call it an EA, fine. He's still disloyal, cruel and immoral. You mentioned he initiates contact with her, if so, he sounds like a real creep, possibly a predator.
Do you know how much she reciprocates and if she is M or in a R?
I'm not sure I have the whole picture yet, but he's white knuckling it with her and callously abusing and mistreating you.
You're going to have to take a very strong stand here. It does seem that betrayed spouses who set strong boundaries and take decisive action are the ones who see their WS most often come to heel.
Also, maybe don't show him the list. It's almost like a guide to having an affair.