It can be so confusing, can't it? I remember having these exact same types of conversations with my wife and even in MC sometimes, and me having the same thoughts and feelings as you are. The idea of, "Why ask me how I feel about something if you're just going to get pissed and tell my answer was all about me? You asked me about me, didn't you? WTF?" And yes, from a purely grammatical point of view, that's entirely true. You were asked a question and you gave an honest answer, which is what you should have done. So what went wrong?
Now, this is just my own two cents, so take that for what it is, just an opinion from someone who isn't there and involved in your life. I think that what our betrayed spouses are looking for in those moments is simply to see if "they are in there somewhere". They are testing the waters to see if the thoughts and feelings of their wayward spouses include them in any way, shape or form. They also want to gauge how their WS's are thinking and feeling about themselves. It IS a loaded question to some degree, that's true, but it's also an honest one.
To better explain this, let me give an example. One day, while going about your normal business, you get mugged. The mugger beats you badly and takes your valuables, but just as he's leaving the cops show up. They throw you both in the same ambulance, as you need severe medical care from the beating he gave you, and he broke his hand and cut himself while beating you. In the ambulance, you ask the mugger how he feels after the incident. He says to you, "Thanks for asking. I feel crappy. Lost my job recently, I've been feeling sick, my wife is being a real pain and keeps riding me about the bills, the car broke down and now I have to deal with a broken hand and raft of legal charges for mugging you. I feel pretty shitty, that's how I feel." And that's an honest response from him, right? But let me ask you, when you asked how he was feeling, were you really looking to know about his personal life problems and his current sense of aggravation? Or did you want to know how he feels about who he is, what he allowed himself to do and how it affected you?
Consider this response instead. "I feel awful. I mean, things for me have been really rough lately, my job, my wife, my health, everything seems to be crashing down around me. But I can't believe I let it get to this. I can't believe I actually decided to mug someone, to hurt someone else, just because I'm hurting. It didn't solve anything for me and only hurt you. I can't believe I put you in the hospital. You had nothing to do with my troubles. This is not how I pictured who I am and it's not who I want to be. I didn't know what else to do, and in my panic and stress, I made the really bad decision to mug someone, because I thought that would make me feel better. I'm sorry I did that to you. It didn't help. It made everything else even worse for me, and now I've dumped that part of my life on to you as well."
Do you see the difference here? In both cases, the mugger answered honestly. But one answer was all about him, his hurt, his needs, his pain, but YOU weren't in there at all. And that feels kinda shitty even when it's just a mugger, not a spouse. The other answer showed that he has thoughts beyond what's "happening to him" currently. He also shows that he has regrets and remorse and disappointment in himself for what he did and how it affected you.
Infidelity is a nasty business. When we cheat, our spouses get the VERY clear message of, "You do not matter to me, at all, period. Nothing and no one matters to me, except for me. I am incapable of considering your feelings or anyone else's, and don't care that I hurt you. I matter. You don't."
That's a very tough pill to swallow, and once your spouse has accepted it as a painful reality, well, "you can't unring that bell". That is who they see you as, perpetually, daily. And every single time we fail to show remorse, empathy and regret, it just reinforces that point of view. If your spouse is to ever see you in a different light, you are going to have to give them some reason to do so. Lots of reasons actually.
Could your husband have worded things a little better? Could he have shared his innermost thoughts and fears too, and made it clear that he was hurting before asking you about it? Sure, he could of. But why should he? I mean, going back to the mugger scenario, was it your responsibility to be vulnerable and open with your mugger and to gently nudge him to a more empathetic and remorseful point of view? Or did you simply want to gauge where he was at on his own? It isn't the victim's job to be vulnerable with the person who most betrayed that vulnerability in the first place. He has his walls up and with good reason. Him asking you how you feeling was him pulling the walls down just enough to see what's going on. His question was meant to gauge how safe it is for him to put the walls down more. He didn't get what he was looking for, and so the walls went right back up.
I know you said this yourself, but it's important to remember. He is suffering too. He is depressed too. He is dealing with Covid and politics and work and health and everything else in life too, just like you are. He's tired and overwhelmed and has lost all sense of joy too. It's not that you are doing anything wrong by being depressed and overwhelmed, but the fact is, it leaves him in the position of needing to care about YOUR depression on top of his, and you CAUSED his depression in the first place. He needs some support. He needs some empathy. He needs to feel like someone is willing to pick him up and carry him for a while, instead of the other way around. And because infidelity is fucked up, the person he needs those things from the most, is the one who hurt him. And that too, throws a wrench into the works for him. He wants to be there for you, because he loves you and because he loves himself. But he also likely hates himself for loving you or wanting anything from you. He can't win. So how do you respond when you can't win, and then the person who hurt you can't be there for you either? You get upset. And you want to leave.
My only advice for you is this. When anyone else in the world asks how you are feeling, feel free to make it all about you. But when your betrayed spouse asks, answer the question "within the context of the relationship", and also within the context how it affects how you see yourself. Don't just state how you feel, but also the outcomes and ramifications of how you feel. If asked how Covid is making you feel, you might say, "I feel awful, my nose is stuffed up, my throat is sore, and I'm tired." or you can say, "I feel really shitty, but what's pissing me off the most is that I can't even get my work done, can't help out around the house, can't spend time with the family..." The second answer gives some insight into who you are and what you are thinking, and that's what our spouses are usually looking for.
I hope any part of this helps.