My [25F] partner [23M] allows his friends [20sM] to think I’m a controlling girlfriend to save face.

I’ll keep this short.
We live together and have two children [5F and 7mo] (eldest mine, youngest ours).
He has a group of friends from school who, up until we moved in, saw each other very regularly, 2/3 times a week (for nights out, gym, sports etc). We moved in together half an hour from where his friends live. His one friend, D, still regularly asks him to go play tennis after work.
I am quite happy for him to go when he wants to, which he rarely does, I encourage him to spend time with his friends. He rarely asks to go. I just want to point here that we both ask if we can go out with friends, because it impacts the other as they would have to put the kids to bed, etc (it’s a courtesy thing, we would rarely say no).
My boyfriend often doesn’t want to go due to not being able to see the kids that evening, or being tired from work, or just that he wants to spend time with me. However I’ve noticed a few times when he says he can’t go he will phrase it ‘I need to stay home tonight with ——- and help with the kids’.
Recently on a night out his friend made a sarcastic, and pointed, comment to me about whether my boyfriend would be ‘allowed’ out to play tennis the next week and whether he would be ‘allowed’ on a night out after the liverpool game on Saturday. As though I forced him to stay in.
I see two issues with this; 1) My boyfriends phrasing implies it’s my responsibility by default to look after the kids and not his. Therefore his friends take from that, that I am forcing him to help me. As if he isn’t SUPPOSED to help me every day as the father. 2) I resent the implication that I am the one asking him not to go! Which is never the case. I feel like he does this to take away the responsibility from himself and means his friend cannot keep bugging him to go. If he blames me, his friend will have to leave him alone.
I feel like he should make it clear to his friend that he is a father first and a friend second, and therefore he shouldn’t give him grief or make comments to me! Also I resent his friends thinking I’m controlling and therefore I feel he should make it clear he doesn’t want to go of his own volition. My boyfriend says it doesn’t matter and he just says it to save his friend badgering him.
Am I right here? Should I just accept the controlling girlfriend roll?
TL;DR my boyfriends friends infer that I am a controlling girlfriend, from things my boyfriend says. I feel it’s unfair and don’t want this reputation.
Edit to clarify. Though no ones asked I should point out 5F is mine, not his. I was not having sex with him when he was 17


Source.