My SO (27M) wants to go on a 2 week vacation with his friends, while I (26F) stay at home with our child (2.5F). He doesn't see why I'm upset.

We have been together for 6 and a half years.
A few nights ago my SO, Paul, was hanging out in the living room with his friend. His friend started talking about the vacation he and a few other friends are taking next month. It obviously wasn't the first time they talked about it. Paul looked at me and said "oh yeah, i might go away for a couple weeks next month." I thought he was joking because that doesn't seem like a serious way to phrase telling your girlfriend and mother of your child that you're going to spend 2 weeks away.
I forgot about it last night until he said some friends were stopping by later and brought it up. He said he wasnt sure if he was going yet but he wanted to. He said he would cost him next to nothing cause its just a road trip to free places and parks to hike at. I said that it would make me upset if he went and he said that we can discuss it when he makes a decision. He said there was no sense in getting emotional over something that might not even happen. I said that I can't stop him, but he should know that if he decides to go it will hurt me and our relationship and tried to explain why.
It wouldnt cost next to nothing. Thats 2 weeks off of work, gas money, food, new hiking clothes, boots and supplies etc. The main stop is a trail that I've always wanted to hike at so it hurts to think of him doing something like that without me, knowing that I'll probably never have the chance, at least not until our child is grown up (assuming that my then my now mild joint issues don't get worse). He has stayed home with our daughter so i could go out once, for two hours. We have never had a big vacation just the two of us. We go 2 hours away from home for 3 days once a year. And with our child being so young, it's not much of a vacation for me anymore. We havent even had a real date in a year. My birthday is in a week and we aren't even doing anything for that. I love hiking and have been trying to get him to go on hikes with me for years and he has never gone with me.
Anyways, Paul just seemed angry and said that if it's apparently something I'd love to do myself(phrased as if I have never mentioned it before), it would be cruel to take it away from him because I'm jealous. He says since he is the one who carries our family by working he deserves this. His friends showed up early so we haven't talked about it since. Its not that i don't want him to have fun and he does deserve this. I'm just hurt that he didnt consider me. They've been talking about this for a month. I feel like an afterthought.
Should I just get over this? Should I be happy for him?
Tldr; SO wants to go away for two weeks to a place I've dreamed of without me and im really upset


Source.