My husband (31M) is pressuring me (27F) to buy a house I don't want

My husband and I have been together for 5 years and married for one month.
We have been living in a rental house for almost two years, and this is the house he wants to buy. I am the only income source. Our landlady wants to sell it and is giving us first dibs. However, it is a duplex and our side is too small for a family of four to be long-term. (We have two-year-old twin girls) At first, I was on-board to buy the house because his mom lived on the other side, but she moved out. She's still paying rent because she didn't want a stranger living with us. The plan is she would stop paying once we purchased. This would give us three bedrooms instead of two and two bathrooms instead of one. I told my husband that I would buy the house if and only if we could have the WHOLE house. Then he moved his brother in. I love his brother, but he pays for nothing and refuses to get a job, so I would essentially be buying him a house too. On top of that, the backyard is dangerous for our kids because it has a huge drop-off, the kitchen is the size of a hotel's, it has no driveway, if not smaller, and our neighbors are loud and vulgar. None of those are necessarily dealbreakers, but I was trying to give my husband what he wants. I already compromised by staying in a town I don't like. I can't use my degree here and it's far away from my own family and other towns. Do I really have to buy the house only he wants too?
TL/DR: I am the only income source, and my husband refuses to consider another house. I don't want to buy the one he wants.


Source.