My [33F] ex-husband [43] constantly scolds me for going out and I am at my wits end.

Edit: I?ve noticed I?ve gotten some comments in regards as to why I mentioned my going out. My ex and I do not live in the same city?maybe 30 minutes apart. I never fail to get caught in traffic and considering I was heading into town anyways, I offered to drop my son off at his house. He told me he wouldn?t be able to take him to work for the day (he runs a business that he?s usually able to take our son with him). That?s when I told him that I would take him for the day but that he would need to pick him up later on due to my plans.
(This will most likely be a bit long, but I?ll try my best to make it short.)
My ex husband and I share a son [12] together. I also have a daughter [16] from a previous relationship. While our custody arrangements are a bit different, I always have my son and daughter on the same weekends.
When my ex husband and I were together, he was extremely controlling and demanding, which ultimately led to our divorce. I could never go out, even if I was simply going out to eat dinner with a friend. We now share 50/50 custody of our son, alternating the weeks and weekends.
I?m with my children very often. 90% of my time is consumed by them and my work. I rarely go out, even on the weekends that I don?t have my children, but my birthday is coming up so my mother decided we would go out of town (maybe 30-40 minutes from where I live). We made sure to schedule our trip on my ex?s weekend considering we were going to be renting a hotel for the night.
My ex has never been fond of my going out and on the rare occasion that I do, he never fails to blow up my phone to let me know what a horrible mother I am. Considering this is my sons fathers weekend, I would usually take my son to school and his father would pick him up, but considering my son had the day off, my ex told me to keep him for the day and that he would pick him up later in the evening (I also had the day off).
As the evening came around, he refused to pick him up in an attempt to ruin my plans. I told him that I would be going regardless. I had planned on leaving around noon because I was trying to beat the rush hour traffic, and I had also needed to pick my mother up before checking into our hotel. I told him that no one would be able to watch our son considering my daughter was also going to her fathers for the weekend. I advised him that if he didn?t pick him up, he would be left home alone, to which he threatened to call the cops on me and have me arrested for ?child neglect?.
I know this may be one of those ?mind over matter? type of things but his comments (and actions) have genuinely become so hurtful and are borderline psychotic. I shouldn?t feel bad for going out on MY weekends. I shouldn?t be threatened to have the cops called on me for going out on MY weekends. He is an extremely controlling and manipulative person and I am at my wits end with him. I?ve yet to go out without being bombarded with hurtful and threatening text messages. I couldn?t even count the number of times he?s just showed up and dropped our son off without my knowing. He does this solely for the purpose of preventing me from going out.
He ended up picking him up but I don?t doubt this will happen again. How should I go about this? I?ve tried talking to him, explaining that I have the right to go out on my weekends but he refuses to listen.
TL;DR: I am at my wits ends with my controlling ex who constantly harassed me for going out on MY weekends.


Source.