Unless someone sits up front next to the driver, you might need an XL for both size parties. Do you really want to cram three people into the back of what might end up being a small compact car, plus another up front? I got an Altima for an UberX recently, and that would not have even been possible because the driver had a car seat and a large handbag, stored in the front seat.
To answer the question about price difference, it depends on the distance and other factors, but the XL is in the range of $20ish more than a regular X. But that really does depend on other circumstances.