First, let me say that Disney has let party guests in early at 4 pm. This helps to justify the cost and you don't need to be there until midnight.
We really like to start our Disney vacation with the party on our arrival night. You will get the most park hours in going to the park that arrival day.
If you get your toddler to take a long nap you have a good chance this will be fine with your child, too.
To me, it's always worth seeing a special event at least once; it's an experience to share with one another.
You get a wonderful Christmas feeling doing the party; it's not just about the rides. You get cocoa and a snack included with admission, too.
The one drawback is it can get cold in the evening. We come prepared with layering our outfit to be safer. Bring a blanket for the stroller. You can go inside at times; so, you can warm up that way.