i agree, go!
dh joined us on our last trip, for the first time. he is not a disney fan, but wanted to see it with the kids, so he flew down with us and came home after 5 days. the kids and i stayed 8 more days!
the whole trip was great! but, it was great in different ways. we all have our little quirks, many that make vacationing different; ie. dh likes to get up early, but it takes him a long while to get ready to go. while he was with us, he got up early, and the kids and i slept in or watched tv, while he tended to his morning routine. he isn't used to traveling (read-walking!) and wanted to go back to the room early, so we swam some evenings. he likes to try different foods, so we did that while he was with us.
after he left, we did exactly as we pleased! we slept in, went to the parks til close, then to dtd til it closed, getting back to the room sometimes at midnight. we ate at every hamburger joint at wdw and rode the tomorrowland speedway over and over, at ds's insistence. we waited in lines that were 20 minutes, which he wouldn't have handled well. we went swimming in the morning, the afternoon and the evening.
dh enjoyed the trip while he was there, as did we. and we were glad to have him there. but, we also enjoyed our time without him, and going at a different pace. I truly enjoyed having the kids to myself and encourage you to take off and enjoy!