There are over a million AP holders. On any given day, I would guess at least 50% of the crowd is AP holders. Taking away a huge chunk of AP holders brings down the overall crowd level. There simply aren't that many "day guests" or multi day vacationers at any given time.
I have a Signature Pass and go frequently. It is definitely less crowded when more AP blackouts are in effect. Saturdays when AP SoCal and Deluxe AP holders are blocked out are practically empty compared to days where there are NO blackouts.
To illustrate this point, between Christmas and New Years last year (first year of the more strict blackouts), the parks NEVER reached capacity. The DAY the blackouts were lifted and the following 3 days, the parks hit capacity every day. I believe this was Jan 2-5, 2017. It seems counterintuitive, but the numbers don't lie.