It's true that when politicians and legislators have almost unlimited power to tax and spend and legislate everything that isn't nailed down, then allowing any special interest group access to the levers of political power that allow them to tax and spend and legislate with such gay abandon is inherently dangerous.
But isn't that an even greater argument that the levers of power should pull so much less weight? That there should be constitutional restrictions not on how much we can spend on our favoured party so it can gain power, but instead on how much parties can do once they have power. Isn't that infinitely more important, and far more supportive of genuine free speech?
As PJ O'Rourke says,
When buying and selling are controlled by legislation, the first things to be bought and sold are legislators.Isn't that the point in a nutshell? Restrict the range of areas in which legislators can meddle, and you immediately lessen the interest in buying political power.