We do a lot of A/B testing at SlideShare. Such tests (or at least the ones we run with Google website optimizer) are mostly tactical. They are about getting your call to action, the size of your button, or the copy of your landing page right.
While it’s important to do these experiments, they are mostly useful for refining ideas. As Andrew Chen would put it, you can get caught in local maxima if you focus only on them. The substantive decisions : to try a different product strategy, to build ne...