Bob O’Hara has a nice post on how you need to be careful with statistics, even if someone seems to know what they are doing. Just a short summary:

If you want to study the effect of some intervention in an observational study (like the introduction of a double-blind review system in a scientific journal, but of course the argument applies to all kinds of observational intervention studies), don’t just look if there is a significant change between pre- and post-intervention time — maybe there has been some change for all the observational units (e.g. journals) even without the intervention.