Positive results and "amazing" results have higher odds to get published.
Which is "publication bias" - the average published result is stronger than the true effect.
A naive method to "fix" this is to include in a meta analysis all unpublished data as well. Which is an error.
There are reasons why some works are not published.
Unpublished papers are on average:
1) weaker in terms of any parameter of quality. Sample size, procedural rigor etc.
2) Less prominent authors.
3) Authors less tenacious in getting it published. Sometimes you have to fight to get published. Try multiple journals, run additional tests, re-write stuff etc. etc.
4) Authors whose topic of the work is more peripheral to their career / expertise are more likely to "let it go", or even not know where is the best venue to get it published etc.
All those criteria are clearly correlated with lower quality of the ultimate study being done.
Weaker result, less experienced researchers, lower tenacity (which has an effect on study design, perfectionism in carrying it out etc.), or authors which center of focus is elsewhere.
All these are logically related to less meaningful studies.
Full inclusion of lower quality unpublished studies is dumb. Ignoring it leaves us with publication bias. So?
I think a n intuitive solution is to include those works with a lower weighting to take account for them being on average of lower quality. (my feeling is around 40%, but take your guess)
PS. One might do a Bayesian calculus that takes care on the fact that only the unpublished studies failed. In which case, i would expect intuitively that their ultimate weight will be even lower. But I have not done the math.
Which is "publication bias" - the average published result is stronger than the true effect.
A naive method to "fix" this is to include in a meta analysis all unpublished data as well. Which is an error.
There are reasons why some works are not published.
Unpublished papers are on average:
1) weaker in terms of any parameter of quality. Sample size, procedural rigor etc.
2) Less prominent authors.
3) Authors less tenacious in getting it published. Sometimes you have to fight to get published. Try multiple journals, run additional tests, re-write stuff etc. etc.
4) Authors whose topic of the work is more peripheral to their career / expertise are more likely to "let it go", or even not know where is the best venue to get it published etc.
All those criteria are clearly correlated with lower quality of the ultimate study being done.
Weaker result, less experienced researchers, lower tenacity (which has an effect on study design, perfectionism in carrying it out etc.), or authors which center of focus is elsewhere.
All these are logically related to less meaningful studies.
Full inclusion of lower quality unpublished studies is dumb. Ignoring it leaves us with publication bias. So?
I think a n intuitive solution is to include those works with a lower weighting to take account for them being on average of lower quality. (my feeling is around 40%, but take your guess)
PS. One might do a Bayesian calculus that takes care on the fact that only the unpublished studies failed. In which case, i would expect intuitively that their ultimate weight will be even lower. But I have not done the math.
Posts
No comments:
Post a Comment