Kip Hansen’s guest essay – October 18, 2022
Every time someone in our community, the scientific skeptic or Realists®, speaks out about uncertainty and how it affected the peer-reviewed scientific results, they were immediately accused of being Science Deniers or try to sabotage the entire field of Science.
I have written again and again here about how the results of most studies in climate science underestimate the uncertainty of results. Let me make this as clear as possible: Any untruthful finding includes a frank discussion of the uncertainties associated with the study, starting with the uncertainty of the data. raw material and then all the uncertainties more By each data processing step, no digital ink is used to publish it.
A new, accepted, and upcoming major multi-study group study in Proceedings of the National Academy of Sciences, is set to shake up the world of research. This article, for once, was not written by John PA Ioannidisof the “Why are most published research findings false?” reputation.
The article is: “Observing multiple researchers using the same data and hypotheses reveals a potentially independent universe of uncertainty.”. [ or as .pdf here ].
This is a good science. This is how science should do it. And this is how science should be published.
First, who wrote this article?
Nate Breznau et so much al. Breznau is studying at the University of Bremen. For co-authors, there is a list 165 co-authors are from 94 different academic institutions. The implication of this is that this is not the job of one person or a disgruntled research team.
What did they do?
The research question is: “Do different researchers converge on similar findings when analyzing the same data?”
They did this:
“Thirty-three independent research groups have used identical cross-country survey data to test an established social science hypothesis: that more immigration reduces public support. them to the delivery of government social policies”.
What did they find?
“Rather than converging, the numerical outcomes of the groups vary widely, from large negative to large positive effects of immigration on public support.”
Another way to look at this is to look at actual numerical results produced by different groups, asking the same question, using identical data:
The discussion begins with the following:
“Discussion: Results from our controlled study design in a large community-based research effort with 73 groups demonstrate that analyzing the same hypothesis with the same data has may lead to significant differences in statistical estimates and significant conclusions. In fact, no two teams achieve the same set of numerical results or make the same critical decisions during data analysis.”
You want to know more?
If you really want to know why researchers who are asking the same question using the same data are giving completely different and conflicting answers, you’ll really have to read the paper. .
How does this relate to Multi-analyst approach?
Last June, I wrote about how to approach scientific questions named Multi-analyst approach.
The advertised multi-analyst approach is:
“We argue that the current mode of scientific publication – focusing on a single analysis – leads to ‘model myopia’, a restriction that considers statistical assumptions. That leads to overconfidence and poor prediction. …. To assess the certainty of their conclusions, researchers should include data in multiple analyses; Ideally, these would be done by one or more independent groups.“
This new paper, being discussed today, has this to say:
“Even highly skilled scientists are motivated to arrive at accurate results that differ greatly from what they find when given the same data and hypotheses to test. The standard presentation and consumption of scientific results did not reveal the overall research decisions made during the study. Our conclusion was that we had tapped into a hidden universe of idiosyncratic researcher variation. “
And, that means, to you and me, that neither the multi-analyst approach nor the multi-analysis team approach will solve the Real World™ problem presented by the inherent uncertainty of the process. Modern scientific research – “multiple analysts/teams” will use slightly different approaches, different statistical techniques, and slightly different versions of available data. Teams make hundreds of small assumptions, essentially treating each one as a “best practice”. And because of these small differences, each team achieves a completely defensible result that is sure to pass peer review, but each team gives different, even contradictory, answers, for the same question asked about the same data.
This is the exact problem we see at CliSci on a daily basis. We see this problem in Covid statistics, nutritional science, epidemiology of all kinds and many other fields. This is a separate issue with diverse biases affecting politically and ideologically sensitive audiences, pressure within academia to find an outcome that aligns with current consensus. in a person’s field and friend review disease is escalating.
In Climate Science, we see the false belief that more handling – averaging, anomaly, folding, smoothing, etc. – reducing uncertainty. The flip side is true: more processing increases uncertainty. Climate science doesn’t even recognize the simplest type of uncertainty – the initial uncertainty – but wants to get rid of it.
Another approach that would certainly be suggested is that the results of the different findings should now be averaged or found the mean – a kind of consensus – of the multitude of findings. The resulting image shows that this approach is circular with 57.7% weighted distribution. This idea is no more valid than averaging chaotic model results as is done in Climate Science – in other words, worthless.
Pielke Jr. hint in a recent presentation and follow-up Q&A with National Association of Scholars that gathering the really best experts together in one room and hashing out our controversies is probably the best approach. Pielke Jr. are an acknowledged fan of the approach used by the IPCC – but only as long as their findings are not swayed by politicians. Even so, I tend to agree that attracting the best and most honest scientists (not fighting) in a field, along with experts in statistics and mathematical evaluation of programming, all in one virtual room with the command to evaluate and hash out the biggest differences in findings can produce improved results.
Do not ask me
I am not an active researcher. I don’t have the right solution for the “Three Cs” – the reality is that the world is 1) Complex, 2) Complicated and 3) Strange. Those three factors add up to create uncertainty inherent to every problem. This new study adds another layer – the uncertainty caused by countless petty decisions by researchers when analyzing a research question.
It seems hope that the multi-analyst/multi-analysis team approach will help solve some of the complex scientific questions of the day have been dashed. It also seems possible that when groups of self-proclaimed independents arrive at answers that seem like too much of a consensus – we should be skeptical, not reassurance.
# # # # #
Author’s comment:
If you are interested in why scientists disagree, even on simple questions, then you must definitely read this article, right now. Pre-print .pdf here.
If it doesn’t change your understanding of the difficulties of doing good honest science, you may need a brain transplant. …. Or at least a new advanced critical thinking skills course.
As always, don’t listen to me for any of this. Read the article, and maybe go back and read my previous part on Many analysts.
Good science is not easy. And when we ask harder and harder questions, it doesn’t get easier.
The easiest thing in the world is to come up with new hypotheses that seem plausible or to make heavenly predictions for futures beyond our lifetimes. Popular Science Magazine make such a business plan. Today’s “theoretical physics” seems to be its game – who can come up with the craziest yet believable idea of ”how things really are”.
Thanks for reading.
# # # # #
Related
