Listen to this article
Academic research that seems to defy common sense is nothing new. We have all read articles where the findings appear counterintuitive. Now a trio of academics have pointed out what many have always suspected, that it is easy to have research published in respectable journals without paying sufficient attention to the evidence and whether or not it supports the findings.
“It is unacceptably easy to publish ‘statistically significant’ evidence consistent with any hypothesis,” say Joseph Simmons and Uri Simonsohn, operations and information management professors at the Wharton school at the University of Pennsylvania and their colleague Leif Nelson, an associate professor in the marketing group at Haas at UC Berkeley.
The academics suggest that because it is relatively straightforward to find evidence to support any hypothesis and counterintuitive findings are more likely to be noticed and discussed there is a “temptation to [conduct research] that in the end doesn’t contribute to society very much,” says Prof Simonsohn.
As a result they say, instead of asking important questions which will benefit society academics too often ask questions that are more likely to generate media attention.
“We are missing out on important truths about the world” adds Prof Simonsohn.
In their paper “False-positive psychology: undisclosed flexibility in data collection and analysis allow presenting anything as significant,” the writers suggest that academics should reveal how they have collected and analysed the data they have used to support their evidence.
The most costly error in the research process they say is a false positive in which evidence has been obtained even though the effect is not real. While they acknowledge that false positives will naturally occur from time to time they occur far too often because researchers have made decisions during the course of collecting and analysing their data. When this is combined with a researcher’s desire to find a statistically significant result it can ultimately result in wasted resources “by inspiring investment in fruitless research programmes”.
The authors suggest several guidelines for researchers, including the proposal that researchers should “demonstrate that their results do not hinge on arbitrary analytic decisions”.
“Our whole reason to exist as scientists is that we are supposed to be better equipped at finding out what’s true,” adds Prof Simonsohn.
“If our methodology takes away that advantage, we might as well just ask for researchers’ opinions instead of their statistical analyses.”
The paper is published online on the Social Science Research Network.
● Although many people save for a rainy day and are careful when spending on routine and planned purchases, this is not the case when it comes to unexpected spending.
When faced with an exceptional purchase – such as replacing a broken television or buying a birthday present – individuals tend to throw their budgetary caution to the wind, spending more than they had perhaps anticipated.
Authors Abigail Sussman of Princeton University and Adam Alter, an assistant professor of marketing at the Stern School of Business at NYU have looked at consumer spending patterns and discovered that while consumers can accurately gauge ordinary expenditure – the weekly grocery bill for example – they routinely underestimate how much they spend on unexpected purchases.
“This tendency to under-budget for so-called ‘exceptional’ purchases occurs because although each purchase is unusual in isolation, when combined they tend to occur with unexpected frequency” say the writers in their paper “The exception is the rule: underestimating and overspending on exceptional expenses.”
For example they say, a fiftieth birthday, tickets to a long-anticipated concert and replacing a large household item are all one-off events and when seen in isolation would all appear to justify a loosening of the fiscal reins. But if consumers were to understand the connections and accept that such one-off events happen at frequent intervals they might well make different spending decisions.
“Overall, this tendency results in overspending and under-saving” says the writers. They suggest that if consumers understood the differences in accounting for ordinary and exceptional expenses they would be able to make “wiser budgeting decision”.
The paper will be published in the December edition of the Journal of Consumer Research.
Get alerts on Business education when a new story is published