Statistical significance: What it does and does not mean

At a recent statistics workshop I was conducting to a group of medical students I asked how many could define statistically significance. Everyone had heard of it and knew that p < .05 constituted a statistically significant effect, but no one really understood what it meant.

To understand what it means, it is crucial to remember that statistics is typically used for 2 purposes: 1) to describe and 2) to make inferences about a population from a sample drawn from the population.

With few exceptions, when we conduct a study, the data we collect is from a sample. But it is not really the sample per se in which we are interested. We almost always want to be able to generalize from the sample to the population from which the sample was drawn. If we administer a new painkiller to one group of individuals and a placebo to another group, and find that the results indeed show greater pain reduction in the drug group, we need to know whether the difference obtained from the sample reflects a “real” difference in the population or whether, alternatively, there is no difference between the groups and the observed difference is due to chance alone.

Let’s look at an example.

The anchoring bias: The powerful pull of first impressions

In an early experiment by Daniel Kahneman and Amos Tversky, participants were asked to compute the product of a series of numbers in 5 secs. But in one group the series was ascending,

1 x 2 x 3 x 4 x 5 x 6 x 7 x 8

whereas in the other it was descending:

8 x 7 x 6 x 5 x 4 x 3 x 2 x 1

As 5 secs was insufficient time to finish, all Ps were interrupted before finishing. They were then asked to estimate the final answer. Those given the ascending sequence, who began with small numbers, gave a median estimate of 512, whereas those given the descending sequence gave a median estimate of 2250. The beginning numbers thus anchored the estimates, proving a very dramatic influence on estimates.

Do standing and treadmill desks benefit workers or improve productivity?

A systematic review was recently published in the journal Preventive Medicine. The authors looked at 23 studies involving a standing or treadmill walking intervention compared to regular seated desk work. Included studies examined a wide range of physiological (resting heart rate, blood pressure, total fasting cholesterol, fasting glucose, weight loss, etc) and psychological (work performance, mood states, job satisfaction, quality of life, etc.) outcomes. Of the 23 studies examined, 19 lacked appropriate randomization or control; only 4 were randomized controlled trials.

Pearson vs Spearman: Which should I use to test correlations?

The most commonly used method of assessing correlation is the Pearson Product Moment Correlation, more commonly (and easily) called Pearson’s r. It tests the degree to which two variables are linearly dependent on each other (i.e, correlated). Pearson’s r can take on a value from -1 to +1. An r = 0 means the two variables are completely independent (i.e., unrelated). An r =  +1 means the two variables are completely dependent such that an increase in one variable is associated with an exact same increase in other variable. An r = -1 means the two variables are completely dependent such that an increase in one variable is associated with an exact same decrease in the other variable.

Oral cancer screening

Today I was walking along a major street in Toronto and noticed a billboard advertising for a free oral cancer screening. In the ad they also point to estimates by the Canadian Cancer Society that about 4300 Canadians were diagnosed with oral cancer in 2014. So should one be advised to take the dentist up on his offer?

What Apple’s ResearchKit means for research

ResearchKit

This past Monday, amid the excitement surrounding the introduction of Apple’s new watch and an impossibly thin and light new MacBook computer, what got me most excited was the introduction of something called ResearchKit. The blandness of the name obscures the immense possibilities it makes possible. ResearchKit is Apple’s new software framework for developing health research apps.

National Center for Health Statistics

The NCHS is a veritable goldmine of health related information and statistics. Their stated mission is to, “is to provide statistical information that will guide actions and policies to improve the health of the American people.”

Here is an overview of the NCHS. Here are some of the topics they cover:

NCHS produces data on a wide range of health indicators such as:

• Health insurance coverage and its relationship to access and utilization of health care services.
• Prevalence of health conditions such as obesity and overweight, cholesterol, hypertension, HIV status, and smoking among the U.S. population.
• Exposure to environmental chemicals.
• Nutrition and physical activity.
• Growth charts to monitor the development of children.
• Patient safety and quality including adverse effects of medical treatment.
• Injuries and disabilities and their impact on health status and functioning.
• Infant mortality, stillbirths, life expectancy, and teen births.
• Leading causes of death specific to age, race, ethnic and gender groups.
• Practice of medicine in the U.S., evolution of health information technology, changes in roles and practices of health care providers, and use of resources.
• Changes in the health care delivery system, including emergency department use and capacity, increasing use of prescription drugs, and increasing demand for community-based long term care.

Here is summary of what and how data are collected.

Here is a summary of current surveys and data collection systems.

StatCrunch: Web-based statistics

StatCrunch is a web-based statistics system that provides the full workflow including data collection, analyses and presentation of results. It was developed by Webster West, a statistics professor at North Carolina State University. The software has been designed for educational purposes and in fact access is typically via textbook purchase although six months of access can be purchased for $12 and 12 months for $23.

Here is a list of statistical features of StatCrunch. Looking at the list it is apparent that this software is lightweight, meaning that it will perform only the more basic statistical functions such as t-tests, multiple/logistic regression, ANOVA, as well as commonly used visualizations such as bar, box and scatter plots, histograms and pie charts.

I have not myself had the chance to try out StatCrunch but if you’re looking to get exposed to statistical analysis and want a dedicated tool (rather than using Microsoft Excel) that is inexpensive, then it may be worth a look.

Click here to go the StatCrunch website.

Time off means enhanced worker productivity

An Oxford Economic report published in February 2014 presents the findings of a survey of 971 employees focusing on how paid time off is perceived and used in the United States. Although their report contains less information regarding their sampling strategy, which does raise questions of validity and generalizability, the results are consistent with a growing body of literature on the subject and therefore are noteworthy. They found that paid time off (PTO) leads to productivity boosts as workers report returning to work refreshed and with a better attitude.

These results suggest that while PTO can be costly for employers, there is evidence of a net gain to productivity, even after accounting for down time resulting from employee absence. Evident from the graph below, taking PTO results in improvements in a wide range of outcomes, including attitude toward one’s work, mental and physical health, as well as improvements in family and social life.

We know from previous research that healthy employees are better employees, take fewer sick days, report less work-related stress, are more engaged in their work, and are less likely to seek out job changes. On the other hand, employees receiving less than 5 PTO days have been found to be more likely to show signs of clinical depression, have trouble sleeping and be planning on changing jobs.

As can be seen in the graph below, after time off, more than 40% of employees reporting feeling recharged and refreshed, more focused and less stressed.

While the obvious costs to organizations surrounding paid time off would seem to promote its minimization, a growing body of literature suggests otherwise. Healthy workers are better more productive workers and we humans need rest and rejuvenation, as well as time with family and friends, in order to be healthy.

---

Oxford Economics (2004). An Assessment of Paid Time Off in the U.S.: Implications for employees, companies, and the economy. http://www.oxfordeconomics.com/my-oxford/projects/280061

When it comes to exercise, faster is not necessarily better

A recent study by researchers in Denmark examined data from 1098 healthy joggers and 3950 healthy non-joggers. These data were collected as part of the prospective Copenhagen City Heart Study.

Adjusting for age, sex, smoking, alcohol intake, education, and diabetes, compared to people who were sedentary, 1 to 2.5 hours of jogging per week was associated with the lowest risk of mortality with a hazard ratio, HR= 0.29. When the researchers divided joggers into light, moderate and strenuous joggers, the lowest risk for mortality was found for the light joggers (HR = 0.22), followed by moderate joggers (HR = 0.66). Strenuous joggers had an increase risk in mortality compared to their sedentary counterparts with HR = 1.97!

So the relationship between jogging intensity and mortality is a “U” shaped one in which light to moderate activity has benefits over being sedentary but too much can be a killer!

The authors conclude, “if the goal is to decrease the risk of death and improve life expectancy, going for a leisurely jog a few times per week at a moderate pace is a good strategy. Higher doses of running are not only unnecessary but may also erode some of the remarkable longevity benefits conferred by lower doses of running.”

---

DMSc, P. S. M., MD, J. H. O., MSc, J. L. M., DMSc, P. L. M., & DMSc, G. B. J. M. (2015). Dose of Jogging and Long-Term Mortality. Journal of the American College of Cardiology, 65(5), 411–419