\(~\)

Review

The Happy Planet dataset contains country-level measures from the Happy Planet Index (HPI), which evaluates how effectively countries achieve long, happy lives while using environmental resources sustainably. Each row represents a country and includes measures related to well-being, life expectancy, ecological impact, and overall HPI performance. More details can be found here: https://happyplanetindex.org/countries/

happy_planet = read.csv("https://remiller1450.github.io/data/HappyPlanet.csv")

Question #1:

  • Part A: Create a scatter plot displaying the relationship between GDPperCapita and Happiness including a LOESS smoother. Describe the strength, form, and direction of the relationship between these variables.
  • Part B: Based upon the visualization you created in Part A, should Pearson’s or Spearman’s correlation be preferred for these data. Briefly explain your choice.
  • Part C: Find the correlation coefficient you chose in Part B.
  • Part D: Suppose your friend looks at these data and concludes: “The wealthier someone is, the happier they are, at least up until a certain point. But past that point additional wealth isn’t associated with increased happiness.” Is your friend’s assessment supported by these data? Briefly explain.

\(~\)

Hypothesis Testing for Correlation

Like what we’ve done with one and two sample data, we can form hypotheses about the correlation between two variables in population we are studying and evaluate them statistically using the sample data.

In a correlation analysis, our null hypothesis is that there is no correlation between the two variables of interest in the population we are studying, while our alternative hypothesis is that there is a non-zero correlation between the variables: \[H_0: \rho = 0 \text{ vs. } H_a: \rho \ne 0\]

We can test these hypotheses using the cor.test() function:

## Load example data
grinnell_peers = read.csv("https://remiller1450.github.io/data/grinnell_peers.csv")

## Correlation test on "Ret_Rate" and "Enrollment"
cor.test(x = grinnell_peers$Ret_Rate, grinnell_peers$Enrollment)
## 
##  Pearson's product-moment correlation
## 
## data:  grinnell_peers$Ret_Rate and grinnell_peers$Enrollment
## t = -3.3365, df = 16, p-value = 0.004185
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.8524774 -0.2477627
## sample estimates:
##        cor 
## -0.6405396

For the moment, we’ll focus on the \(p\)-value of the test, the correlation observed in the sample, and the conclusion we draw from those two pieces of information. In this example, we might conclude:

The data show very strong statistical evidence (p = 0.004) of a negative relationship between retention rate and enrollment among Grinnell’s peer institutions (r = -0.641).

Since cor.test() is performing a T-test, the conditions for this test to produce a reliable estimate of the \(p\)-value are:

  1. Independent observations (no repeated measurements of the same individual, etc.)
  2. The sample size is reasonably large (\(n \geq 30\)), or the variables are approximately Normally distributed.
  3. The involved variables are linearly related.

Additionally, if you are wondering why cor.test() is performing a T-test (instead of a Z-test or something else), the correlation test is mathematically equivalent to testing the slope of a regression line (something we’ll talk about next time), which involves estimating extra population parameters that were not used in the numerator of the test statistic.

Finally, note that we can use the argument method = "spearman" to perform a test on Spearman’s correlation that does not require both variables to be Normally distributed or have a linear relationship to produce a reliable \(p\)-value:

cor.test(x = grinnell_peers$Ret_Rate, grinnell_peers$Enrollment, method = "spearman")
## 
##  Spearman's rank correlation rho
## 
## data:  grinnell_peers$Ret_Rate and grinnell_peers$Enrollment
## S = 1564, p-value = 0.007926
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##        rho 
## -0.6140351

Question #2: In this question you’ll continue using the Happy Planet dataset. Although these data are not a random sample of countries, they provide a convenient example for us to learn the nuances of hypothesis testing for correlation.

  • Part A: Use cor.test() to test whether the Happy Planet data provide statistical evidence of an association between GDPperCapita and Happiness across countries. Report the \(p\)-value of the test as part of a proper 1-sentence conclusion. Be sure to use a method that reflects what you learned in Question 1 about the form of the relationship between these variables.
  • Part B: Consider the hypothesis that the population of a country is associated with the happiness of its citizens. As a first step, construct a scatterplot of these two variables, then decide whether Pearson’s or Spearman’s correlation is a more appropriate measure of association.
  • Part C: Perform the hypothesis test described in Part B and provide a proper 1-sentence conclusion that includes evidence, context, and effect size.

\(~\)

Question #3: This question doesn’t involve any particular dataset.

  • Part A: Is it possible that sample data could show a strong correlation, such as \(r=0.7\), but a hypothesis test using these data might find insufficient evidence of a correlation between these variables in the population of interest? If yes, under what circumstances might this occur?
  • Part B: Is it possible that sample data could show a weak correlation, such as \(r=0.1\), but a hypothesis test using these data might find strong evidence of a correlation between these variables in the population of interest? If yes, under what circumstances might this occur?

\(~\)

Scatterplot and Correlation Matrices

Some analyses might involve several quantitative variables. Rather than creating scatterplots and finding correlations for each individual pairing ourselves, we can rely on functions created by others to do so.

However, the first step in using these functions is to create a version of our data containing only the quantitative variables of interest. This is done using the select() function from the dplyr library:

## Select only a few variables of interest
library(dplyr)
gp_select_vars = select(grinnell_peers, Adm_Rate, Med_ACT, Enrollment, Med_Grad_Debt)

The new data frame, gp_select_vars, now contains only the four variables we named inside of select(). We can now use the pairs.panels() function from the psych library to create a scatterplot matrix (with correlations shown above the diagonal):

# install.packages("psych")  # You might need to install this package
library(psych)
pairs.panels(gp_select_vars, method = "pearson", smooth = TRUE, ellipses = FALSE)

We could also get just the correlation matrix using the cor() function introduced in our most recent lecture:

cor(x = gp_select_vars, method = "pearson")
##                 Adm_Rate    Med_ACT Enrollment Med_Grad_Debt
## Adm_Rate       1.0000000 -0.8242186  0.6343496     0.7743119
## Med_ACT       -0.8242186  1.0000000 -0.5722073    -0.7711736
## Enrollment     0.6343496 -0.5722073  1.0000000     0.6253372
## Med_Grad_Debt  0.7743119 -0.7711736  0.6253372     1.0000000

Note that instead of providing separate x and y variables, we obtain a correlation matrix by providing a data frame to the x argument without specifying a y variable.

Question #4:

  • Part A: Create a data frame containing only the Happiness, LifeExpectancy, Footprint, GDPperCapita, and Population variables from the Happy Planet data.
  • Part B: Use the pairs.panels() function to create a scatterplot matrix of the data using Pearson’s correlation. Does the use of Pearson’s correlation appear problematic for any of the combinations of variables displayed in this plot? Briefly explain.
  • Part C: Use cor() to create a correlation matrix of the data using Spearman’s correlation. Which variable is most strongly associated with Happiness?
  • Part D: Suppose you performed a hypothesis test on every combination of variables involved in the correlation matrix you created in Part C, using \(\alpha = 0.1\) as the threshold for statistical significance. If none of these variables were related, how many significant findings would you expect to occur purely by chance?
  • Part E: Suppose we were concerned about Type 1 errors and decided to only test the variable showing the strongest relationship with happiness. Would this eliminate the problems associated with performing multiple hypothesis tests? Briefly explain your reasoning.

\(~\)

Application

The Hollywood Movies dataset contains audience ratings and box office performance data for a sample of Hollywood films released between 2007 and 2013.

movies = read.csv("https://remiller1450.github.io/data/HollywoodMovies.csv")

The variables of interest in our analysis will be:

  • Budget - how much money was spent to produce the movie (millions of dollars)
  • AudienceScore - how viewers of the movie rated it on a 0-100 scale
  • RottenTomatoes - how official movie critics from the website Rotten Tomatoes rated the movie on a 0-100 scale
  • Profitability - the percentage of movie’s total costs that were earned as gross revenue, with values lower than 100 reflecting movies that lost money, and values above 100 reflecting movies that made money

Question #5:

  • Part A: Create a data frame containing only the variables described above, then use the pairs.panels() function to create a scatterplot matrix of the data using Pearson’s correlation. Does the use of Pearson’s correlation appear problematic for any of the combinations of variables displayed in this plot? Briefly explain.
  • Part B: Using your scatterplot matrix from Part A, identify a variable that you think shares the most interesting relationship with Profitability, and qualitatively describe the form, strength, and direction of that relationship. You are encouraged to use ggplot() to create an individual scatterplot involving only these two variables to help you provide a clearer qualitative assessment.
  • Part C: Perform a hypothesis test assessing the statistical evidence for the variables you analyzed in Part B being correlated. Be sure to choose an appropriate correlation test (Pearson’s or Spearman’s). Report a 1-sentence conclusion including the \(p\)-value of the test and appropriate context.
  • Part D: Suppose that your friend was not sure whether to use Pearson’s or Spearman’s correlation for their hypothesis test, so they decided to perform both tests and report only the one that produced the smaller \(p\)-value. Is anything about this approach problematic? Briefly explain. Hint: Think about another scenario where we perform 100 hypothesis tests, and the null hypothesis is true for every test. Would it be advisable to report only the test with the lowest \(p\)-value?