Part II: Data Manipulation, Visualization, and Basic Analysis in R

Prepared by: Yuxiao Zhang & Michel Cordoba
Maintained by: PUPIL Lab
Last updated: July 2026

Data note. The datasets included with this resource are small demonstration datasets provided for instructional practice. They are intended for learning R syntax, data manipulation, data visualization, basic analysis, and related examples in this handout.

1. Preparing the Example Data

1.1 Load packages and data

This part uses the 200-case PISA example dataset introduced in Part 1. The examples use relative paths, so render or run this file from inside the Part_2_Data_Manipulation_Visualization_Analysis folder.

library(haven)
library(dplyr)
library(ggplot2)

data_demo <- read_sav("data/PISA_200.sav")

# Remove rows with missing school type values for the examples in this part.
data_demo <- data_demo[!is.na(data_demo$Sch_Type), ]

dim(data_demo)

## [1] 197   7

The dataset includes student identifiers, gender, school type, and several math achievement scores.

2. Data Manipulation with Base R

2.1 What is data manipulation?

Data manipulation refers to the process of changing data to make it more suitable for analysis. This can include tasks like selecting specific columns, filtering rows based on conditions, creating new variables, summarizing groups, and combining datasets.

2.2 Base R indexing and logical operators

Base R includes logical operators that allow you to select rows and columns using conditions:

• & is the logical AND operator; it returns TRUE only if both conditions are true.
• | is the logical OR operator; it returns TRUE if at least one condition is true.
• ! is the logical NOT operator; it negates a condition.
• == is the equals operator; it checks whether a value is equal to a specified value.
• != is the not-equal operator; it checks whether a value is different from a specified value.

Select all rows where Math_Reasoning is greater than 500 and less than 520, and keep only columns 2 through 5.

data_demo[data_demo$Math_Reasoning > 500 & data_demo$Math_Reasoning < 520, 2:5]

## # A tibble: 12 × 4
##    Gender     Sch_Type   Math_Overall Math_Reasoning
##    <dbl+lbl>  <dbl+lbl>         <dbl>          <dbl>
##  1 2 [Male]   3 [Public]         515.           507.
##  2 1 [Female] 3 [Public]         487.           511.
##  3 1 [Female] 3 [Public]         560.           518.
##  4 1 [Female] 3 [Public]         589.           502.
##  5 1 [Female] 3 [Public]         441.           505.
##  6 2 [Male]   3 [Public]         463.           501.
##  7 1 [Female] 3 [Public]         435.           515.
##  8 2 [Male]   3 [Public]         554.           502.
##  9 2 [Male]   3 [Public]         584.           509.
## 10 1 [Female] 3 [Public]         525.           501.
## 11 2 [Male]   3 [Public]         482.           517.
## 12 2 [Male]   3 [Public]         465.           502.

Select all rows where Gender is male (2) and Sch_Type is not private independent (1); keep only columns 2, 3, and 4.

data_demo[data_demo$Gender == 2 & data_demo$Sch_Type != 1, 2:4]

## # A tibble: 96 × 3
##    Gender    Sch_Type   Math_Overall
##    <dbl+lbl> <dbl+lbl>         <dbl>
##  1 2 [Male]  3 [Public]         542.
##  2 2 [Male]  3 [Public]         399.
##  3 2 [Male]  3 [Public]         403.
##  4 2 [Male]  3 [Public]         561.
##  5 2 [Male]  3 [Public]         463.
##  6 2 [Male]  3 [Public]         446.
##  7 2 [Male]  3 [Public]         463.
##  8 2 [Male]  3 [Public]         687.
##  9 2 [Male]  3 [Public]         436.
## 10 2 [Male]  3 [Public]         522.
## # ℹ 86 more rows

2.3 Exercise: base R filtering

Create a dataset containing all female students from private schools (Sch_Type values 1 or 2) who have a Math_Overall score above 550.

private_high_math <- data_demo[
  data_demo$Gender == 1 &
    data_demo$Sch_Type %in% c(1, 2) &
    data_demo$Math_Overall > 550,
  c("Gender", "Sch_Type", "Math_Overall")
]

private_high_math

## # A tibble: 1 × 3
##   Gender     Sch_Type                Math_Overall
##   <dbl+lbl>  <dbl+lbl>                      <dbl>
## 1 1 [Female] 1 [Private independent]         645.

This subset is small, which is a useful reminder to check row counts after filtering.

3. Data Manipulation with dplyr

3.1 Key dplyr functions

The dplyr package is designed to make data manipulation simpler and more intuitive. It helps you work with your data in a clean and readable way by providing easy-to-use functions.

Key functions in dplyr include:

• select(): Select specific columns from your dataset.
• filter(): Filter rows based on specific conditions.
• mutate(): Create new variables based on existing ones.
• group_by(): Group data by specific variables.
• summarize(): Summarize data, such as calculating means or counts, for each group.

3.2 Pipe operator

The |> symbol, called the pipe, is used to link multiple operations together in a clear and readable way. Instead of creating multiple intermediate variables, you can use |> to chain a series of steps together.

3.3 Data manipulation examples

Create a new dataset containing only the columns Gender, Sch_Type, Math_Reasoning, and Math_Quantity; retain only female participants; and convert Math_Reasoning by dividing it by 100.

# Using intermediate objects
new_data_a <- select(data_demo, Gender, Sch_Type, Math_Reasoning, Math_Quantity)

new_data_a <- filter(new_data_a, Gender == 1)

new_data_a <- mutate(new_data_a, Math_Reasoning_new = Math_Reasoning / 100)

head(new_data_a)

## # A tibble: 6 × 5
##   Gender     Sch_Type   Math_Reasoning Math_Quantity Math_Reasoning_new
##   <dbl+lbl>  <dbl+lbl>           <dbl>         <dbl>              <dbl>
## 1 1 [Female] 3 [Public]           268.          203.               2.68
## 2 1 [Female] 3 [Public]           649.          586.               6.49
## 3 1 [Female] 3 [Public]           405.          377.               4.05
## 4 1 [Female] 3 [Public]           453.          516.               4.53
## 5 1 [Female] 3 [Public]           401.          459.               4.01
## 6 1 [Female] 3 [Public]           478.          400.               4.78

Here is the same example using the pipe operator.

new_data_b <- data_demo |>
  select(Gender, Sch_Type, Math_Reasoning, Math_Quantity) |>
  filter(Gender == 1) |>
  mutate(Math_Reasoning_new = Math_Reasoning / 100) |>
  select(-Math_Reasoning)

head(new_data_b)

## # A tibble: 6 × 4
##   Gender     Sch_Type   Math_Quantity Math_Reasoning_new
##   <dbl+lbl>  <dbl+lbl>          <dbl>              <dbl>
## 1 1 [Female] 3 [Public]          203.               2.68
## 2 1 [Female] 3 [Public]          586.               6.49
## 3 1 [Female] 3 [Public]          377.               4.05
## 4 1 [Female] 3 [Public]          516.               4.53
## 5 1 [Female] 3 [Public]          459.               4.01
## 6 1 [Female] 3 [Public]          400.               4.78

Calculate the average of Math_Reasoning, Math_Relation, and Math_Quantity scores for each student. Then summarize the average of these scores for each gender group.

gender_summary <- data_demo |>
  mutate(Average_Math = (Math_Reasoning + Math_Relation + Math_Quantity) / 3) |>
  group_by(Gender) |>
  summarize(
    mean_reasoning = mean(Math_Reasoning),
    mean_relation = mean(Math_Relation),
    mean_quantity = mean(Math_Quantity),
    mean_total = mean(Average_Math),
    frequency = n(),
    .groups = "drop"
  )

knitr::kable(
  gender_summary,
  digits = 2,
  caption = "Table 1. Average math sub-scores and frequency by gender."
)

Table 1. Average math sub-scores and frequency by gender.

Gender	mean_reasoning	mean_relation	mean_quantity	mean_total	frequency
1	455.92	461.90	457.40	458.41	96
2	465.13	470.16	469.21	468.17	101

The summary table compares average math sub-scores and total frequency across gender groups.

3.4 Exercise: dplyr filtering and grouping

Create a new dataset containing only the columns Math_Reasoning, Math_Relation, and Gender. Add a new column called Average_Score that represents the average of the Math_Reasoning and Math_Relation scores, then filter the dataset to keep only rows where Average_Score is 600 or greater.

exercise_data <- data_demo |>
  select(Math_Reasoning, Math_Relation, Gender) |>
  mutate(Average_Score = (Math_Reasoning + Math_Relation) / 2) |>
  filter(Average_Score >= 600)

head(exercise_data)

## # A tibble: 6 × 4
##   Math_Reasoning Math_Relation Gender     Average_Score
##            <dbl>         <dbl> <dbl+lbl>          <dbl>
## 1           649.          560. 1 [Female]          604.
## 2           695.          690. 2 [Male]            692.
## 3           654.          705. 1 [Female]          680.
## 4           600.          621. 2 [Male]            610.
## 5           611.          688. 2 [Male]            649.
## 6           611.          596. 2 [Male]            604.

dim(exercise_data)

## [1] 15  4

Count the number of students by gender in this new dataset.

exercise_summary <- exercise_data |>
  group_by(Gender) |>
  summarize(frequency = n(), .groups = "drop")

knitr::kable(
  exercise_summary,
  caption = "Table 2. Number of students by gender after filtering to high average scores."
)

Table 2. Number of students by gender after filtering to high average scores.

Gender	frequency
1	7
2	8

The table shows how many students remain in each gender group after the filtering rule is applied.

4. Data Visualization

4.1 Introduction to ggplot2

ggplot2 is one of the most popular tools in R for data visualization. It allows you to create a wide range of plots that can help you understand your data better.

4.2 Grammar of graphics concept

The idea behind ggplot2 is the “grammar of graphics,” which means that every plot is built layer by layer. You start with data, map it visually using different types of plots, and then add more elements to tell your story.

4.3 Histogram: visualizing continuous variables

The following plot is built in layers:

• Layer 1: Initialize the plot with ggplot().
• Layer 2: Map Math_Quantity to the x-axis using aes().
• Layer 3: Add the histogram layer with geom_histogram().
• Layer 4: Customize labels and axis breaks.

ggplot(data_demo, aes(x = Math_Quantity)) +
  geom_histogram(binwidth = 50, fill = "gray65", color = "gray20") +
  labs(
    title = "Distribution of Math_Quantity",
    x = "Math_Quantity",
    y = "Frequency"
  ) +
  scale_x_continuous(breaks = seq(0, 700, by = 50)) +
  theme_minimal()

Figure 1. Histogram of Math Quantity scores using ggplot2.

The histogram shows that most Math_Quantity scores are clustered in the middle of the score range, with fewer observations at the extremes.

This is the same idea using basic R.

hist(
  data_demo$Math_Quantity,
  breaks = seq(175, 725, by = 50),
  main = "Distribution of Math_Quantity",
  xlab = "Math_Quantity",
  xaxt = "n",
  col = "gray70",
  border = "gray20"
)
axis(1, at = seq(175, 725, by = 50))

Figure 2. Histogram of Math Quantity scores using base R.

4.4 Bar charts: visualizing categorical variables

plot_data <- data_demo |>
  mutate(
    Gender_label = as_factor(Gender),
    Sch_Type_label = as_factor(Sch_Type)
  )

ggplot(plot_data, aes(x = Gender_label)) +
  geom_bar(width = 0.5, fill = "gray65", color = "gray20") +
  labs(
    title = "Distribution of Gender",
    x = "Gender",
    y = "Frequency"
  ) +
  theme_minimal()

Figure 3. Bar chart of gender distribution using ggplot2.

This chart again shows that the example dataset has a nearly balanced gender distribution.

4.5 Scatter plots: exploring relationships between variables

ggplot(data_demo, aes(x = Math_Relation, y = Math_Reasoning)) +
  geom_point(aes(shape = as_factor(Gender)), size = 2, alpha = 0.8) +
  geom_smooth(method = "lm", se = FALSE, color = "gray20", linewidth = 1) +
  labs(
    title = "Math_Relation vs. Math_Reasoning",
    x = "Math_Relation",
    y = "Math_Reasoning",
    shape = "Gender"
  ) +
  theme_minimal()

Figure 4. Scatter plot of Math Relation and Math Reasoning scores.

The scatter plot shows a positive relationship: students with higher Math_Relation scores tend to have higher Math_Reasoning scores. Shape, not only color, is used to distinguish gender groups.

4.6 Boxplots: comparing groups

Use geom_boxplot() to compare Math_Overall scores across genders.

ggplot(plot_data, aes(x = Gender_label, y = Math_Overall)) +
  geom_boxplot(fill = "gray80", color = "gray20") +
  labs(
    title = "Math Overall Score by Gender",
    x = "Gender",
    y = "Math Overall Score"
  ) +
  theme_minimal()

Figure 5. Boxplot comparing Math Overall scores by gender.

The boxplot suggests that the two gender groups have broadly similar Math_Overall distributions in this example dataset.

Outcome: You should now be able to create basic plots in R using ggplot2, customize your visualizations, and understand how these visual tools can help you explore and communicate your data effectively.

5. Basic Statistical Analysis

5.1 T-test

Use the function t.test() in R to compare the means of two independent groups. In this example, Math_Quantity is the dependent variable and Gender is the grouping variable.

t.test(Math_Quantity ~ Gender, data = data_demo, var.equal = TRUE)

## 
##  Two Sample t-test
## 
## data:  Math_Quantity by Gender
## t = -0.85929, df = 195, p-value = 0.3912
## alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
## 95 percent confidence interval:
##  -38.92070  15.29775
## sample estimates:
## mean in group 1 mean in group 2 
##        457.4008        469.2122

The t-test output reports the group means, test statistic, degrees of freedom, p-value, and confidence interval for the difference between group means.

5.2 Correlation analysis

Use the cor() function to calculate correlations among continuous variables and cor.test() to obtain additional statistics, such as p-values and confidence intervals.

cor(data_demo[, c("Math_Reasoning", "Math_Quantity", "Math_Relation")])

##                Math_Reasoning Math_Quantity Math_Relation
## Math_Reasoning      1.0000000     0.8455608     0.8288252
## Math_Quantity       0.8455608     1.0000000     0.9280377
## Math_Relation       0.8288252     0.9280377     1.0000000

The correlation matrix shows that these math sub-scores are positively related to one another.

cor.test(data_demo$Math_Reasoning, data_demo$Math_Quantity)

## 
##  Pearson's product-moment correlation
## 
## data:  data_demo$Math_Reasoning and data_demo$Math_Quantity
## t = 22.117, df = 195, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.8003739 0.8811943
## sample estimates:
##       cor 
## 0.8455608

The correlation test provides the estimated correlation between Math_Reasoning and Math_Quantity, along with a confidence interval and p-value.

5.3 Linear regression

Use the lm() function to perform a linear regression.

linear_model <- lm(Math_Reasoning ~ Math_Quantity + Math_Relation, data = data_demo)
summary(linear_model)

## 
## Call:
## lm(formula = Math_Reasoning ~ Math_Quantity + Math_Relation, 
##     data = data_demo)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -150.085  -30.188    2.439   27.609  131.571 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   85.35310   17.01096   5.018 1.18e-06 ***
## Math_Quantity  0.50408    0.09188   5.486 1.27e-07 ***
## Math_Relation  0.30393    0.09592   3.169  0.00178 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 46.17 on 194 degrees of freedom
## Multiple R-squared:  0.729,  Adjusted R-squared:  0.7262 
## F-statistic: 260.9 on 2 and 194 DF,  p-value: < 2.2e-16

The regression output estimates how Math_Quantity and Math_Relation are associated with Math_Reasoning when both predictors are included in the same model.

5.4 Exercise: additional statistical analysis

Perform a t-test to compare the mean Math_Overall scores between students from private independent schools (1) and public schools (3).

data_ttest2 <- data_demo |>
  filter(Sch_Type %in% c(1, 3))

t.test(Math_Overall ~ Sch_Type, data = data_ttest2, var.equal = TRUE)

## 
##  Two Sample t-test
## 
## data:  Math_Overall by Sch_Type
## t = 1.081, df = 194, p-value = 0.2811
## alternative hypothesis: true difference in means between group 1 and group 3 is not equal to 0
## 95 percent confidence interval:
##  -28.39877  97.28148
## sample estimates:
## mean in group 1 mean in group 3 
##        496.3496        461.9083

Calculate the correlation between Math_Reasoning and Math_Relation for each gender category.

correlation_by_gender <- data_demo |>
  group_by(Gender) |>
  summarize(
    correlation = cor(Math_Reasoning, Math_Relation),
    frequency = n(),
    .groups = "drop"
  )

knitr::kable(
  correlation_by_gender,
  digits = 3,
  caption = "Table 3. Correlation between Math Reasoning and Math Relation by gender."
)

Table 3. Correlation between Math Reasoning and Math Relation by gender.

Gender	correlation	frequency
1	0.844	96
2	0.811	101

The table shows that the two math sub-scores are positively correlated within each gender group.

Outcome: You should now be able to conduct basic statistical analyses in R, including t-tests, correlation analyses, and linear regression.

6. Further Learning

Use these resources to continue learning:

• dplyr Cheatsheet
• ggplot2 Cheatsheet
• R for Data Science, second edition

Next Step

After completing Part II, continue to Part III: Core Programming Tools in R.