25  Matching

In Chapter 4, we discussed three basic causal diagrams and suggested that the concept of nonparametric conditioning on \(Z\) is more demanding than simply including \(Z\) as another regressor in a linear regression model. This chapter develops this idea more fully and examines practical approaches to properly conditioning on confounding variables.

Tip

The code in this chapter uses the following packages. For instructions on how to set up your computer to use the code found in this book, see Section 1.2. Quarto templates for the exercises below are available at https://github.com/iangow/far_templates.

25.1 Background on auditor choice

For concreteness, we will explore the basic ideas of this chapter using the setting of auditor choice and its effects. An extensive literature has examined the question of whether “Big N” auditors produce higher audit quality. This question is examined in two of the papers we will study in this chapter (DeFond et al., 2017; Lawrence et al., 2011).

For data on auditors, the table comp.funda has a column labelled au , which lines up with aucd in comp.r_auditors.

pg <- dbConnect(RPostgres::Postgres(), bigint = "integer")

company <- tbl(pg, Id(schema = "comp", table = "company"))
funda <- tbl(pg, Id(schema = "comp", table = "funda"))
r_auditors <- tbl(pg, Id(schema = "comp", table = "r_auditors"))

While Lawrence et al. (2011) refers to the “Big Four”, they are actually referring to the current group that DeFond et al. (2017) refer to as the “Big N”. The latter term alludes to the prior incarnations of grouping of the top audit firms, including the “Big Five” (prior to the demise of Arthur Andersen” and the “Big Eight” (prior to mergers in the late 1980s) (see Gow and Kells, 2018 for more on this history).

r_auditors |> collect(n = 12)
Table 25.1: Codes for top auditors
aucd audesc
0 Unaudited
1 Arthur Andersen
2 Arthur Young
3 Coopers & Lybrand
4 Ernst & Young
5 Deloitte & Touche
6 KPMG
7 PricewaterhouseCoopers
8 Touche Ross
9 Other
10 Altschuler, Melvoin, and Glasser
11 BDO

Looking closer at r_auditors, we see that Arthur Andersen has an aucd of 1. Note that Arthur Young is now part of Ernst & Young, Coopers & Lybrand is now part of PricewaterhouseCoopers, and Touche Ross is now part of Deloitte & Touche. So a reasonable approach to “Big N” would appear to be big4 = aucd %in% 1:8L.1 It’s commonly understood that most large firms choose a Big Four auditor. But rather than simply accepting that, we can look at the data.

Here we focus on firms with meaningful financial statements (sale > 0, at > 0) and fiscal 2019, the latest fiscal year with complete data at the time of first writing this chapter.

size_big4 <-
  funda |>
  filter(indfmt == "INDL", datafmt == "STD",
         consol == "C", popsrc == "D") |>
  filter(sale > 0, at > 0, fyear == 2019, !is.na(au)) |>
  mutate(au = as.integer(au)) |>
  select(gvkey, datadate, fyear, au, prcc_f, csho) |>
  mutate(big4 = au %in% 1:8L,
         mkt_cap = prcc_f * csho * 1e6) |>
  arrange(gvkey, datadate) |>
  collect()

We calculate market share by “bins” where each bin represents an order of magnitude of market capitalization. For example, (9, 10] includes all firms with market capitalization over $1 billion (\(10^9\)) and less than or equal to $10 billion (\(10^{10}\)).

size_big4 |> 
  filter(mkt_cap > 1e6) |> 
  mutate(log_mkt_cap = log10(mkt_cap)) |>
  mutate(bin = cut(log_mkt_cap, breaks = seq(6, 13, 1))) |>
  group_by(bin) |>
  summarize(big4_perc = 100 * mean(big4),
            big4_num = sum(big4),
            non_big4_num = sum(!big4))
bin big4_perc big4_num non_big4_num
(6,7] 16.364 63 322
(7,8] 27.555 337 886
(8,9] 58.034 1051 760
(9,10] 89.671 1580 182
(10,11] 96.761 687 23
(11,12] 97.849 91 2
(12,13] 100.000 1 0

Interestingly, one of the two cases of companies with an apparent market capitalization over $100 billion and a non–Big Four auditor is Vanjia, which apparently misstated the number of its shares outstanding as 8,550 million rather than 30 million.2

size_big4 |>
  filter(!big4, mkt_cap > 1e11)
gvkey datadate fyear au prcc_f csho big4 mkt_cap
014447 2019-12-31 2019 9 92.988 2518.262 FALSE 234168146856
026478 2019-12-31 2019 9 14.380 8550.000 FALSE 122949000000

We can also present the data as a histogram. From either the table or the histogram, it is clear that the Big Four have overwhelming market share among the largest firms and have most of the market even among firms with market capitalization in the $100 million-to-$1 billion range.

size_big4 |> 
  filter(mkt_cap > 1e6) |> 
  ggplot(aes(x = log10(mkt_cap), fill = big4)) +
  geom_histogram(breaks = seq(6, 12.25, 0.25))
Figure 25.1: Histogram of market capitalization with auditor type

25.2 Simulation analysis

We will use simulation analysis to examine more closely how propensity-score matching works and what it can and cannot do. We take the actual empirical distribution of market capitalization from the data depicted in the histogram above and give each firm a probability of using a Big Four auditor equal to the observed conditional probability for the histogram bin it is in.

prob_big4 <-
  size_big4 |>
  filter(mkt_cap > 1e6) |> 
  select(big4, mkt_cap) |>
  mutate(log_mkt_cap = log10(mkt_cap)) |>
  mutate(bin = cut(log_mkt_cap, breaks = seq(6, 12.25, 0.25))) |>
  group_by(bin) |>
  mutate(p_big4 = mean(big4)) |>
  ungroup()

We then draw 5000 firms from this distribution and assign each firm an auditor based on the conditional probabilities calculated above. A critical assumption here is that conditional on market capitalization whether a Big Four auditor is chosen is completely random. Finally, we calculate a measure of audit quality that is a non-linear function of market capitalization and a random noise component.

n_firms <- 5000

set.seed(2021)
sim_auditor <-
  prob_big4 |>
  sample_n(size = n_firms) |>
  mutate(rand = runif(nrow(pick(everything()))),
         big4 = rand < p_big4,
         epsilon = rnorm(nrow(pick(everything()))),
         id = 1:nrow(pick(everything()))) |>
  mutate(audit_quality = big4 * 0 + mkt_cap ^ (1/3) * 0.003 + epsilon) |>
  select(id, big4, mkt_cap, log_mkt_cap, audit_quality)

Given this is a simulation, we know that the correct causal diagram is the following:

Figure 25.2: Market capitalization is a confounder

We draw the arrow connecting \(X\) and \(Y\) with a dashed line because the true coefficient on big4 is zero, so in this simulation there is no actual causal relation between big4 and audit_quality.

The good news from this causal diagram is that correctly conditioning on \(Z\) should give us an unbiased estimator of the causal effect of \(X\) on \(Y\). So let’s regress audit_quality on big4 and mkt_cap.

fm <- lm(audit_quality ~ big4 + I(mkt_cap/1e9), 
         data = sim_auditor)         
modelsummary(fm,
             estimate = "{estimate}{stars}",
             gof_map = c("nobs", "r.squared"),
             stars = c('*' = .1, '**' = 0.05, '***' = .01))
Table 25.2: Regression of audit quality using simulated data
 (1)
(Intercept) 1.561***
(0.050)
big4TRUE 2.310***
(0.064)
I(mkt_cap/1e+09) 0.057***
(0.001)
Num.Obs. 5000
R2 0.555

So we have a clearly positive coefficient on big4 even “controlling for” mkt_cap, and we might be tempted to conclude that we have identified a causal effect. Of course, we know that this conclusion is invalid. The issue is that we are effectively assuming a linear relation between mkt_cap and audit quality, whereas the true relation is non-linear in mkt_cap. As a result, “controlling for” mkt_cap by including mkt_cap does not adequately control for the relation between mkt_cap and audit quality. Because there is a non-linear relation between big4 and mkt_cap, the regression specification can use information in big4 to explain variation in audit_quality in addition to that variation explained by a linear function of mkt_cap. This additional explanatory power shows up as a statistically significant coefficient on big4 in the regression above.

Would some kind of matching analysis help?

Given its prominence in accounting research, we will focus on propensity-score matching here. The first step is the estimation of propensity scores. The most common approach is to use logistic regression, which we can do using the glm function (which fits “generalized linear models”) with family = binomial(link = "logit")).

sim_fm <- glm(!big4 ~ mkt_cap, data = sim_auditor, 
              family = binomial(link = "logit"))

We can estimate propensity scores as the fitted values from the estimated logit model (we specify type = "response" to get fitted values on the original \([0, 1]\) scale of the dependent variable).

sim_match_pscores <-
  sim_auditor |>
  mutate(pscore = predict(sim_fm, type = "response"))

But even with the propensity score, we need an algorithm to match treatment observations to their nearest counterparts among the control observations, where “nearest” means closest in terms of propensity scores. To this end, we can use the matchit() function from the MatchIt package. This function returns a matchit object and we can apply the match.data() function to this to retrieve the resulting matches and store these in sim_matches. Following Lawrence et al. (2011), we use a caliper of 3%.

sim_matches <- 
  matchit(!big4 ~ mkt_cap, data = sim_match_pscores) |> # caliper = 0.03) |>
  match.data()

Inspecting sim_matches, we see that it contains the variable distance that exactly matches pscore that we estimated above.

sim_matches |> count(pscore == distance)
# A tibble: 1 × 2
  `pscore == distance`     n
  <lgl>                <int>
1 TRUE                  3728

In addition to the original contents of sim_auditor for the matched observations, sim_matches contains data on the matched pairs in the column subclass. Picking observations for three arbitrary values of subclass, we can see that each value is associated with a pair of observations, one with big4 == FALSE and one with big4 == TRUE and very similar values of mkt_cap and hence very similar pscore values.

sim_matches |> 
  filter(subclass %in% c(220, 481, 333)) |> 
  select(id, big4, mkt_cap, pscore, subclass) |>
  arrange(subclass, big4)
# A tibble: 6 × 5
     id big4     mkt_cap pscore subclass
  <int> <lgl>      <dbl>  <dbl> <fct>   
1  1523 FALSE  238947920  0.509 220     
2  4404 TRUE  1220480140  0.420 220     
3  1783 FALSE  303631240  0.504 333     
4  4510 TRUE  1386042900  0.406 333     
5  2169 FALSE   17701600  0.530 481     
6  2740 TRUE   157943150  0.517 481     

To examine the success of the match, we can compare two pairs of histograms. The first pair Figure 25.3 shows the propensity scores for all observations. Here we see that there are many treatment observations with p-scores above say 0.70, but very few control observations with comparable p-scores to match with these. In this relatively simple setting, this is really just another way of saying there are many large Big Four clients and few similarly sized non–Big Four clients to match with them. This suggests that many observations will be unmatched.

The first pair of histograms Figure 25.4 shows the propensity scores for the matched observations. We can see that the distributions of p-scores for treatment and control observations look very similar, suggesting a fairly good match.

sim_match_pscores |> 
  ggplot(aes(x = pscore, fill = big4)) + 
  geom_histogram(binwidth = 0.025) + 
  facet_grid(big4 ~ .)
Figure 25.3: Propensity scores distribution: All observations
sim_matches |> 
  ggplot(aes(x = pscore, fill = big4)) + 
  geom_histogram(binwidth = 0.025) + 
  facet_grid(big4 ~ .)
Figure 25.4: Propensity scores distribution: Matched observations

If we are happy with the match, we can finish our analysis by estimating regressions. Like Lawrence et al. (2011), we estimate regressions with and without the variables from the p-score regression (Shipman et al., 2016 refer to these approaches as “MR” and “t-test,” respectively).

fms <- list()
fms[[1]] <-  
  feols(audit_quality ~ big4, 
       data = sim_matches)

fms[[2]] <-  
  feols(audit_quality ~ big4 + I(mkt_cap/1e9), 
       data = sim_matches)
modelsummary(fms,
             estimate = "{estimate}{stars}",
             gof_map = c("nobs", "r.squared"),
             stars = c('*' = .1, '**' = 0.05, '***' = .01))
 (1)   (2)
(Intercept) 1.601*** 1.505***
(0.037) (0.033)
big4TRUE 0.971*** 0.908***
(0.053) (0.047)
I(mkt_cap/1e+09) 0.135***
(0.004)
Num.Obs. 3728 3728
R2 0.083 0.275

Interestingly, we see that big4 still has a statistically significant relation with audit quality. This is likely due to a residual effect of mkt_cap being picked up by big4 due to the imperfection of the propensity-score match.

25.2.1 Exercises

  1. What happens in the above analyses if the true effect of big4 on audit quality is 10 instead of 0?
  2. What happens in the above analyses if the caliper is reduced to 0.01 from 0.03 (but the true effect of big4 on audit quality is still 0)?
  3. What happens in the above analyses if the caliper is reduced to 0.01 from 0.03 and the true effect of big4 on audit quality is 10?
  4. Given the above, what are your conclusions about the value of propensity-score matching and OLS in causal inference?
  5. What regression results do you get if you repeat the analysis shown in Table 25.2, but replace I(mkt_cap/1e+09) with I(mkt_cap ^(1/3) * 0.003)? Does this suggest a solution to the issues we see above? If so, what challenges does such a solution face?
  6. Was endogeneity an issue in the simulation above? If not, why not? If so, in what way, and what could you do to address it?

25.3 Replication of Lawrence et al. (2011)

To facilitate discussion, we conduct an approximate replication of Lawrence et al. (2011) and begin by constructing a subset of Compustat with calculations used in that paper.

comp <-
  funda |>
  filter(indfmt == "INDL", datafmt == "STD",
         consol == "C", popsrc == "D") |>
  filter(!is.na(sich), !between(sich, 6000, 6999),
         sale > 0, at > 0, (dltt >= 0 | is.na(dltt)) & (dlc >= 0 | is.na(dlc)),
         prcc_f * csho > 0, ceq > 0) |>
  select(gvkey, datadate, fyear, au, prcc_f, csho, at, ib,
         dltt, dlc, rect, ppent, sale, act, lct, sich,
         oancf) |>
  mutate(sic2 = floor(sich/100)) |>
  filter(between(fyear, 1988, 2006)) |>
  collect() |>
  group_by(gvkey) |>
  arrange(fyear) |>
  mutate(lag_fyear = lag(fyear),
         avg_at = (lag(at) + at)/2,
         log_at = log(at),
         aturn = if_else(lag(at) > 0, sale/lag(at), NA_real_),
         inv_avg_at = 1/avg_at,
         d_sale = sale - lag(sale),
         d_rect = rect - lag(rect),
         d_sale_at = (d_sale - d_rect)/avg_at,
         ppe_at = ppent/avg_at,
         curr = if_else(lct > 0, act/lct, NA_real_),
         lag_curr = lag(curr),
         lev = if_else(avg_at > 0, 
                       (dltt + coalesce(dlc, 0))/avg_at, NA_real_),
         lag_lev = lag(lev),
         roa = if_else(avg_at > 0, ib/avg_at, NA_real_),
         lag_roa = lag(roa),
         accruals = ib - oancf,
         ta = accruals/avg_at,
         mkt_cap = prcc_f * csho,
         log_mkt = if_else(mkt_cap > 0, log(mkt_cap), NA_real_)) |>
  ungroup() |>
  filter(lag_fyear == fyear - 1L)

Lawrence et al. (2011) estimate discretionary accrual models by industry and year, subject to a requirement of at least 10 data points for each model. So we compile a list of industry-years that meet this criterion.

industries <- 
  comp |>
  group_by(fyear, sic2) |>
  summarize(n_firms = n(), .groups = "drop") |>
  filter(n_firms >= 10)

We next construct a function to estimate discretionary accruals along the lines described in footnote 11 of Lawrence et al. (2011).

get_klw_data <- function(sic2, fyear) {

  reg_data <-
    comp |>
    filter(sic2 == !!sic2, fyear == !!fyear) 
           
  fm <- tryCatch(lm(ta ~ inv_avg_at + d_sale_at + ppe_at,
                    data = reg_data, na.action = na.exclude),
                 error = function(e) NULL)
    
  if (is.null(fm)) return(NULL)
  
  results <-
    reg_data |> 
    select(gvkey, fyear, roa) |>
    bind_cols(da = resid(fm)) |>
    filter(!is.na(da))
  
  ada_data <-
    results |>
    inner_join(results, by = "fyear",
               suffix = c("", "_match"),
               relationship = "many-to-many") |>
    filter(gvkey != gvkey_match) |>
    mutate(roa_diff = abs(roa - roa_match)) |>
    group_by(gvkey, fyear) |>
    filter(roa_diff == min(roa_diff, na.rm = TRUE)) |>
    mutate(ada = abs(da - da_match)) |>
    select(gvkey, fyear, da, ada) |>
    ungroup()
  
  ada_data
}

We can then Map this function onto our industry-years and store the results in klw_results.3

klw_results <- bind_rows(Map(get_klw_data, industries$sic2, industries$fyear))
full_sample <-
  comp |>
  inner_join(klw_results, by = c("gvkey", "fyear")) |>
  filter(avg_at > 0) |>
  mutate(big4 = au %in% 1:8L,
         mkt_cap = prcc_f * csho,
         log_mkt = if_else(mkt_cap > 0, log(mkt_cap), NA_real_))

fms <- list()
fms[[1]] <- feols(ada ~ big4 + log_mkt + lag_roa + 
                   lag_lev + lag_curr | sic2 + fyear,
                  data = full_sample)
NOTE: 13,505 observations removed because of NA values (RHS: 13,505).

Lawrence et al. (2011) use the widely used approach of winsorizing variables used in their analysis. We use the winsorize function from the farr package (as discussed in Chapter 24) to do this.

win_vars <- c("ada", "log_mkt", "lag_roa", "lag_lev", "lag_curr",
              "roa", "lev", "curr", "mkt_cap")

full_sample_win <-
  full_sample |>
  mutate(across(all_of(win_vars),
                \(x) winsorize(x, prob = 0.01)))

fms[[2]] <- 
  feols(ada ~ big4 + log_mkt + lag_roa + lag_lev + lag_curr | 
              sic2 + fyear, 
        ~ gvkey + fyear,
        data = full_sample_win)
NOTE: 13,505 observations removed because of NA values (RHS: 13,505).
lmz_match <-
  full_sample |>
  filter(!is.na(log_at) &
           !is.na(log_mkt) &
           !is.na(aturn) &
           !is.na(curr) &
           !is.na(lev) &
           !is.na(roa)) |>
  matchit(!big4 ~ log_at + log_mkt + aturn +
                    curr + lev + roa, 
                  data = _,
                  caliper = 0.0300, std.caliper = FALSE,
                  m.order = "largest", discard = "both")
full_sample_win_no_miss <-
  full_sample_win |>
  filter(!is.na(log_at) &
           !is.na(log_mkt) &
           !is.na(aturn) &
           !is.na(curr) &
           !is.na(lev) &
           !is.na(roa))

lmz_match_win <- matchit(!big4 ~ log_at + log_mkt + aturn +
                    curr + lev + roa, 
                  data = full_sample_win_no_miss,
                  caliper = 0.0300, std.caliper = FALSE,
                  m.order = "largest", discard = "both")

We can estimate propensity scores as the fitted values from the estimated logit model (we specify type = "response" to get fitted values on the original \([0, 1]\) scale of the dependent variable).

Following Lawrence et al. (2011), we discard matches where the difference in p-scores is greater than 3%.

lmz_matched <- match.data(lmz_match)

fms[[3]] <-  
  feols(ada ~ big4 + log_mkt + lag_roa + lag_lev + lag_curr | 
              sic2 + fyear, ~ gvkey + fyear, 
       data = lmz_matched)
NOTE: 4,702 observations removed because of NA values (RHS: 4,702).
lmz_matched_win <- match.data(lmz_match_win)

fms[[4]] <-  
  feols(ada ~ big4 + log_mkt + lag_roa + lag_lev + lag_curr | 
              sic2 + fyear, 
        ~ gvkey + fyear,
        data = lmz_matched_win)
NOTE: 4,665 observations removed because of NA values (RHS: 4,665).
modelsummary(fms,
             estimate = "{estimate}{stars}",
             gof_map = c("nobs", "r.squared", "adj.r.squared"),
             stars = c('*' = .1, '**' = 0.05, '***' = .01))
 (1)   (2)   (3)   (4)
big4TRUE −0.016*** −0.014*** −0.007*** −0.006***
(0.002) (0.001) (0.002) (0.002)
log_mkt −0.006*** −0.005*** −0.007*** −0.005***
(0.001) (0.000) (0.001) (0.001)
lag_roa −0.108*** −0.124*** −0.107*** −0.136***
(0.006) (0.005) (0.017) (0.005)
lag_lev −0.045*** −0.045*** −0.054*** −0.050***
(0.003) (0.002) (0.005) (0.005)
lag_curr 0.000*** −0.001*** 0.000 −0.001**
(0.000) (0.000) (0.000) (0.000)
Num.Obs. 73011 73011 24210 24237
R2 0.127 0.150 0.116 0.134
R2 Adj. 0.127 0.149 0.113 0.132

25.4 DeFond et al. (2017)

DeFond et al. (2017) re-examine the question of Lawrence et al. (2011), but with some differences in approach.

The first set of differences relates to design choices in the application of propensity-score matching. While Lawrence et al. (2011) focus on a single propensity-score model with five primary covariates, DeFond et al. (2017) augment the propensity-score model with twenty additional variables, namely the squares and cubes of these covariates and the ten interactions between the five covariates. DeFond et al. (2017) then select 1,500 random subsets from the twenty variables to develop propensity-score models. For each estimated propensity-score model, the caliper is set at a random value (less than 30%) that yields a “balanced matched sample”.

A second design-choice variation that DeFond et al. (2017) consider is matching with replacement. While Lawrence et al. (2011) focus on matching one treatment observation with (at most) one control observation without replacement (i.e., once a control observation is matched to a treatment observation, it is not available to match with another treatment observation), DeFond et al. (2017) consider matches with replacement and consider matches where each treatment observation is matched to one, two, or three control firms.

The second set of differences between DeFond et al. (2017) and Lawrence et al. (2011) relate to the measures of audit quality considered. Lawrence et al. (2011) consider cost of equity capital and analyst forecast accuracy, while DeFond et al. (2017) omit these two measures on the basis that they have “poor construct validity, and as a result are rarely used in the prior literature” (DeFond et al., 2017, p. 3637). Instead, DeFond et al. (2017) consider three other measures of audit quality: income-increasing discretionary accruals, restatements, and going concern opinions.

25.5 Discussion questions

  1. Given the evidence presented in the simulation, how do you interpret the regression results presented above? How do DeFond et al. (2017) appear to interpret the results?

  2. Referring back to the basic causal relations described in Section 4.2, what is the causal diagram implied by equation (1) of and Table 2 of Lawrence et al. (2011)? What variables are in PROXY_CONTROLS for Table 2? Why is LOG_ASSETS not found in Table 2?

  3. Do Lawrence et al. (2011) report results from estimating equation (1)? What happens to the “difference in means” between the Full Sample and the Propensity-Score Matched Sample in Table 1? Does this make sense?

  4. Why do you think the difference in means is still statistically significant for two variables in the Propensity-Score Matched Sample column of Table 1 in Lawrence et al. (2011)? Do you think this is a concern?

  5. Lawrence et al. (2011) evaluate three outcomes as measures of audit quality: discretionary accruals, cost of equity, and analyst forecast accuracy? Evaluate each of these measures. Which do you think is best? What are the strengths and weaknesses of this best measure?

  6. Can you think of other measures of audit quality that might make sense?

  7. Apply the check list of Shipman et al. (2016, pp. 217–218) and the questions in Panels B and C of Table 1 of Shipman et al. (2016) to Lawrence et al. (2011). How does Lawrence et al. (2011) stack up against these?


  1. This appears to be the approach used in DeFond et al. (2017), but whether this lines up with the approach used in Lawrence et al. (2011) is hard to say.↩︎

  2. The higher number can be seen here, but this appears to have been quietly corrected here. The other company is LVMH, the French luxury goods firm, which appears to have been audited in part by Ernst & Young.↩︎

  3. Here KLW refers to Kothari et al. (2005).↩︎