1 Introduction

In this notebook, we analyse the survey result collected from 199 online questionnaires. This code is structured as follows. Section 2 is the preparation. Section 3 shows the process of cleaning survey results and section 4 provides the preliminary analysis of survey results.

2 Preparation

2.1 Upload libraries

Upload required library. You may need to install those packages first if you haven’t never installed them before.

load.lib<-c("dplyr","readxl", "reshape","ggplot2","data.table", "knit", "kableExtra", "stargazer")
# install.lib<-load.lib[!load.lib %in% installed.packages()]
# for(lib in install.lib) install.packages(lib,dependencies=TRUE)
lapply(load.lib,require,character.only = TRUE)

2.2 Input data

Input the survey base table that includes text sources for 1000 sample paragraphs, and the survey results table that includes the survey scores from survey respondents.

#Survey base data
survey_base_1000_scored_flag <- read_excel("./data_input/survey_base_1000_scored_flag.xlsx")
# survey result data
survey_response_score <- read_excel("./data_input/survey_response_score.xlsx")

## check the number of rows in both data set
print(paste("The original survey base dataset include:", nrow(survey_base_1000_scored_flag), "and the rated paragraphs is", nrow(subset(survey_base_1000_scored_flag, scoed_flag_final=="Yes")), sep = " "))
## [1] "The original survey base dataset include: 1000 and the rated paragraphs is 833"
print(paste("The original survey base dataset include:", nrow(survey_response_score), "and the rated unique paragraphs is", length(unique(survey_response_score$Question_Viewed)), sep = " "))
## [1] "The original survey base dataset include: 1965 and the rated unique paragraphs is 833"
#create a unique dataset for future use
unique_para <- unique(survey_response_score$Question_Viewed) %>% as.data.frame()

3 Survey data cleaning

In this section, we:

  1. Standardise surveys scores to have a mean of 0 and sd of 1 for each respondent;
  2. Match the survey result data to the survey base data to create the final data output;
  3. Save the merged dataset as a excel workbook as a record for further analysis.

3.1 Standardise survey scores

To address the response bias as described in the section 3.2.2 in the RDP, we standardise each respondent’s survey results using the scale() function. And a snapshot of the data after standardization is shown as below.

survey_score_scale <- survey_response_score %>% 
  group_by(Respondent_ID) %>%
  mutate(content.scale = scale(as.numeric(Content)), 
         clarity.scale=scale(as.numeric(Clarity)),
         content.mean = mean(as.numeric(Content)), 
         clarity.mean =mean(as.numeric(Clarity)),
         content.sd = sd(Content),
         clarity.sd = sd(Clarity),
         content.demean = Content - content.mean,
         clarity.demean = Clarity - clarity.mean)


survey_score_scale %>% head() %>% 
  kbl() %>%
  kable_paper() %>%
  scroll_box(width = "100%", height = "200px")
rank index survey_link Respondent_ID Collector_ID economic_literacy economics_education education_index economic_job completion_flag Question_Viewed Clarity Content Q_Index duplicate_flag Score_Times policy_score_job policy_score_2_dimension para_part_extraction original_job_type record_score_by para_scored_by content.scale clarity.scale content.mean clarity.mean content.sd clarity.sd content.demean clarity.demean
1 1 s1 11312343615 252969670 3 None 0 Yes 1 This will tend to push down on real interest rates and push up on property prices. At the same time, younger households will be able to service a higher level of debt to fund consumption and higher housing costs. 2 4 Q1 Duplicate 7 P NP on property prices. At the same time, younger hou NA Non-economist NP&P 0.8549108 -1.8973666 2.9 4.0 1.2866839 1.0540926 1.1 -2.0
2 2 s1 11312322112 252969670 3 High school economics course 0 No 1 Some German bank-watchers would probably have preferred that Jens Weidmann, the hawkish boss of the Bundesbank, was picked to succeed Mr Draghi. A few worry that, as a former French finance minister, Ms Lagarde might further politicise matters by pursuing policies that are redistributive across borders, rather than sticking to the European Central Bank’s (ECB’s) price-stability mandate. But some hawks hold out hope that Ms Lagarde might change the bank’s policy. Some see her promise to review the bank’s strategy as an opening to rethink its commitment to stimulus. And as she is not a trained economist, they reckon she might prove more pliable than Mr Draghi. 3 1 Q2 Duplicate 2 NP NP y as an opening to rethink its commitment to stimu NA Non-economist NP -1.0569964 0.3162278 2.2 2.6 1.1352924 1.2649111 -1.2 0.4
3 3 s1 11312320324 252969670 3 None 0 Yes 1 The increase in dwelling investment to date has already seen a pick-up in growth of greenfield land sales, particularly along the eastern seaboard. However, developers are now holding unusually low levels of unsold lots and, in some cities, supply is equivalent to less than three months of sales. Unsold lots are recorded as being most scarce in Sydney and south-east Queensland, where the Bank’s liaison contacts have suggested that developers are having difficulty obtaining further suitable land approved for development. Some liaison contacts have also raised concerns about the availability of land for apartment developments in Sydney, as the stock of suitable sites has been gradually depleted over recent years. 4 1 Q3 unique 1 P NP rns about the availability of land for apartment d NA Non-economist P -1.4491377 0.8502651 3.1 3.3 1.4491377 0.8232726 -2.1 0.7
4 4 s1 11312320104 252969670 3 High school economics course 0 No 1 This will tend to push down on real interest rates and push up on property prices. At the same time, younger households will be able to service a higher level of debt to fund consumption and higher housing costs. 4 2 Q4 Duplicate 7 NP NP on property prices. At the same time, younger hou NA Non-economist NP&P -0.9298956 1.1061625 3.4 2.9 1.5055453 0.9944289 -1.4 1.1
5 5 s1 11312298486 252969670 3 High school economics course 0 No 1 The Bank of China is also providing an important source of finance for Chinese companies operating in Australia as well as for Australian companies. It is making trade easier between our two countries by providing trade finance. It is also helping on the investment side, with securities and investment banking services. 4 3 Q1 Duplicate 2 NP NP sier between our two countries by providing trade NA Non-economist NP -0.1355262 0.9486833 3.1 3.5 0.7378648 0.5270463 -0.1 0.5
6 8 s1 11311859082 252969670 3 None 0 No 1 And macro-prudential policy cannot solve all problems. It may fall to other authorities and policy instruments to respond to any adverse structural effect from changing business models on the provision of financial services to some sectors of the economy, for example credit to small to medium enterprises. 2 3 Q4 unique 1 NP NP rom changing business models on the provision of f NA Non-economist NP -0.7378648 -1.7919573 3.7 3.7 0.9486833 0.9486833 -0.7 -1.7

3.2 Prepare analysis data

In this section, we map the survey result table to the survey base table and save the output table for later use.

3.2.1 Create unique paragraph index

Some paragraphs were scored more than once in the survey, thus there will be multiple record for a given paragraph in the survey result table. To avoid duplication, we create a variable called para_unique_index to each paragraph. A snapshot of the final table is shown as below:

#create the unique index for the 833 paragraphs
unique_para_index_dt <- survey_score_scale %>% ungroup() %>%
  select(para_part_extraction) %>% 
  distinct() %>%
  mutate(para_unique_index = paste("p_",1:833, sep = ""))

# attach the para_unique_index to the survey base table and the survey response data
survey_table_final_temp <- 
  left_join(survey_base_1000_scored_flag, unique_para_index_dt, by = "para_part_extraction") %>%
  dplyr::arrange(scoed_flag_final)
  

survey_table_final <- 
  left_join(survey_table_final_temp, survey_score_scale, by = "para_part_extraction") %>%
  filter(scoed_flag_final == "Yes")

# take a quick look of the data:
survey_table_final %>% 
  head() %>%  kbl() %>%
  kable_paper() %>%
  scroll_box(width = "100%", height = "200px")
index.x year month issue para source source_group survey_group question_group para_part_extraction scoed_flag_final para_unique_index rank index.y survey_link Respondent_ID Collector_ID economic_literacy economics_education education_index economic_job completion_flag Question_Viewed Clarity Content Q_Index duplicate_flag Score_Times policy_score_job policy_score_2_dimension original_job_type record_score_by para_scored_by content.scale clarity.scale content.mean clarity.mean content.sd clarity.sd content.demean clarity.demean
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 28 32 s1 11302939887 252969670 5 Masters or PhD in economics or a related discipline 1 Yes 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 3 3 Q4 Duplicate 3 P P NA Economist NP&P 0.25354628 -0.08352691 2.8 3.1 0.7888106 1.1972190 0.2 -0.1
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 29 35 s1 11295748232 252969670 2 None 0 No 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 4 4 Q3 Duplicate 3 NP NP NA Non-economist NP&P 1.03711608 1.03711608 3.3 3.3 0.6749486 0.6749486 0.7 0.7
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 37 46 s1 11295512712 252969670 2 High school economics course 0 No 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 2 3 Q2 Duplicate 3 NP NP NA Non-economist NP&P -0.08352691 -1.18175790 3.1 3.5 1.1972190 1.2692955 -0.1 -1.5
19 2019 2019_Commonwealth_Orange Grattan The NSRB’s first review proposed a new and better parental ‘capacity-to-contribute’ model for non-government schools, based on family income rather than where parents live. But in accepting the NSRB model, the Coalition also introduced a $1.2 billion ‘Choice and Affordability Fund’. This was a backward step, subsidising low-fee private schools even when parents can afford to pay their way. 11_grattan G8 1 1 billion ‘Choice and Affordability Fund’. This was Yes p_15 27 31 s1 11302945320 252969670 4 Masters or PhD in economics or a related discipline 1 Yes 1 The National Schools Resourcing Board (NSRB)‘s first review proposed a new and better parental ’capacity-to-contribute’ model for non-government schools, based on family income rather than where parents live. But in accepting the NSRB model, the Coalition also introduced a $1.2 billion ‘Choice and Affordability Fund’. This was a backward step, subsidising low-fee private schools even when parents can afford to pay their way. 2 2 Q3 unique 1 P P NA Economist P -1.65615734 -0.97618706 3.6 3.2 0.9660918 1.2292726 -1.6 -1.2
34 2017 9 2017 September The Growing Demand for Cash The velocity of cash is not constant. It evolves with changes in technology, changes to the operation of the banknote distribution system, the stage of the business cycle, and consumer and business preferences. The rising stock of banknotes in circulation alongside declining cash payments suggests that the velocity of cash has fallen over the past 10 years. That is, each banknote in circulation is being used in fewer transactions now than in the past. Several factors have contributed to this, related to both the stock of banknotes held for transactional purposes and the stock held for store-of-value purposes. 2_bulletin G2 1 1 ctors have contributed to this, related to both th Yes p_18 38 47 s1 11295478439 252969670 4 Bachelor’s degree in economics or a related discipline 1 No 1 The velocity of cash is not constant. It evolves with changes in technology, changes to the operation of the banknote distribution system, the stage of the business cycle, and consumer and business preferences. The rising stock of banknotes in circulation alongside declining cash payments suggests that the velocity of cash has fallen over the past 10 years. That is, each banknote in circulation is being used in fewer transactions now than in the past. Several factors have contributed to this, related to both the stock of banknotes held for transactional purposes and the stock held for store-of-value purposes. 4 3 Q3 unique 1 NP NP NA Non-economist NP -0.44447832 0.17616607 3.3 3.8 0.6749486 1.1352924 -0.3 0.2
50 2018 Guy Debelle[*]Deputy Governor Remarks at FINSIA Signature Event: The Regulators Melbourne – 15 November 2018 One lesson from any number of crises past is that it is almost inevitable the next crises will originate somewhere different. The inevitability of this in part arises because of the reforms introduced to address the sources of the previous crisis. So where might one possible location of the next crisis lie? 3_rba_speeches G3 1 1 t arises because of the reforms introduced to addr Yes p_17 35 43 s1 11295529047 252969670 5 Masters or PhD in economics or a related discipline 1 Yes 1 One lesson from any number of crises past is that it is almost inevitable the next crises will originate somewhere different. The inevitability of this in part arises because of the reforms introduced to address the sources of the previous crisis. So where might one possible location of the next crisis lie? 3 5 Q3 unique 1 P P NA Economist P 0.90504206 -1.52127766 4.1 4.2 0.9944289 0.7888106 0.9 -1.2

3.2.2 Create datasets based on scores times

As mentioned previously, there are some paragraphs that were scored more than once in the online survey. Based on this information, we create the following five datasets that will be used later for analysis.

  • unique_scored: response data for those paragraphs that were only scored once
  • multiple_scored: response data for those paragraphs that were scored twice or more
  • multiple_scored_para_eco: multiple-scored paragraphs/responses by economist only
  • multiple_scored_para_noneco: multiple-scored paragraphs/responses by non-economist only
  • multiple_scored_para: multiple-scored paragraphs/responses by both economist and non-economist only

A variable ‘unique_para_scored’ is created to show if a given paragraph is scored by economist only (economist), non-economist only (non-economist) or both economist and non-economists (both). A snapshot of the data is shown as below.

unique_scored <- survey_table_final %>% filter(duplicate_flag=="unique")
multiple_scored <- survey_table_final %>% filter(duplicate_flag=="Duplicate")

unique_scored_para <- unique_scored %>% 
  mutate(unique_para_scored = record_score_by) %>% 
  select(para_unique_index, unique_para_scored) #extract paragraph index number that scored by one respondent only


multiple_scored_para_eco <- multiple_scored %>% 
  select(para_unique_index, record_score_by) %>% 
  filter(record_score_by=="Economist")

multiple_scored_para_noneco <- multiple_scored %>% 
  select(para_unique_index, record_score_by) %>% 
  filter(record_score_by=="Non-economist")

#update the record_score_by as "both" if that paragraph is scored by both economist and non-economist
multiple_scored$unique_para_scored <- ifelse(
  multiple_scored$para_unique_index %in% multiple_scored_para_eco$para_unique_index &
    multiple_scored$para_unique_index %in% multiple_scored_para_noneco$para_unique_index, "both",
  multiple_scored$record_score_by
  )

multiple_scored_para <- multiple_scored %>% select(para_unique_index, unique_para_scored) %>% unique() # 622 rows. extract paragraphs that are scored by multiple respondents

#create the final output table with a column 'unique_para_scored' to show how this paragraph is scored.
scored_para_area <- rbind(unique_scored_para,multiple_scored_para)

#update the previous table
survey_table_final <- left_join(survey_table_final,scored_para_area, by = "para_unique_index") #the unique_para_scored column records the working area of respondents for each paragraph that has been rated multiple times.

survey_table_final %>% head() %>% 
  kbl() %>%
  kable_paper() %>%
  scroll_box(width = "100%", height = "200px")
index.x year month issue para source source_group survey_group question_group para_part_extraction scoed_flag_final para_unique_index rank index.y survey_link Respondent_ID Collector_ID economic_literacy economics_education education_index economic_job completion_flag Question_Viewed Clarity Content Q_Index duplicate_flag Score_Times policy_score_job policy_score_2_dimension original_job_type record_score_by para_scored_by content.scale clarity.scale content.mean clarity.mean content.sd clarity.sd content.demean clarity.demean unique_para_scored
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 28 32 s1 11302939887 252969670 5 Masters or PhD in economics or a related discipline 1 Yes 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 3 3 Q4 Duplicate 3 P P NA Economist NP&P 0.25354628 -0.08352691 2.8 3.1 0.7888106 1.1972190 0.2 -0.1 both
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 29 35 s1 11295748232 252969670 2 None 0 No 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 4 4 Q3 Duplicate 3 NP NP NA Non-economist NP&P 1.03711608 1.03711608 3.3 3.3 0.6749486 0.6749486 0.7 0.7 both
4 2016 October Household and Business Finances | Financial Stability Review – October 2016 | RBA In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 1_frs G1 1 1 al for rents to fall and vacancy rates to rise als Yes p_16 37 46 s1 11295512712 252969670 2 High school economics course 0 No 1 In these circumstances, developers may have trouble finding buyers for their new apartments in some areas. While liaison with industry suggests that settlement failure rates remain low, developers are continuing to report delays in settlement for some purchasers. One reported contributor to settlement delays is tighter access to finance, particularly for buyers relying on foreign income. Liaison also indicates that valuations at settlement are sometimes coming in below what buyers had anticipated and, in some cases, below contracted purchase prices, reducing the amount banks will lend. For investors buying these new apartments, declines in apartment prices raise the likelihood that they fall into negative equity at settlement. The potential for rents to fall and vacancy rates to rise also raises the risk that investors may find it more difficult to subsequently service their mortgages. 2 3 Q2 Duplicate 3 NP NP NA Non-economist NP&P -0.08352691 -1.18175790 3.1 3.5 1.1972190 1.2692955 -0.1 -1.5 both
19 2019 2019_Commonwealth_Orange Grattan The NSRB’s first review proposed a new and better parental ‘capacity-to-contribute’ model for non-government schools, based on family income rather than where parents live. But in accepting the NSRB model, the Coalition also introduced a $1.2 billion ‘Choice and Affordability Fund’. This was a backward step, subsidising low-fee private schools even when parents can afford to pay their way. 11_grattan G8 1 1 billion ‘Choice and Affordability Fund’. This was Yes p_15 27 31 s1 11302945320 252969670 4 Masters or PhD in economics or a related discipline 1 Yes 1 The National Schools Resourcing Board (NSRB)‘s first review proposed a new and better parental ’capacity-to-contribute’ model for non-government schools, based on family income rather than where parents live. But in accepting the NSRB model, the Coalition also introduced a $1.2 billion ‘Choice and Affordability Fund’. This was a backward step, subsidising low-fee private schools even when parents can afford to pay their way. 2 2 Q3 unique 1 P P NA Economist P -1.65615734 -0.97618706 3.6 3.2 0.9660918 1.2292726 -1.6 -1.2 Economist
34 2017 9 2017 September The Growing Demand for Cash The velocity of cash is not constant. It evolves with changes in technology, changes to the operation of the banknote distribution system, the stage of the business cycle, and consumer and business preferences. The rising stock of banknotes in circulation alongside declining cash payments suggests that the velocity of cash has fallen over the past 10 years. That is, each banknote in circulation is being used in fewer transactions now than in the past. Several factors have contributed to this, related to both the stock of banknotes held for transactional purposes and the stock held for store-of-value purposes. 2_bulletin G2 1 1 ctors have contributed to this, related to both th Yes p_18 38 47 s1 11295478439 252969670 4 Bachelor’s degree in economics or a related discipline 1 No 1 The velocity of cash is not constant. It evolves with changes in technology, changes to the operation of the banknote distribution system, the stage of the business cycle, and consumer and business preferences. The rising stock of banknotes in circulation alongside declining cash payments suggests that the velocity of cash has fallen over the past 10 years. That is, each banknote in circulation is being used in fewer transactions now than in the past. Several factors have contributed to this, related to both the stock of banknotes held for transactional purposes and the stock held for store-of-value purposes. 4 3 Q3 unique 1 NP NP NA Non-economist NP -0.44447832 0.17616607 3.3 3.8 0.6749486 1.1352924 -0.3 0.2 Non-economist
50 2018 Guy Debelle[*]Deputy Governor Remarks at FINSIA Signature Event: The Regulators Melbourne – 15 November 2018 One lesson from any number of crises past is that it is almost inevitable the next crises will originate somewhere different. The inevitability of this in part arises because of the reforms introduced to address the sources of the previous crisis. So where might one possible location of the next crisis lie? 3_rba_speeches G3 1 1 t arises because of the reforms introduced to addr Yes p_17 35 43 s1 11295529047 252969670 5 Masters or PhD in economics or a related discipline 1 Yes 1 One lesson from any number of crises past is that it is almost inevitable the next crises will originate somewhere different. The inevitability of this in part arises because of the reforms introduced to address the sources of the previous crisis. So where might one possible location of the next crisis lie? 3 5 Q3 unique 1 P P NA Economist P 0.90504206 -1.52127766 4.1 4.2 0.9944289 0.7888106 0.9 -1.2 Economist

4 Survey Respondent Analysis

Then, we analyse respondents composition using the three background questions from the survey.

4.1 Respondents by education background

The composition of respondents by different economic education background is shown as:

survey_result <- survey_table_final %>% as.data.frame()


respondents_data <- survey_result %>% select(Respondent_ID, economic_literacy, economics_education,
                                          education_index, economic_job, completion_flag) %>%
  dplyr::mutate(economist_flag = ifelse(
    economics_education %in% c("Bachelor's degree in economics or a related discipline",
                             "Masters or PhD in economics or a related discipline") & economic_job == "Yes",
    "Economist","Non-economist")) %>%
  distinct()

##The count of respondents by education background:
respondents_data %>% group_by(economics_education) %>%
  dplyr::summarise(count=n()) %>% 
  dplyr::mutate(share = round(count/nrow(respondents_data),4)) %>%  
  kbl(caption = "Survey respondent by economics education background") %>%
  kable_classic(full_width = T, html_font = "Cambria")
Survey respondent by economics education background
economics_education count share
Bachelor’s degree in economics or a related discipline 47 0.2362
High school economics course 52 0.2613
Masters or PhD in economics or a related discipline 42 0.2111
None 58 0.2915

4.2 Respondents by education literacy

The composition of survey respondents by their self-rated education literacy is shown as:

respondents_data %>% group_by(economic_literacy) %>%
  dplyr::summarise(count=n()) %>% 
  dplyr::mutate(share = round(count/nrow(respondents_data),4)) %>%  
  kbl(caption = "Survey respondent by self-assessed economic literacy") %>%
  kable_classic(full_width = T, html_font = "Cambria")
Survey respondent by self-assessed economic literacy
economic_literacy count share
1 23 0.1156
2 32 0.1608
3 62 0.3116
4 37 0.1859
5 45 0.2261

4.3 Respondents by working area

The composition of respondents by working area. Economic related jobs are defined as policy-related working areas (Yes).

respondents_data %>% group_by(economic_job) %>%
  dplyr::summarise(count=n()) %>% 
  dplyr::mutate(share = round(count/nrow(respondents_data),4)) %>% kbl() %>%
  kable_classic(full_width = T, html_font = "Cambria")
economic_job count share
No 111 0.5578
Yes 88 0.4422

4.4 Count of economist and non-economist

The economist is defined as those who hold an economic bachelor degree and above and work in policy-related area. The count of economists and non-economists is shown in the table below:

# The number of respondents defined as economists or non-economists
respondents_data %>% group_by (economist_flag) %>% dplyr::summarise(count = n()) %>%  
  kbl(caption = "Count of survey respondents defined as economist") %>%
  kable_classic(full_width = T, html_font = "Cambria")
Count of survey respondents defined as economist
economist_flag count
Economist 71
Non-economist 128

5 Survey results analysis

In this section, we conduct a number of preliminary analysis using survey results. We first check the distribution of survey results, and then we checked if there is an obvious pattern in the survey scores for a given source of sample text. After that, we analyse correlations between:

  • clarity and content
  • our survey scores and the FK readability
  • scores given by economists and non-economists

5.1 Distribution of survey scores

The distribution of standardised survey scores is close to a normal distribution as shown in the graph below. We use mean of multiple survey scores for those that were rated by more than one respondent.

para_score_data <- survey_result %>% 
  select(Respondent_ID, economic_literacy, economics_education, education_index, economic_job,
         completion_flag,source, source_group, survey_group, scoed_flag_final, para_unique_index,
         survey_link, Clarity, Content, clarity.scale, content.scale, duplicate_flag, policy_score_2_dimension,
         original_job_type, record_score_by, unique_para_scored) %>% distinct()

# create the dataset for generating graph
para_score_dt <- para_score_data %>% 
  group_by(para_unique_index) %>% 
  dplyr::mutate(avg.content.scale = mean(content.scale), 
         avg.clarity.scale = mean(clarity.scale)) %>% 
  select(para_unique_index, avg.content.scale, avg.clarity.scale) %>% distinct()


para_score_dt_content <-para_score_dt %>% 
  select(para_unique_index, Normalised_score=avg.content.scale)  %>% mutate(Variable = "Content")

para_score_dt_clarity <-para_score_dt %>% 
  select(para_unique_index, Normalised_score=avg.clarity.scale)  %>% mutate(Variable = "Clarity")

para_score_dt2 <- rbind(para_score_dt_content,para_score_dt_clarity) %>% filter(!is.na(Normalised_score))

# generate graph
g1 <- ggplot(data = para_score_dt2, aes(x=Normalised_score)) + 
  geom_density(aes(fill=Variable), alpha = 0.5) +
  theme(axis.text=element_text(size=16),
        axis.title=element_text(size=16),
        panel.background = element_blank(),
        legend.text = element_text(colour="black", size=16),
        legend.title = element_text(colour="black", size=16))

g1 + scale_fill_brewer(palette = "Set1")

#generate data in csv file for Figure 2 in the RDP

data_graph2_reasoning <- para_score_dt2 %>% ungroup() %>%
  filter(Variable=="Content") %>% select(Normalised_score) %>% 
  dplyr::rename(., reasoning = Normalised_score) %>% arrange(reasoning) %>% mutate(round_value = round(reasoning, 1)) 

data_graph2_readability <- para_score_dt2 %>% ungroup() %>%
  filter(Variable=="Clarity") %>% select(Normalised_score) %>% 
  dplyr::rename(., readability = Normalised_score) %>% arrange(readability) %>% mutate(round_value = round(readability, 1)) 

reasoning_density <- density(data_graph2_reasoning$reasoning)
readability_density <- density(data_graph2_readability$readability)

density_data_for_g2 <- cbind.data.frame(reasoning = reasoning_density$y, readability = readability_density$y)

# write.csv(density_data_for_g2, "density_data_for_graph2.csv")

5.2 Score variation acorss multiple-rated paragraphs

For those paragraphs that were rated by more than one people, we want to know how different the scores are from each other. If there is ‘true’ score for this paragraph and no bias in the comprehension among respondents, those scores should be close to each other. To test for this hypothesis, we caclulate the distance between of each score to the average of those multiple scores and draw a density chart that is shown as Figure 4 in the paper.

score_variation_demean <- read.csv("./data_input/survey_score_demean.csv")

##Transpose data
score1 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score1) %>% filter(!is.na(score))
score2 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score2) %>% filter(!is.na(score))
score3 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score3) %>% filter(!is.na(score))
score4 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score4) %>% filter(!is.na(score))
score5 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score5) %>% filter(!is.na(score))
score6 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score6) %>% filter(!is.na(score))
score7 <- score_variation_demean %>% dplyr::select(Para, respondent=response, model_type, score=score7) %>% filter(!is.na(score))

## combine the data together
score_variation_demean_dt <- rbind(score1, score2, score3, score4, score5, score6, score7)

score_variation_demean_dt$model_type2 <- ifelse(score_variation_demean_dt$model_type=="content","Reasoning","Readability")
# head(score_variation_demean_dt)

score_variation_demean_dt$score <- as.numeric(as.character(score_variation_demean_dt$score))

p1 <- ggplot(data = score_variation_demean_dt, aes(x=score)) + 
  geom_density(aes(fill=respondent, y=..density..), alpha = 0.5) +
  scale_y_continuous(name = "%", breaks =  c(0.0,0.2,0.4,0.6),
                     labels = c("0","2","4","6")) +
  scale_x_continuous(name = "Survey scores around the average")

p1 + facet_wrap(~ model_type2) + 
  theme(strip.background = element_blank(), strip.text = element_text(size = 18)) +
  theme(axis.text=element_text(size=18),
        axis.title=element_text(size=18),
        panel.background = element_blank(),
        legend.text = element_text(colour="black", size=18),
        legend.title = element_text(colour="black", size=18),
        strip.text = element_text(size=18))

5.3 Survey scores by text sources

In this part, we check if there is a pattern in the survey score for paragraphs from a given text source. The output of the Figure 5 in the paper.

text_source_dt <- survey_result%>% 
  select(para_unique_index, Respondent_ID, source, content.scale, clarity.scale, Respondent_ID,
         record_score_by,economics_education, education_index,economic_job, economic_literacy, duplicate_flag,Score_Times)

text_source_dt <- text_source_dt %>% mutate(source_short = case_when
                                            ( source=="1_frs" ~ "FRS",
                                              source=="11_grattan" ~ "Grattan",
                                              source=="2_bulletin" ~ "RBA Bulletin",
                                              source=="3_rba_speeches" ~ "RBA Speech",
                                              source=="4_smp_intro_2006_2019" ~ "RBA SMP Intro",
                                              source=="5_smp_main" ~ "RBA SMP Main",
                                              source=="6_smp_boxes_06_19" ~ "RBA SMP Boxes",
                                              source=="7_boe_main" ~ "BoE IR Main",
                                              source=="8_boe_ir_intro" ~ "BoE IR Intro",
                                              source=="9_boe_speeches" ~ "BoE Speech",
                                              source=="10_economist" ~ "The Economist",
                                            )
                                            )

graph_data_1 <- text_source_dt %>% select(para_unique_index, source, content.scale, clarity.scale) %>%
  group_by(para_unique_index) %>% 
  dplyr::mutate(content.scale_avg = mean(content.scale),clarity.scale_avg = mean(clarity.scale)) %>% 
  select(-content.scale, -clarity.scale) %>%
  distinct() %>% 
  dplyr::mutate(source_2 = case_when
         (source %in% c("4_smp_intro_2006_2019","5_smp_main","6_smp_boxes_06_19") ~ "RBA",
           source %in% c("8_boe_ir_intro","7_boe_main" ) ~ "BoE",
           source == "1_frs"  ~ "RBA",
           source == "11_grattan"  ~ "Grattan",
           source == "2_bulletin" ~ "RBA",
           source == "3_rba_speeches" ~ "RBA",
           source == "9_boe_speeches" ~ "BoE",
           source == "10_economist" ~ "The Economist"
         ),
         source_3 = case_when
         (source %in% c("4_smp_intro_2006_2019","5_smp_main","6_smp_boxes_06_19") ~ "SMP",
           source %in% c("8_boe_ir_intro","7_boe_main" ) ~ "BoE_IR",
           source == "1_frs"  ~ "FSR",
           source == "11_grattan"  ~ "Grattan",
           source == "2_bulletin" ~ "Bulletin",
           source == "3_rba_speeches" ~ "Speech",
           source == "9_boe_speeches" ~ "Speech",
           source == "10_economist" ~ "The Economist"
         )
  )

5.4 Correlation between clarity and content

Here we take a look to see if there is a correlation between the content scores and clarity scores.

graph_data_1 <- text_source_dt %>% select(para_unique_index, source, content.scale, clarity.scale) %>%
  group_by(para_unique_index) %>% 
  dplyr::mutate(content.scale_avg = mean(content.scale),clarity.scale_avg = mean(clarity.scale)) %>% 
  select(-content.scale, -clarity.scale) %>%
  distinct() %>% 
  dplyr::mutate(source_2 = case_when
         (source %in% c("4_smp_intro_2006_2019","5_smp_main","6_smp_boxes_06_19") ~ "RBA",
           source %in% c("8_boe_ir_intro","7_boe_main" ) ~ "BoE",
           source == "1_frs"  ~ "RBA",
           source == "11_grattan"  ~ "Grattan",
           source == "2_bulletin" ~ "RBA",
           source == "3_rba_speeches" ~ "RBA",
           source == "9_boe_speeches" ~ "BoE",
           source == "10_economist" ~ "The Economist"
         ),
         source_3 = case_when
         (source %in% c("4_smp_intro_2006_2019","5_smp_main","6_smp_boxes_06_19") ~ "SMP",
           source %in% c("8_boe_ir_intro","7_boe_main" ) ~ "BoE_IR",
           source == "1_frs"  ~ "FSR",
           source == "11_grattan"  ~ "Grattan",
           source == "2_bulletin" ~ "Bulletin",
           source == "3_rba_speeches" ~ "Speech",
           source == "9_boe_speeches" ~ "Speech",
           source == "10_economist" ~ "The Economist"
         )
  )

graph_data_1 <- na.omit(graph_data_1)

my.formula <- clarity.scale_avg ~ content.scale_avg

ggplot(graph_data_1, aes(content.scale_avg, clarity.scale_avg)) + 
  geom_point() +
  geom_smooth(method = "lm", se = FALSE, formula = my.formula)  +
  # stat_poly_eq(formula = my.formula,
  #              aes(label = paste(..eq.label.., rr.label.., seq = "~~~")),
  #              parse = TRUE) +
  facet_wrap(~source_2)

5.5 Correlation with FK

The Flesch–Kincaid readability formula is a widely used metric for measuring text quality, especially readability. We would like to check if our survey measures of clarity and content are correlated with the FK. To test for it, we run some simple linear regressions and the results are shown as below, which are also discussed in the Section 4.3 of the paper.

(1) Content and FK

There is no significant correlation between the content survey scores and the FK grade levels as shown in the graph and summary of linear regression results below.

model_para_base1 <- readRDS("./data_input/model_para_base.rds")

fk_data <- model_para_base1 %>% select(question_index,source.x, unique_para_scored,
                                       fk_grade_level, content.scale.avg, clarity.scale.avg)
# write.csv(fk_data, "G:/Research/JoanH/20181022_SMP Project/RDP/graphit/data/fk_para_data.csv")

ggplot(fk_data, aes(fk_grade_level, content.scale.avg)) + 
  geom_point() +
  geom_smooth(method = "lm")

fk_m1 <- lm(fk_data$content.scale.avg ~ fk_data$fk_grade_level)
summary(fk_m1)
## 
## Call:
## lm(formula = fk_data$content.scale.avg ~ fk_data$fk_grade_level)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.59947 -0.51944  0.00317  0.52885  2.41101 
## 
## Coefficients:
##                         Estimate Std. Error t value Pr(>|t|)
## (Intercept)             0.092939   0.109066   0.852    0.394
## fk_data$fk_grade_level -0.007290   0.007793  -0.936    0.350
## 
## Residual standard error: 0.7472 on 823 degrees of freedom
## Multiple R-squared:  0.001062,   Adjusted R-squared:  -0.0001515 
## F-statistic: 0.8752 on 1 and 823 DF,  p-value: 0.3498

(2) Clarity and FK

The survey results of clarity is significantly related to the FK grade level, but the magnitude is small.

fk_m2 <- lm(fk_data$clarity.scale.avg ~ fk_data$fk_grade_level)

ggplot(fk_data, aes(fk_grade_level, clarity.scale.avg)) + 
  geom_point() +
  geom_smooth(method = "lm")

summary(fk_m2)
## 
## Call:
## lm(formula = fk_data$clarity.scale.avg ~ fk_data$fk_grade_level)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.36901 -0.50622  0.05408  0.57973  2.08492 
## 
## Coefficients:
##                         Estimate Std. Error t value Pr(>|t|)   
## (Intercept)             0.278908   0.109196   2.554  0.01082 * 
## fk_data$fk_grade_level -0.021227   0.007802  -2.721  0.00665 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7481 on 823 degrees of freedom
## Multiple R-squared:  0.008914,   Adjusted R-squared:  0.00771 
## F-statistic: 7.402 on 1 and 823 DF,  p-value: 0.006652

5.6 Economists v.s non-eocnomists

Using a sample of 465 paragraphs that were rated by both economists and non-economists, we can check the correlation between scores given by economists and non-economists as discussed in the section 4.4 of the paper. The output graph from this section is the Figure 7 in the paper.

(1) content

graph_data_3 <- text_source_dt %>% filter(duplicate_flag == "Duplicate") %>%
  select(para_unique_index, source, content.scale, clarity.scale, record_score_by) %>%
  dplyr::group_by(para_unique_index, record_score_by) %>% 
  dplyr::mutate(content.scale_avg = mean(na.omit(content.scale)),clarity.scale_avg = mean(na.omit(clarity.scale))) %>%
  select(-content.scale, -clarity.scale) %>%
  distinct() 


#extract paragraphs that rated by both economists and non-economists

economist_rated_para <- graph_data_3 %>% filter(record_score_by == "Economist")
non_econ_rated_para <- graph_data_3 %>% filter(record_score_by == "Non-economist")

## find the paragraphs that rated by both

both_rated_para <- 
  inner_join(economist_rated_para, non_econ_rated_para, by = c("para_unique_index","source"),
             suffix = c(".economist", ".non_economists"))

print(paste("The number of multiply rated paragraph is:", nrow(both_rated_para), sep = " "))
## [1] "The number of multiply rated paragraph is: 465"
## A scatter plot to show the correlation between Economsits and Non-economists in terms of their ratings


#Content graph
p_content <- ggplot(data = both_rated_para, 
            aes(x=content.scale_avg.economist, y=content.scale_avg.non_economists)) +
  geom_point() + 
  geom_smooth(method = "lm") +
  labs(title = "Content", 
       x = "Economist", 
       y = "Non-Economist") +
  theme (plot.title = element_text(size=16, hjust = 0.5),
         axis.text=element_text(size=16),
         axis.title=element_text(size=16),
        # panel.background = element_blank(),
        panel.background = element_rect(fill = "white",
                     colour = "grey50",
                     size = 0.5, linetype = "solid"),
        legend.text = element_text(colour="black", size=16),
        legend.title = element_text(colour="black", size=16),
        strip.text = element_text(size=16))

content_m1 <- lm(content.scale_avg.non_economists ~ content.scale_avg.economist, data = both_rated_para)
summary(content_m1)
## 
## Call:
## lm(formula = content.scale_avg.non_economists ~ content.scale_avg.economist, 
##     data = both_rated_para)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.81045 -0.54797  0.06651  0.61127  2.23742 
## 
## Coefficients:
##                             Estimate Std. Error t value Pr(>|t|)  
## (Intercept)                  0.04100    0.03817   1.074   0.2834  
## content.scale_avg.economist  0.10951    0.04356   2.514   0.0123 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.8187 on 458 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.01361,    Adjusted R-squared:  0.01146 
## F-statistic: 6.321 on 1 and 458 DF,  p-value: 0.01227
p_content
## `geom_smooth()` using formula 'y ~ x'

(2) clarity

p_clarity <- ggplot(data = both_rated_para, 
            aes(x=clarity.scale_avg.economist, y=clarity.scale_avg.non_economists)) +
  geom_point() + 
  geom_smooth(method = "lm") +
  labs(title = "Clarity", 
       x = "Economist", 
       y = "Non-Economist") +
  theme (plot.title = element_text(size=16, hjust = 0.5),
         axis.text=element_text(size=16),
         axis.title=element_text(size=16),
        # panel.background = element_blank(),
        panel.background = element_rect(fill = "white",
                     colour = "grey50",
                     size = 0.5, linetype = "solid"),
        legend.text = element_text(colour="black", size=16),
        legend.title = element_text(colour="black", size=16),
        strip.text = element_text(size=16)) 

clarity_m1 <- lm(clarity.scale_avg.non_economists ~ clarity.scale_avg.economist, data = both_rated_para)
summary(clarity_m1)
## 
## Call:
## lm(formula = clarity.scale_avg.non_economists ~ clarity.scale_avg.economist, 
##     data = both_rated_para)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.76378 -0.56046  0.02042  0.67960  2.23819 
## 
## Coefficients:
##                             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                 -0.01832    0.04035  -0.454 0.650061    
## clarity.scale_avg.economist  0.17550    0.04572   3.838 0.000142 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.8596 on 452 degrees of freedom
##   (11 observations deleted due to missingness)
## Multiple R-squared:  0.03156,    Adjusted R-squared:  0.02942 
## F-statistic: 14.73 on 1 and 452 DF,  p-value: 0.0001416
p_clarity
## `geom_smooth()` using formula 'y ~ x'

6 Session information

The session information for this program is:

sessionInfo()
## R version 4.0.3 (2020-10-10)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 17763)
## 
## Matrix products: default
## 
## locale:
## [1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252   
## [3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
## [5] LC_TIME=English_Australia.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] stargazer_5.2.2   kableExtra_1.3.4  data.table_1.14.0 ggplot2_3.3.3    
## [5] reshape_0.8.8     readxl_1.3.1      dplyr_1.0.6      
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_1.1.1   xfun_0.22          purrr_0.3.4        lattice_0.20-41   
##  [5] splines_4.0.3      colorspace_2.0-1   vctrs_0.3.8        generics_0.1.0    
##  [9] htmltools_0.5.1.1  viridisLite_0.4.0  mgcv_1.8-33        yaml_2.2.1        
## [13] utf8_1.2.1         rlang_0.4.11       pillar_1.6.0       glue_1.4.2        
## [17] withr_2.4.2        DBI_1.1.1          RColorBrewer_1.1-2 lifecycle_1.0.0   
## [21] plyr_1.8.6         stringr_1.4.0      munsell_0.5.0      gtable_0.3.0      
## [25] cellranger_1.1.0   rvest_1.0.0        evaluate_0.14      labeling_0.4.2    
## [29] knitr_1.33         fansi_0.4.2        highr_0.9          Rcpp_1.0.6        
## [33] scales_1.1.1       webshot_0.5.2      farver_2.1.0       systemfonts_1.0.1 
## [37] digest_0.6.27      stringi_1.5.3      grid_4.0.3         tools_4.0.3       
## [41] magrittr_2.0.1     tibble_3.1.1       crayon_1.4.1       pkgconfig_2.0.3   
## [45] Matrix_1.2-18      ellipsis_0.3.2     xml2_1.3.2         assertthat_0.2.1  
## [49] rmarkdown_2.8      svglite_2.0.0      httr_1.4.2         rstudioapi_0.13   
## [53] R6_2.5.0           nlme_3.1-149       compiler_4.0.3