This website accompanies the book chapter ‘Peer assessment using criteria or comparative judgement? A conceptual replication study on the learning effect of two assessment methods’ which is part of the book ‘The Power of Peer Learning. Fostering Students’ Learning Processes and Outcomes’. It contains all the code that was used to clean and prepare the data sets, run the analyses and report on the results. All data files (raw and processed), fitted models and other R-output, tables (as xlsx-file) and figures can be found in a project hosted on the Open Science Framework. This project also contains the RMarkdown-file that was used to create this website.

1 Data preparation

The code below is used to clean and prepare the data for analysis. This section also contains all results on exploratory factor analysis on the self-efficacy scales.

1.1 Sample A

1.1.1 Import and clean data

Background data
The file “background_A.xlsx” is loaded.

background_raw_A <- read_excel(here("01_Data", "01_Raw data", "Sample A", 
                                    "background_A.xlsx"))

The relevant variables are selected and renamed. The variables condition, gender, track and ID_class are recoded into a factor.

background_A <- 
  background_raw_A %>%
  select(ID_student = ID,
         condition  = Interv.,
         gender     = Gesl.,
         track      = Richting,
         ID_class   = KlasCat) %>%
  mutate(condition = condition - 1,
         conditionF = factor(condition,
                             levels = c(0, 1),
                             labels = c("absolute", "comparative"),
                             ordered = TRUE),
         genderF = factor(gender, 
                          levels  = c(0, 1), 
                          labels  = c("female", "male"),
                          ordered = TRUE),
         trackF  = factor(track, 
                          levels  = c(1, 2),
                          labels  = c("sport science", "science"),
                          ordered = TRUE),
         ID_classF = factor(ID_class,
                            levels = c(1:4),
                            labels = paste0("class ", c(1:4)),
                            ordered = TRUE))

Recoding of the variables is checked.

Check recoding of variable 'condition'

             
               0  1
  absolute    39  0
  comparative  0 39

Check recoding of variable 'gender'

        
          0  1
  female 32  0
  male    0 46

Check recoding of variable 'track'

               
                 1  2
  sport science 25  0
  science        0 53

Check recoding of variable 'ID_class'

         
           1  2  3  4
  class 1 22  0  0  0
  class 2  0 20  0  0
  class 3  0  0 17  0
  class 4  0  0  0 19

Pretest data
The file “pretest1_A.xlsx” is loaded.

pre_raw1_A <- read_excel(here("01_Data", "01_Raw data", "Sample A", 
                              "pretest1_A.xlsx"))

The relevant variables are selected and renamed.

pretest1_A <- 
  pre_raw1_A %>%
  select(ID_student         = ID,
         prior_know_recog   = Va,
         prior_know_formula = Vb,
         prior_know_unit1   = Vc,
         prior_know_approx  = Vd,
         prior_know_symbol  = Ve,
         prior_know_seman   = Vf,
         prior_know_solve   = Vg,
         prior_know_unit2   = Vh)

The file “pretest2_A.xlsx” is loaded.

pre_raw2_A <- read_excel(here("01_Data", "01_Raw data", "Sample A", 
                              "pretest2_A.xlsx"))

The relevant variables are selected and renamed.

pretest2_A <- 
  pre_raw2_A %>%
  select(ID_student       = ID,
         seff_mastery_    = ME1:ME6,
         seff_vicarious_  = VE1:VE6,
         seff_persuasion_ = SP1:SP6,
         seff_state_      = PS1:PS6)

Posttest data
The file “posttest1_A.xlsx” is loaded.

post_raw1_A <- read_excel(here("01_Data", "01_Raw data", "Sample A", 
                               "posttest1_A.xlsx"))

The relevant variables are selected and renamed.

posttest1_A <- 
  post_raw1_A %>%
  select(ID_student    = ID,
         know1_recog   = O1a,
         know1_formula = O1b,
         know1_unit1   = O1c,
         know1_approx  = O1d,
         know1_symbol  = O1e,
         know1_seman   = O1f,
         know1_solve   = O1g,
         know1_unit2   = O1h)

The file “posttest2_A.xlsx” is loaded.

post_raw2_A <- read_excel(here("01_Data", "01_Raw data", "Sample A", 
                               "posttest2_A.xlsx"))

The relevant variables are selected and renamed.

posttest2_A <- 
  post_raw2_A %>%
  select(ID_student    = ID,
         know2_recog   = O2a,
         know2_formula = O2b,
         know2_unit1   = O2c,
         know2_approx  = O2d,
         know2_symbol  = O2e,
         know2_seman   = O2f,
         know2_solve   = O2g,
         know2_unit2   = O2h)

1.1.2 Factor analysis

Preliminary
An exploratory factor analysis with oblique rotation is performed to examine the factor structure underlying the self-efficacy items. First, the suitability of factor analysis is checked by examining the correlations between all self-efficacy items. As can be seen, all items show several correlations of at least 0.30 with multiple other items.

Then, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is looked at. This measure is appropriate when cases to variable ratio is less than 1:5. KMO-values should be at least 0.50 (see Williams et al., 2010). One item (seff_vicarious_4) has an KMO-value smaller than 0.5. This item is removed.

Item	KMO
I make excellent grades on physics tests (seff_mastery_1)	0.71
I have always been successful with physics (seff_mastery_2)	0.70
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.73
I got good grades in physics on my last report card (seff_mastery_4)	0.62
I do well on physics assignments (seff_mastery_5)	0.80
I do well on even the most difficult physics assignments (seff_mastery_6)	0.78
Seeing adults do well in physics pushes me to do better (seff_vicarious_1)	0.59
When I see how my physics teacher solves a problem, I can picture myself solving the problem in the same way (seff_vicarious_2)	0.72
Seeing kids do better than me in physics pushes me to do better (seff_vicarious_3)	0.67
When I see how another student solves a physics problem, I can see myself solving the problem in the same way (seff_vicarious_4)	0.47
I imagine myself working through challenging physics problems successfully (seff_vicarious_5)	0.77
I compete with myself in physics (seff_vicarious_6)	0.56
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.64
People have told me that I have a talent for physics (seff_persuasion_2)	0.69
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.64
I have been praised for my ability in physics (seff_persuasion_4)	0.82
Other students have told me that I’m good at learning physics (seff_persuasion_5)	0.65
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.65
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.72
Doing physics work takes all of my energy (seff_state_2)	0.82
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.75
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.73
I get depressed when I think about learning physics (seff_state_5)	0.77
My whole body becomes tense when I have to do physics (seff_state_6)	0.76

Initial factor analysis
The number of factor to retain is determined using the Kaiser criterion (eigen values > 1), a scree plot and parallel analysis (Henson & Roberts, 2006). The plot below shows that the criteria indicate a different number of factors to retain. Application of the three criteria points to respectively 2 (scree plot), 3 (Kaiser criterion) and 5 factors (parallel analysis). Therefore, four exploratory factor analyses with oblique rotation are performed assuming respectively two, three, four or five factors.

The tables below show the solutions of the factor analyses whereby factor loadings of 0.40 (or higher) are printed in bold. In each solution, items with two loadings higher than 0.40, without any loading higher than 0.40, and/or a cross-loading differing less than 0.15 to the highest factor loading were found. Information about such items is added as footnote to each table. Looking at the patterns of loadings across the different solutions, it can be seen that the items measuring ‘social persuasion’ and ‘psychological state’ as sources of self-efficacy load on their own factor in the solutions assuming two, three or four factors. In the solution with five factors, two items measuring social persuasion load on a fifth factor. Most items on mastery experiences load on the same factor in the solution assuming two or three factors. In the four and five factor solutions, loadings are spread across three factors. Finally, in all solutions the factor loadings of the items measuring vicarious experiences are scattered across two different factors. Also, several items do not have a loading of at least 0.4 on one of the factors. Therefore, it is decided to rerun the factor analysis without the items on vicarious experiences .

Exploratory factor analysis with oblique rotation *assuming two factors*
Item	Factor		Communalities
	1	2
I make excellent grades on physics tests (seff_mastery_1)	0.42	0.12	0.19
I have always been successful with physics (seff_mastery_2)	0.51	0.21	0.30
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.66	0.13	0.46
I got good grades in physics on my last report card (seff_mastery_4)	0.36	0.09	0.14
I do well on physics assignments (seff_mastery_5)	0.53	0.23	0.33
I do well on even the most difficult physics assignments (seff_mastery_6)	0.42	0.33	0.28
Seeing adults do well in physics pushes me to do better (seff_vicarious_1)	-0.11	0.46	0.22
When I see how my physics teacher solves a problem, I can picture myself solving the problem in the same way (seff_vicarious_2)	0.50	0.08	0.26
Seeing kids do better than me in physics pushes me to do better (seff_vicarious_3)	-0.21	0.35	0.16
I imagine myself working through challenging physics problems successfully (seff_vicarious_5)	0.35	0.40	0.29
I compete with myself in physics (seff_vicarious_6)	-0.07	0.39	0.16
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.05	0.57	0.33
People have told me that I have a talent for physics (seff_persuasion_2)	-0.14	0.74	0.57
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.02	0.70	0.49
I have been praised for my ability in physics (seff_persuasion_4)	0.06	0.81	0.66
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.12	0.64	0.42
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.11	0.66	0.46
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.71	-0.15	0.53
Doing physics work takes all of my energy (seff_state_2)	0.60	-0.09	0.37
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.84	-0.11	0.72
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.70	0.02	0.48
I get depressed when I think about learning physics (seff_state_5)	0.48	0.21	0.28
My whole body becomes tense when I have to do physics (seff_state_6)	0.67	-0.15	0.47
Problematic items:
All factor loadings of the items ‘seff_mastery_4’, ‘seff_vicarious_3’ and ‘seff_vicarious_6’ are below 0.4. For the items ‘seff_mastery_6’ and ‘seff_vicarious_5’, a cross-loading smaller than 0.15 is found.

Exploratory factor analysis with oblique rotation *assuming three factors*
Item	Factor			Communalities
	1	2	3
I make excellent grades on physics tests (seff_mastery_1)	0.02	-0.15	0.68	0.45
I have always been successful with physics (seff_mastery_2)	0.10	-0.03	0.71	0.54
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.52	0.10	0.28	0.44
I got good grades in physics on my last report card (seff_mastery_4)	0.23	0.01	0.26	0.16
I do well on physics assignments (seff_mastery_5)	0.10	-0.04	0.76	0.63
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.02	0.03	0.79	0.63
Seeing adults do well in physics pushes me to do better (seff_vicarious_1)	-0.21	0.37	0.18	0.23
When I see how my physics teacher solves a problem, I can picture myself solving the problem in the same way (seff_vicarious_2)	0.41	0.05	0.21	0.27
Seeing kids do better than me in physics pushes me to do better (seff_vicarious_3)	-0.36	0.19	0.27	0.21
I imagine myself working through challenging physics problems successfully (seff_vicarious_5)	-0.04	0.11	0.70	0.52
I compete with myself in physics (seff_vicarious_6)	-0.31	0.18	0.39	0.24
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.08	0.61	-0.03	0.36
People have told me that I have a talent for physics (seff_persuasion_2)	-0.06	0.85	-0.12	0.71
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.05	0.80	-0.08	0.61
I have been praised for my ability in physics (seff_persuasion_4)	0.06	0.81	0.08	0.68
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.20	0.54	0.18	0.40
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.05	0.59	0.17	0.42
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.74	-0.06	0.01	0.57
Doing physics work takes all of my energy (seff_state_2)	0.52	-0.06	0.15	0.35
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.94	0.02	-0.03	0.87
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.73	0.13	-0.01	0.53
I get depressed when I think about learning physics (seff_state_5)	0.39	0.19	0.18	0.27
My whole body becomes tense when I have to do physics (seff_state_6)	0.66	-0.10	0.07	0.48
Problematic items:
All factor loadings of the items ‘seff_mastery_4’, ‘seff_vicarious_1’, ‘seff_vicarious_3’, ‘seff_vicarious_6’ and ‘seff_state_5’ are below 0.4. For the items ‘seff_mastery_6’ and ‘seff_vicarious_5’, a cross-loading smaller than 0.15 is found.

Exploratory factor analysis with oblique rotation *assuming four factors*
Item	Factor				Communalities
	1	2	3	4
I make excellent grades on physics tests (seff_mastery_1)	0.03	-0.15	0.55	0.25	0.44
I have always been successful with physics (seff_mastery_2)	0.00	0.03	0.93	-0.08	0.83
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.49	0.11	0.32	0.01	0.46
I got good grades in physics on my last report card (seff_mastery_4)	0.27	-0.01	0.15	0.16	0.16
I do well on physics assignments (seff_mastery_5)	0.08	-0.00	0.75	0.13	0.68
I do well on even the most difficult physics assignments (seff_mastery_6)	0.04	-0.00	0.54	0.42	0.59
Seeing adults do well in physics pushes me to do better (seff_vicarious_1)	-0.12	0.32	-0.10	0.43	0.35
When I see how my physics teacher solves a problem, I can picture myself solving the problem in the same way (seff_vicarious_2)	0.47	-0.01	0.00	0.31	0.34
Seeing kids do better than me in physics pushes me to do better (seff_vicarious_3)	-0.30	0.13	0.08	0.35	0.25
I imagine myself working through challenging physics problems successfully (seff_vicarious_5)	0.07	0.03	0.30	0.67	0.68
I compete with myself in physics (seff_vicarious_6)	-0.21	0.08	0.03	0.64	0.47
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.12	0.56	-0.13	0.13	0.36
People have told me that I have a talent for physics (seff_persuasion_2)	-0.10	0.92	0.03	-0.16	0.84
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.05	0.80	-0.06	0.02	0.64
I have been praised for my ability in physics (seff_persuasion_4)	0.10	0.74	-0.03	0.22	0.65
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.22	0.54	0.24	0.01	0.41
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.09	0.52	0.05	0.25	0.40
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.70	-0.05	0.13	-0.15	0.58
Doing physics work takes all of my energy (seff_state_2)	0.45	-0.02	0.34	-0.22	0.42
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.97	-0.02	-0.06	0.02	0.91
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.67	0.16	0.16	-0.21	0.57
I get depressed when I think about learning physics (seff_state_5)	0.46	0.12	-0.08	0.38	0.37
My whole body becomes tense when I have to do physics (seff_state_6)	0.68	-0.14	-0.01	0.10	0.50
Problematic items:
All factor loadings of the items ‘seff_mastery_4’ and ‘seff_vicarious_3’ are below 0.4. Item ‘seff_mastery_6’ has two factor loadings higer than 0.4. For the items ‘seff_vicarious_1’, ‘seff_state_2’ and ‘seff_state_5’, a cross-loading smaller than 0.15 is found.

Exploratory factor analysis with oblique rotation *assuming five factors*
Item	Factor					Communalities
	1	2	3	4	5
I make excellent grades on physics tests (seff_mastery_1)	-0.01	0.60	-0.10	0.22	-0.06	0.46
I have always been successful with physics (seff_mastery_2)	-0.01	0.91	0.02	-0.11	0.05	0.81
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.40	0.40	0.24	0.02	-0.23	0.53
I got good grades in physics on my last report card (seff_mastery_4)	0.25	0.17	-0.01	0.16	-0.02	0.16
I do well on physics assignments (seff_mastery_5)	0.06	0.77	-0.04	0.09	0.07	0.68
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.00	0.59	-0.00	0.40	-0.00	0.61
Seeing adults do well in physics pushes me to do better (seff_vicarious_1)	-0.20	-0.02	0.38	0.45	-0.10	0.40
When I see how my physics teacher solves a problem, I can picture myself solving the problem in the same way (seff_vicarious_2)	0.43	0.05	0.03	0.33	-0.09	0.35
Seeing kids do better than me in physics pushes me to do better (seff_vicarious_3)	-0.30	0.07	0.05	0.33	0.16	0.25
I imagine myself working through challenging physics problems successfully (seff_vicarious_5)	0.06	0.34	-0.06	0.63	0.18	0.67
I compete with myself in physics (seff_vicarious_6)	-0.23	0.07	0.03	0.61	0.11	0.47
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.09	-0.11	0.53	0.17	0.09	0.38
People have told me that I have a talent for physics (seff_persuasion_2)	-0.13	0.03	0.86	-0.15	0.15	0.85
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.01	-0.03	0.80	0.05	0.02	0.67
I have been praised for my ability in physics (seff_persuasion_4)	0.10	-0.04	0.58	0.22	0.30	0.65
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.08	0.12	0.14	-0.09	0.86	0.88
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.21	-0.06	0.15	0.22	0.73	0.71
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.74	0.10	-0.12	-0.16	0.12	0.60
Doing physics work takes all of my energy (seff_state_2)	0.46	0.32	-0.02	-0.21	0.01	0.42
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.97	-0.05	-0.03	0.03	0.01	0.91
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.62	0.20	0.27	-0.18	-0.19	0.61
I get depressed when I think about learning physics (seff_state_5)	0.37	0.01	0.25	0.45	-0.27	0.49
My whole body becomes tense when I have to do physics (seff_state_6)	0.69	-0.01	-0.18	0.10	0.06	0.51
Problematic items:
All factor loadings of the items ‘seff_mastery_4’, ‘seff_vicarious_1’, ‘seff_vicarious_3’, ‘seff_vicarious_6’ and ‘seff_state_5’ are below 0.4. Items ‘seff_mastery_3’ and ‘seff_mastery_6’ have two factor loadings higer than 0.4. For the items ‘seff_vicarious_1’, ‘seff_vicarious_2’, ‘seff_state_2’ and ‘seff_state_5’, a cross-loading smaller than 0.15 is found.

Second factor analysis
After removal of all the items measuring vicarious experiences as source of self-efficacy, the criteria for factor retention point again to either two (scree plot), three (Kaiser criterion) or five factors (parallel analysis).

Looking at the tables below, the three factor solution shows the clearest pattern of loadings: the items of each scale loading on a separate factor (factor 1: psychological state, factor 2: social persuasion, factor 3: mastery experiences). One exception is the item seff_mastery_4 (I got good grades in physics on my last report card.) which fails to load on any of the factors (loading > 0.4). Since this is also the case in all other solutions, this item is removed and a new exploratory factor analysis is performed.

Exploratory factor analysis with oblique rotation *assuming two factors*
Item	Factor		Communalities
	1	2
I make excellent grades on physics tests (seff_mastery_1)	0.40	0.04	0.17
I have always been successful with physics (seff_mastery_2)	0.50	0.17	0.28
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.67	0.12	0.46
I got good grades in physics on my last report card (seff_mastery_4)	0.33	0.06	0.11
I do well on physics assignments (seff_mastery_5)	0.51	0.16	0.29
I do well on even the most difficult physics assignments (seff_mastery_6)	0.39	0.23	0.21
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.07	0.60	0.36
People have told me that I have a talent for physics (seff_persuasion_2)	-0.12	0.82	0.68
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.01	0.75	0.57
I have been praised for my ability in physics (seff_persuasion_4)	0.08	0.81	0.67
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.10	0.63	0.41
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.11	0.63	0.41
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.72	-0.12	0.54
Doing physics work takes all of my energy (seff_state_2)	0.62	-0.05	0.38
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.84	-0.08	0.71
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.72	0.07	0.53
I get depressed when I think about learning physics (seff_state_5)	0.48	0.17	0.26
My whole body becomes tense when I have to do physics (seff_state_6)	0.65	-0.15	0.45
Problematic items:
All factor loadings of the items ‘seff_mastery_4’ and ‘seff_mastery_6’ are below 0.4.

Exploratory factor analysis with oblique rotation *assuming three factors*
Item	Factor			Communalities
	1	2	3
I make excellent grades on physics tests (seff_mastery_1)	-0.00	-0.11	0.65	0.41
I have always been successful with physics (seff_mastery_2)	0.00	-0.01	0.85	0.73
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.48	0.10	0.31	0.45
I got good grades in physics on my last report card (seff_mastery_4)	0.23	0.02	0.25	0.16
I do well on physics assignments (seff_mastery_5)	0.04	-0.00	0.83	0.72
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.00	0.08	0.69	0.50
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.10	0.62	-0.08	0.37
People have told me that I have a talent for physics (seff_persuasion_2)	-0.11	0.84	-0.04	0.72
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.03	0.79	-0.06	0.61
I have been praised for my ability in physics (seff_persuasion_4)	0.08	0.81	0.02	0.66
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.23	0.56	0.22	0.43
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.08	0.61	0.10	0.40
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.71	-0.08	0.07	0.56
Doing physics work takes all of my energy (seff_state_2)	0.46	-0.06	0.23	0.35
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.98	0.00	-0.07	0.91
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.69	0.11	0.07	0.52
I get depressed when I think about learning physics (seff_state_5)	0.44	0.19	0.04	0.24
My whole body becomes tense when I have to do physics (seff_state_6)	0.67	-0.11	0.03	0.48
Problematic items:
All factor loadings of the item ‘seff_mastery_4’ are below 0.4.

Exploratory factor analysis with oblique rotation *assuming four factors*
Item	Factor				Communalities
	1	2	3	4
I make excellent grades on physics tests (seff_mastery_1)	-0.04	0.68	-0.06	-0.04	0.43
I have always been successful with physics (seff_mastery_2)	-0.00	0.84	-0.06	0.10	0.73
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.40	0.37	0.21	-0.17	0.49
I got good grades in physics on my last report card (seff_mastery_4)	0.20	0.26	0.05	-0.07	0.16
I do well on physics assignments (seff_mastery_5)	0.03	0.82	-0.05	0.07	0.70
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.07	0.73	0.13	-0.08	0.53
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.03	-0.04	0.64	0.01	0.40
People have told me that I have a talent for physics (seff_persuasion_2)	-0.14	-0.03	0.74	0.18	0.71
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.06	-0.01	0.81	-0.00	0.66
I have been praised for my ability in physics (seff_persuasion_4)	0.05	0.03	0.70	0.21	0.65
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.03	0.06	0.08	0.95	1.00
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.19	0.00	0.28	0.61	0.59
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.77	0.03	-0.17	0.13	0.60
Doing physics work takes all of my energy (seff_state_2)	0.48	0.22	-0.09	0.04	0.36
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.97	-0.07	0.01	-0.02	0.91
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.62	0.12	0.20	-0.15	0.54
I get depressed when I think about learning physics (seff_state_5)	0.31	0.14	0.41	-0.35	0.38
My whole body becomes tense when I have to do physics (seff_state_6)	0.69	0.02	-0.12	0.03	0.49
Problematic items:
All factor loadings of the items ‘seff_mastery_3’ and ‘seff_mastery_4’ are below 0.4. For the item ‘seff_state_5’, a cross-loading smaller than 0.15 is found.

Exploratory factor analysis with oblique rotation *assuming five factors*
Item	Factor					Communalities
	1	2	3	4	5
I make excellent grades on physics tests (seff_mastery_1)	-0.11	0.66	-0.22	0.00	0.39	0.62
I have always been successful with physics (seff_mastery_2)	0.05	0.88	0.08	0.00	-0.15	0.80
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.41	0.38	0.19	-0.25	0.27	0.60
I got good grades in physics on my last report card (seff_mastery_4)	0.20	0.23	-0.04	0.05	0.09	0.15
I do well on physics assignments (seff_mastery_5)	0.09	0.79	0.02	0.05	-0.12	0.68
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.04	0.65	-0.09	0.14	0.21	0.52
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	-0.04	-0.09	0.29	0.24	0.61	0.65
People have told me that I have a talent for physics (seff_persuasion_2)	-0.08	0.00	0.94	0.04	-0.04	0.93
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.02	-0.03	0.74	0.04	0.10	0.62
I have been praised for my ability in physics (seff_persuasion_4)	0.05	0.02	0.47	0.37	0.21	0.61
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.15	0.18	0.23	0.67	-0.20	0.71
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.11	0.01	0.01	0.93	0.10	0.89
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.76	0.06	-0.04	0.05	-0.16	0.60
Doing physics work takes all of my energy (seff_state_2)	0.51	0.24	0.01	0.00	-0.10	0.38
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.95	-0.06	-0.06	0.05	0.07	0.90
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.64	0.13	0.22	-0.19	0.18	0.60
I get depressed when I think about learning physics (seff_state_5)	0.33	0.04	0.04	-0.03	0.47	0.41
My whole body becomes tense when I have to do physics (seff_state_6)	0.71	0.00	-0.15	0.10	-0.12	0.53
Problematic items:
All factor loadings of the item ‘seff_mastery_4’ are below 0.4. For the items ‘seff_mastery_3’, ‘seff_persuasion_4’ and ‘seff_state_5’, a cross-loading smaller than 0.15 is found.

Third factor analysis
After removal of the item seff_mastery_4 the criteria for factor retention point to either two (scree plot), three (Kaiser criterion) or four factors (parallel analysis).

Again, the three factor solution shows the clearest pattern of loadings: the items of each scale load on a separate factor (factor 1: social persuasion , factor 2: psychological state, factor 3: mastery experiences). One exception is the item seff_mastery_3 (Even when I study very hard, I do poorly in physics) which loads on the second factor (instead of the third). Therefore, this item is removed and a new exploratory factor analysis is performed.

Exploratory factor analysis with oblique rotation *assuming two factors*
Item	Factor		Communalities
	1	2
I make excellent grades on physics tests (seff_mastery_1)	0.40	0.04	0.16
I have always been successful with physics (seff_mastery_2)	0.50	0.17	0.27
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.67	0.13	0.46
I do well on physics assignments (seff_mastery_5)	0.50	0.16	0.27
I do well on even the most difficult physics assignments (seff_mastery_6)	0.38	0.22	0.20
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.07	0.60	0.36
People have told me that I have a talent for physics (seff_persuasion_2)	-0.12	0.82	0.69
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.01	0.76	0.57
I have been praised for my ability in physics (seff_persuasion_4)	0.08	0.81	0.66
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.10	0.63	0.41
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.11	0.63	0.41
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.72	-0.12	0.54
Doing physics work takes all of my energy (seff_state_2)	0.63	-0.05	0.40
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.84	-0.08	0.71
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.74	0.08	0.55
I get depressed when I think about learning physics (seff_state_5)	0.49	0.17	0.26
My whole body becomes tense when I have to do physics (seff_state_6)	0.65	-0.15	0.44
Problematic items:
All factor loadings of the item ‘seff_mastery_6’ are below 0.4.

Exploratory factor analysis with oblique rotation *assuming three factors*
Item	Factor			Communalities
	1	2	3
I make excellent grades on physics tests (seff_mastery_1)	-0.11	0.00	0.64	0.40
I have always been successful with physics (seff_mastery_2)	-0.01	0.00	0.86	0.74
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.10	0.49	0.31	0.46
I do well on physics assignments (seff_mastery_5)	-0.00	0.04	0.82	0.70
I do well on even the most difficult physics assignments (seff_mastery_6)	0.08	-0.00	0.69	0.49
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.62	0.10	-0.09	0.37
People have told me that I have a talent for physics (seff_persuasion_2)	0.84	-0.10	-0.04	0.71
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.79	0.03	-0.06	0.61
I have been praised for my ability in physics (seff_persuasion_4)	0.81	0.08	0.02	0.66
Other students have told me that I’m good at learning physics (seff_persuasion_5)	0.56	-0.23	0.23	0.43
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.61	0.07	0.11	0.40
Just being in physics class makes feel stressed and nervous (seff_state_1)	-0.08	0.72	0.07	0.56
Doing physics work takes all of my energy (seff_state_2)	-0.06	0.47	0.24	0.37
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.00	0.97	-0.07	0.89
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.11	0.70	0.07	0.53
I get depressed when I think about learning physics (seff_state_5)	0.20	0.45	0.03	0.24
My whole body becomes tense when I have to do physics (seff_state_6)	-0.11	0.67	0.02	0.48

Exploratory factor analysis with oblique rotation *assuming four factors*
Item	Factor				Communalities
	1	2	3	4
I make excellent grades on physics tests (seff_mastery_1)	-0.04	0.68	-0.06	-0.05	0.43
I have always been successful with physics (seff_mastery_2)	-0.00	0.85	-0.06	0.09	0.74
Even when I study very hard, I do poorly in physics (seff_mastery_3)	0.39	0.39	0.22	-0.19	0.51
I do well on physics assignments (seff_mastery_5)	0.04	0.81	-0.05	0.08	0.69
I do well on even the most difficult physics assignments (seff_mastery_6)	-0.07	0.72	0.13	-0.08	0.51
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.03	-0.04	0.64	0.01	0.41
People have told me that I have a talent for physics (seff_persuasion_2)	-0.15	-0.03	0.74	0.19	0.71
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	-0.06	-0.01	0.81	0.00	0.66
I have been praised for my ability in physics (seff_persuasion_4)	0.05	0.02	0.69	0.22	0.65
Other students have told me that I’m good at learning physics (seff_persuasion_5)	-0.03	0.07	0.08	0.94	0.97
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.19	0.00	0.28	0.62	0.59
Just being in physics class makes feel stressed and nervous (seff_state_1)	0.78	0.03	-0.17	0.14	0.61
Doing physics work takes all of my energy (seff_state_2)	0.48	0.24	-0.08	0.03	0.37
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.96	-0.06	0.02	-0.02	0.89
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.62	0.13	0.21	-0.16	0.55
I get depressed when I think about learning physics (seff_state_5)	0.32	0.13	0.41	-0.35	0.38
My whole body becomes tense when I have to do physics (seff_state_6)	0.70	0.01	-0.13	0.03	0.49
Problematic items:
All factor loadings of the items ‘seff_mastery_3’ are below 0.4. For the item ‘seff_state_5’, a cross-loading smaller than 0.15 is found.

Fourth (final) factor analysis
After removal of the item seff_mastery_3 the criteria for factor retention point again to either two (scree plot), three (Kaiser criterion) or four factors (parallel analysis).

Below, the solution with three factors is shown.

Exploratory factor analysis with oblique rotation *assuming three factors*
Item	Factor			Communalities
	1	2	3
I make excellent grades on physics tests (seff_mastery_1)	-0.11	-0.00	0.62	0.38
I have always been successful with physics (seff_mastery_2)	-0.00	0.01	0.84	0.70
I do well on physics assignments (seff_mastery_5)	-0.00	0.04	0.85	0.74
I do well on even the most difficult physics assignments (seff_mastery_6)	0.08	-0.00	0.70	0.51
My physics teachers have told that I am good at learning physics (seff_persuasion_1)	0.62	0.09	-0.09	0.37
People have told me that I have a talent for physics (seff_persuasion_2)	0.83	-0.10	-0.04	0.71
Adults in my family have told me what a good physics student I am (seff_persuasion_3)	0.78	0.03	-0.06	0.60
I have been praised for my ability in physics (seff_persuasion_4)	0.81	0.08	0.03	0.67
Other students have told me that I’m good at learning physics (seff_persuasion_5)	0.57	-0.22	0.23	0.44
My classmates like to work with me in physics because they think I’m good at it (seff_persuasion_6)	0.62	0.08	0.12	0.42
Just being in physics class makes feel stressed and nervous (seff_state_1)	-0.07	0.72	0.08	0.58
Doing physics work takes all of my energy (seff_state_2)	-0.06	0.48	0.24	0.37
I start to feel stressed-out as soon as I begin my physics work (seff_state_3)	0.01	0.96	-0.06	0.89
My mind goes blank and I am unable to think clearly when doing physics work (seff_state_4)	0.12	0.69	0.06	0.51
I get depressed when I think about learning physics (seff_state_5)	0.20	0.44	0.04	0.24
My whole body becomes tense when I have to do physics (seff_state_6)	-0.10	0.67	0.05	0.49

As can be seen, items load per sub-scale on a separate factor (loading > 0.40). These items explain 54% of the total variance. Next, the internal consistency is checked per sub-scale and found to be high for all sub-scales (\(\alpha\) psychological state = 0.84, \(\alpha\) social persuasion = 0.86, \(\alpha\) mastery experience = 0.84).

1.1.3 Create variables

Prior knowledge and self-efficacy
The three scales on ‘sources of self-efficacy’ are created by summing the items and dividing them by the total number of items of that scale. In a similar way, the variable ‘prior knowledge’ is created by summing the eight items measuring problem-solving ability before the intervention. For the purpose of analysis, the scales measuring ‘self-efficacy’ and the variable ‘prior knowledge’ are standardized.

pretest_A <- 
  left_join(pretest1_A, pretest2_A, by = "ID_student") %>%
  rowwise() %>%
  mutate(self_eff_mastery = sum(seff_mastery_1, seff_mastery_2,
                                seff_mastery_5, seff_mastery_6) / 4,
         self_eff_state   = sum(seff_state_1, seff_state_2,
                                seff_state_3, seff_state_4,
                                seff_state_5, seff_state_6) / 6,
         self_eff_persuasion = sum(seff_persuasion_1, seff_persuasion_2,
                                   seff_persuasion_3, seff_persuasion_4,
                                   seff_persuasion_5, seff_persuasion_6) / 6,
         prior_knowledge = sum(prior_know_recog,  prior_know_formula,
                               prior_know_unit1,  prior_know_approx,
                               prior_know_symbol, prior_know_seman,
                               prior_know_solve,  prior_know_unit2)) %>%
  ungroup() %>%
  mutate(self_eff_masteryZ = as.numeric(scale(self_eff_mastery)),
         self_eff_persuasionZ = as.numeric(scale(self_eff_persuasion)),
         self_eff_stateZ = as.numeric(scale(self_eff_state)),
         prior_knowledgeZ = as.numeric(scale(prior_knowledge)))

Problem solving performance
The 16 items measuring students’ performance on the physic problems after the intervention are summed to represent students’ performance regarding problem solving in the context of physics. This variable is also standardised.

posttest_A <- 
  left_join(posttest1_A, posttest2_A, by = "ID_student") %>%
  rowwise() %>%
  mutate(ability1 = sum(know1_recog,  know1_formula,
                        know1_unit1,  know1_approx,
                        know1_symbol, know1_seman,
                        know1_solve,  know1_unit2),
         ability2 = sum(know2_recog,  know2_formula,
                        know2_unit1,  know2_approx,
                        know2_symbol, know2_seman,
                        know2_solve,  know2_unit2),
         ability  = ability1 + ability2) %>%
  ungroup() %>%
  mutate(abilityZ  = as.numeric(scale(ability)))

1.1.4 Create data for analysis

The data of sample A is only used to examine research question 2. The data set should contain the following variables:
- ID student (variable ID_student)
- ID class (variable ID_class)
- condition (variables condition and conditionF)
- gender (variables gender and genderF)
- the study track of the student (variables track and trackF)
- self-efficacy at pre-test (variables self_eff_state, self_eff_persuasion, self_eff_mastery, self_eff_stateZ, self_eff_persuasionZ and self_eff_masteryZ)
- prior knowledge (variables prior_knowledge and prior_knowledgeZ)
- problem-solving ability at post-test (variables ability and abilityZ)

data_RQ2_A <-
  background_A %>%
  left_join(pretest_A,  by = "ID_student") %>%
  left_join(posttest_A, by = "ID_student") %>%
  select(ID_student, ID_class, 
         contains("condition"), contains("gender"), contains("track"), 
         contains("prior_knowledge"), contains("self_eff_state"),
         contains("self_eff_persuasion"), contains("self_eff_mastery"),
         ability, abilityZ)

The data set is saved. This data set can be found here.

save(data_RQ2_A, file = here("01_Data", "02_Processed data", "Sample A", 
                             "data_RQ2_A.RData"))

1.2 Sample B

1.2.1 Import and clean data

Pre- and posttest data
The file “pretest_posttest_B.xlsx” is loaded.

prepost_raw_B <- read_excel(here("01_Data", "01_Raw data", "Sample B", 
                                 "pretest_posttest_B.xlsx"))

The relevant variables are selected and renamed. The variable condition is recoded into a factor.

prepost_B <- 
  prepost_raw_B %>%
  select(ID_student      = ID,
         condition       = conditie,
         prior_knowledge = vknum,
         seff_idea_      = Item1:Item5,
         seff_conven_    = Item6:Item9,
         seff_selfreg_   = Item10:Item15,
         ability         = Schrijfkwaliteit) %>%
  mutate(conditionF = factor(condition,
                             levels = c(0, 1),
                             labels = c("absolute", "comparative"),
                             ordered = TRUE))

Recoding of the variable conditionF is checked.

             
               0  1
  absolute    21  0
  comparative  0 21

Feedback data
The file “feedback_raw_B.xlsx” is loaded.

feedback_raw_B <- read_excel(here("01_Data", "01_Raw data", "Sample B", 
                                  "feedback_B.xlsx"))

The relevant variables are selected and renamed. The variables condition, ID_product and ID_student are recoded into a factor.

feedback_B <- 
  feedback_raw_B %>%
  select(ID_student     = ID,
         ID_product     = tekstid,
         condition      = Conditie,
         pos_vocabulary = woordpos,
         pos_spelling   = spellingpos,
         pos_grammar    = grampos,
         pos_syntax     = zinsbouwpos,
         pos_content    = inhoudpos,
         neg_vocabulary = woordneg,
         neg_spelling   = spellingneg,
         neg_grammar    = gramneg,
         neg_syntax     = zinsbouwneg,
         neg_content    = inhoudneg) %>%
  mutate(conditionF = factor(condition,
                             levels = c(0, 1),
                             labels = c("absolute", "comparative"),
                             ordered = TRUE),
         ID_studentF = factor(ID_student,
                              levels = c(1:42),
                              labels = paste0("student ", c(1:42)),
                              ordered = TRUE),
         ID_productF = factor(ID_product,
                              levels = c("A", "B", "C", "D", "E"),
                              labels = paste0("product ", c("A", "B", "C", "D", "E")),
                              ordered = TRUE))

Recoding of variable conditionF is checked.

             
                0   1
  absolute    105   0
  comparative   0 105

Recoding of variable ID_studentF is checked.

42 x 42 sparse Matrix of class "dgCMatrix"
                                                                              
1  5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2  . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3  . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4  . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5  . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6  . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7  . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8  . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9  . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10 . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . . .
12 . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . . .
13 . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . . .
14 . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . . .
15 . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . . .
16 . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . . .
17 . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . . .
18 . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . . .
19 . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . . .
20 . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . . .
21 . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . . .
22 . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . . .
23 . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . . .
24 . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . . .
25 . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . . .
26 . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . . .
27 . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . . .
28 . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . . .
29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . . .
30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . . .
31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . . .
32 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . . .
33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . . .
34 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . . .
35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . . .
36 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 . .
37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 .
38 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
          
1  . . . .
2  . . . .
3  . . . .
4  . . . .
5  . . . .
6  . . . .
7  . . . .
8  . . . .
9  . . . .
10 . . . .
11 . . . .
12 . . . .
13 . . . .
14 . . . .
15 . . . .
16 . . . .
17 . . . .
18 . . . .
19 . . . .
20 . . . .
21 . . . .
22 . . . .
23 . . . .
24 . . . .
25 . . . .
26 . . . .
27 . . . .
28 . . . .
29 . . . .
30 . . . .
31 . . . .
32 . . . .
33 . . . .
34 . . . .
35 . . . .
36 . . . .
37 . . . .
38 . . . .
39 5 . . .
40 . 5 . .
41 . . 5 .
42 . . . 5

Recoding of variable ID_productF is checked.

           
             A  B  C  D  E
  product A 42  0  0  0  0
  product B  0 42  0  0  0
  product C  0  0 42  0  0
  product D  0  0  0 42  0
  product E  0  0  0  0 42

1.2.2 Factor analysis

Then, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is looked at. This measure is appropriate when cases to variable ratio is less than 1:5. KMO-values should be at least 0.50 (see Williams et al., 2010). All items have an acceptable KMO-value (KMO > 0.5).

Item	KMO
I can think of many ideas for my writing (seff_idea_1)	0.81
I can put my ideas into writing (seff_idea_2)	0.88
I can think of many words to describe my ideas (seff_idea_3)	0.82
I can think of a lot of original ideas (seff_idea_4)	0.80
I know exactly where to place my ideas in my writing (seff_idea_5)	0.89
I can spell my words correctly (seff_conven_1)	0.89
I can write complete sentences (seff_conven_2)	0.87
I can punctuate my sentences correctly (seff_conven_3)	0.93
I can write grammatically correct sentences (seff_conven_4)	0.84
I can focus on my writing for at least one hour (seff_selfreg_1)	0.61
I can avoid distractions while I write (seff_selfreg_2)	0.73
I can start writing assignments quickly (seff_selfreg_3)	0.90
I can control my frustration when I write (seff_selfreg_4)	0.81
I can think of my writing goals before I write (seff_selfreg_5)	0.90
I can keep writing even when it’s difficult (seff_selfreg_6)	0.91

Below factor loadings are shown of an EFA with oblique rotation assuming one factor. A cut-off of 0.40 (absolute value) is used for item retention (Hair et al., 1998). The item seff_selfreg_1 (I can focus on my writing for at least one hour) has a factor loading smaller than 0.40 and is removed. Next, a new EFA with oblique rotation is performed.

Items	Factor 1	Communalities
I can think of many ideas for my writing (seff_idea_1)	0.63	0.40
I can put my ideas into writing (seff_idea_2)	0.90	0.80
I can think of many words to describe my ideas (seff_idea_3)	0.83	0.70
I can think of a lot of original ideas (seff_idea_4)	0.66	0.44
I know exactly where to place my ideas in my writing (seff_idea_5)	0.56	0.31
I can spell my words correctly (seff_conven_1)	0.81	0.65
I can write complete sentences (seff_conven_2)	0.89	0.80
I can punctuate my sentences correctly (seff_conven_3)	0.69	0.47
I can write grammatically correct sentences (seff_conven_4)	0.83	0.69
I can focus on my writing for at least one hour (seff_selfreg_1)	0.31	0.10
I can avoid distractions while I write (seff_selfreg_2)	0.53	0.28
I can start writing assignments quickly (seff_selfreg_3)	0.81	0.65
I can control my frustration when I write (seff_selfreg_4)	0.56	0.32
I can think of my writing goals before I write (seff_selfreg_5)	0.86	0.74
I can keep writing even when it’s difficult (seff_selfreg_6)	0.89	0.79

Second (final) factor analysis
After removal of the item seff_selfreg_1 all criteria point again to one factor.

Now, all items have a factor loading of at least 0.40. These items explain 57% of the total variance. Next, the internal consistency of the items is checked and found to be high (\(\alpha\) = 0.95).

Items	Factor 1	Communalities
I can think of many ideas for my writing (seff_idea_1)	0.64	0.40
I can put my ideas into writing (seff_idea_2)	0.90	0.80
I can think of many words to describe my ideas (seff_idea_3)	0.84	0.70
I can think of a lot of original ideas (seff_idea_4)	0.67	0.44
I know exactly where to place my ideas in my writing (seff_idea_5)	0.56	0.31
I can spell my words correctly (seff_conven_1)	0.81	0.65
I can write complete sentences (seff_conven_2)	0.89	0.79
I can punctuate my sentences correctly (seff_conven_3)	0.69	0.47
I can write grammatically correct sentences (seff_conven_4)	0.83	0.68
I can avoid distractions while I write (seff_selfreg_2)	0.52	0.27
I can start writing assignments quickly (seff_selfreg_3)	0.81	0.65
I can control my frustration when I write (seff_selfreg_4)	0.56	0.31
I can think of my writing goals before I write (seff_selfreg_5)	0.86	0.74
I can keep writing even when it’s difficult (seff_selfreg_6)	0.89	0.79

1.2.3 Create variables

Prior knowledge and self-efficacy
The variable ‘self-efficacy’ is created by summing the 14 self-efficacy items and dividing them by 14 (the total number of items). For the purpose of analysis, self-efficacy and prior knowledge are standardized.

data_RQ2_B <- 
  prepost_B %>%
  rowwise() %>%
  mutate(self_efficacy = sum(seff_idea_1,   seff_idea_2,
                             seff_idea_3,   seff_idea_4,
                             seff_idea_5,
                             seff_conven_1,  seff_conven_2,
                             seff_conven_3,  seff_conven_4,
                             seff_selfreg_2, seff_selfreg_3,
                             seff_selfreg_4, seff_selfreg_5,
                             seff_selfreg_6) / 14) %>%
  ungroup() %>%
  mutate(self_efficacyZ = as.numeric(scale(self_efficacy)),
         prior_knowledgeZ = as.numeric(scale(prior_knowledge)))

Quality of feedback
The variables on the content of the feedback are converted into dummy variables (code 0: aspect not mentioned; code 1: aspect is mentioned). In addition, the number of positive and negative unique arguments per product (variables pos_total and neg_total) and the total number of unique arguments per product (variable total) are calculated.

feedback_B <- 
  feedback_B %>%
  mutate(pos_vocabularyD = ifelse(pos_vocabulary >=1, yes = 1, no = 0),
         pos_spellingD   = ifelse(pos_spelling >=1,   yes = 1, no = 0), 
         pos_grammarD    = ifelse(pos_grammar >=1,    yes = 1, no = 0),
         pos_syntaxD     = ifelse(pos_syntax >=1,     yes = 1, no = 0),
         pos_contentD    = ifelse(pos_content >=1,    yes = 1, no = 0),
         neg_vocabularyD = ifelse(neg_vocabulary >=1, yes = 1, no = 0),
         neg_spellingD   = ifelse(neg_spelling >=1,   yes = 1, no = 0), 
         neg_grammarD    = ifelse(neg_grammar >=1,    yes = 1, no = 0),
         neg_syntaxD     = ifelse(neg_syntax >=1,     yes = 1, no = 0),
         neg_contentD    = ifelse(neg_content >=1,    yes = 1, no = 0)) %>%
  rowwise() %>%
  mutate(pos_total = sum(pos_vocabularyD, pos_spellingD, 
                         pos_grammarD, pos_syntax, pos_contentD),
         neg_total = sum(neg_vocabularyD, neg_spellingD,
                         neg_grammarD, neg_syntax, neg_contentD),
         total     = pos_total + neg_total) %>%
  ungroup()

Recoding of the dummy variables is checked.

Check recoding of dummy variable 'pos_vocabularyD'

Check recoding of dummy variable 'pos_spellingD'

Check recoding of dummy variable 'pos_grammarD'

Check recoding of dummy variable 'pos_syntaxD'

   
      0   1   2
  0 171   0   0
  1   0  36   3

Check recoding of dummy variable 'pos_contentD'

   
      0   1   2   3
  0 151   0   0   0
  1   0  57   1   1

Check recoding of dummy variable 'neg_vocabularyD'

Check recoding of dummy variable 'neg_spellingD'

   
      0   1   2   5
  0 178   0   0   0
  1   0  30   1   1

Check recoding of dummy variable 'neg_grammarD'

   
      0   1   2   4
  0 165   0   0   0
  1   0  41   3   1

Check recoding of dummy variable 'neg_syntaxD'

   
      0   1   2   3
  0 160   0   0   0
  1   0  45   4   1

Check recoding of dummy variable 'neg_contentD'

   
      0   1   2
  0 181   0   0
  1   0  25   4

Writing performance
The variable that maps students’ writing performance, results from the assessment that set up in Comproved. This variable is also standardised.

data_RQ2_B <- 
  data_RQ2_B %>%
  mutate(abilityZ = as.numeric(scale(ability)))

1.2.4 Create data for analysis

Two data sets are created (one for each research question).

Research question 1 (RQ1)
For RQ1, this data set should contain the following variables:
- ID of the student (variable ID_student)
- ID of the product (variable ID_product)
- condition (variables condition and conditionF)
- number of positive feedback statements (variable pos_total)
- number of negative feedback statements (variable neg_total)
- total number of feedback statements (variable total)
- dummy variables on negative feedback per category (variables neg_vocabularyD, neg_contentD, neg_grammarD, neg_syntaxDand neg_spellingD)
- dummy variables on negative feedback per category (variables pos_vocabularyD, pos_contentD, pos_grammarD, pos_syntaxDand pos_spellingD)

data_RQ1_B <- 
  feedback_B %>%
  select(ID_student, contains("product"), contains("condition"),
         pos_total,  neg_total,  total,
         ends_with("D"))

Research question 2 (RQ2)
For RQ2, the data set should contain the following variables:
- ID student (variable ID_student)
- condition (variables condition and conditionF)
- self-efficacy at pre-test (variables self_efficacy and self_efficacyZ)
- prior knowledge (variables prior_knowledge and prior_knowledgeZ)
- writing ability at post-test (variables ability and abilityZ)

data_RQ2_B <- 
  data_RQ2_B %>%
  select(ID_student, prior_knowledge, self_efficacy,
         contains("condition"),
         ability, ends_with("Z"))

Both data sets are saved. These data sets can be found here.

save(data_RQ1_B, file = here("01_Data", "02_Processed data", "Sample B", 
                             "data_RQ1_B.RData"))
save(data_RQ2_B, file = here("01_Data", "02_Processed data", "Sample B", 
                             "data_RQ2_B.RData"))

1.3 Sample C

1.3.1 Import and clean data

Pre- and posttest data
The file “Pretest_C.xlsx” is loaded. 23 empty rows at the end of the file are removed

pretest_raw_C <- read_excel(here("01_Data", "01_Raw data", "Sample C", 
                                 "Pretest_C.xlsx"),
                            n_max = 23) # remove empty rows

The relevant variables are selected and renamed. The variables condition, gender, session1, processed1 and writing_task1 are recoded into a factor.

pretest_C <- 
  pretest_raw_C %>%
  select(ID_student      = ID_Student,
         condition       = Conditie,
         age             = Leeftijd,
         gender          = Geslacht,
         session1        = Contact1,
         processed       = VerwerkL,
         writing_task1   = Schrijftaak1,
         prior_know_     = Score_V1:Score_V5,
         seff_content_   = ZE_item1:ZE_item2,
         seff_interpret_ = ZE_item3:ZE_item6,
         seff_writing_   = ZE_item7:ZE_item9,
         seff_language_  = ZE_item10:ZE_item13) %>%
  mutate_at(vars(prior_know_1:prior_know_5), as.numeric) %>%
  mutate(conditionF = factor(condition,
                             levels = c(0, 1),
                             labels = c("absolute", "comparative"),
                             ordered = TRUE),
         genderF = factor(gender,
                          levels = c(1, 0),
                          labels = c("female", "male"),
                          ordered = TRUE),
         session1F = factor(session1,
                            levels = c(0, 1),
                            labels = c("absent", "present"),
                            ordered = TRUE),
         processedF = factor(processed,
                             levels = c(0, 1),
                             labels = c("not processed", "processed"),
                             ordered = TRUE),
         writing_task1F = factor(writing_task1,
                                 levels = c(0, 1),
                                 labels = c("not participed",
                                            "participated"),
                                 ordered = TRUE))

Recoding of the variables is checked.

Check recoding of variable 'condition'

             
               0  1
  absolute    12  0
  comparative  0 10

Check recoding of variable 'gender'

        
          0  1
  female  0 18
  male    4  0

Check recoding of variable 'session1'

         
           0  1
  absent   2  0
  present  0 20

Check recoding of variable 'processed'

               
                 0  1
  not processed  9  0
  processed      0 13

Check recoding of variable 'writing_task1'

                
                  0  1
  not participed  7  0
  participated    0 15

The file “Posttest_C.xlsx” is loaded.

posttest_raw_C <- read_excel(here("01_Data", "01_Raw data", "Sample C", 
                                  "Posttest_C.xlsx"))

Both variables are selected and renamed.

posttest_C <- 
  posttest_raw_C %>%
  select(ID_student = ID_Student,
         ability    = Bekwaamheid)

Feedback data
The file “Feedback_Absoluut_C.xlsx” is loaded. This file contains the feedback data of the criteria condition.

feedback_abs_raw_C <- read_excel(here("01_Data", "01_Raw data", "Sample C", 
                                      "Feedback_Absoluut_C.xlsx"))

The relevant variables are selected and renamed. The variable ID_product is recoded into a factor.

feedback_abs_C <- 
  feedback_abs_raw_C %>%
  select(ID_student    = ID_Student,
         ID_product    = ID_Rapportage,
         condition     = Conditie,
         pos_content   = Pos_Inh_opb,
         pos_interpret = Pos_Interpret,
         pos_style     = Pos_WetenschS,
         pos_language  = Pos_Taalgebr,
         pos_holistic  = Pos_Holist,
         pos_length    = Pos_Lengte,
         pos_other     = Pos_Overige,
         neg_content   = Neg_Inh_opb,
         neg_interpret = Neg_Interpret,
         neg_style     = Neg_WetenschS,
         neg_language  = Neg_Taalgebr,
         neg_holistic  = Neg_Holist,
         neg_length    = Neg_Lengte,
         neg_other     = Neg_Overige) %>%
  mutate(ID_productF = factor(ID_product,
                              levels = c("A", "B", "C", "D", "E", "F"),
                              labels = paste0("product ",
                                              c("A", "B", "C", "D", "E", "F")),
                              ordered = TRUE))

Recoding of variable ID_productF is checked.

           
             A  B  C  D  E  F
  product A 13  0  0  0  0  0
  product B  0 13  0  0  0  0
  product C  0  0 13  0  0  0
  product D  0  0  0 13  0  0
  product E  0  0  0  0 13  0
  product F  0  0  0  0  0 13

The file “Feedback_Comparatief_C.xlsx” is loaded. This file contains the feedback data of the comparative condition.

feedback_comp_raw_C <- read_excel(here("01_Data", "01_Raw data", "Sample C", 
                                       "Feedback_Comparatief_C.xlsx"))

The relevant variables are selected and renamed. The variable ID_product is recoded into a factor.

feedback_comp_C <- 
  feedback_comp_raw_C %>%
  select(ID_student    = ID_Student,
         ID_product    = ID_Rapportage,
         condition     = Conditie,
         pos_content   = Pos_Inh_opb,
         pos_interpret = Pos_Interpret,
         pos_style     = Pos_WetenschS,
         pos_language  = Pos_Taalgebr,
         pos_holistic  = Pos_Holist,
         pos_length    = Pos_Lengte,
         pos_other     = Pos_Overige,
         neg_content   = Neg_Inh_opb,
         neg_interpret = Neg_Interpret,
         neg_style     = Neg_WetenschS,
         neg_language  = Neg_Taalgebr,
         neg_holistic  = Neg_Holist,
         neg_length    = Neg_Lengte,
         neg_other     = Neg_Overige) %>%
  mutate(ID_productF = factor(ID_product,
                              levels = c("A", "B", "C", "D", "E", "F"),
                              labels = paste0("product ",
                                              c("A", "B", "C", "D", "E", "F")),
                              ordered = TRUE))

Recoding of variable ID_productF is checked.

           
             A  B  C  D  E  F
  product A 11  0  0  0  0  0
  product B  0 11  0  0  0  0
  product C  0  0 11  0  0  0
  product D  0  0  0 11  0  0
  product E  0  0  0  0 11  0
  product F  0  0  0  0  0 11

1.3.2 Factor analysis

Preliminary
An exploratory factor analysis with oblique rotation will be performed to examine the factor structure underlying the self-efficacy items. First, the suitability of factor analysis is checked by examining the correlations between all self-efficacy items. As can be seen, all items show several correlations of at least 0.30 with multiple other items. However, the items seff_language_3 and seff_language_4 are highly correlated (r = 0.92) which indicates that they are almost interchangable. Therefore, only item seff_language_4 is kept because it refers to writing grammatically correct sentences which was judged as more relevant in the context of this writing task than the item seff_language_3 (I can write a report without making spelling mistakes).

Then, the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is looked at. This measure is appropriate when cases to variable ratio is less than 1:5. KMO-values should be at least 0.50 (see Williams et al., 2010). Three items (seff_writing_1, seff_writing_2, seff_language_2) have an KMO-value smaller than 0.5. These items are also removed.

Item	KMO
I can discuss all the elements that should be included in reporting the results of a t-test (seff_content_1)	0.94
I can report all essential elements of a t-test in a logical order (seff_content_2)	0.73
I can correctly interpret the important elements of a t-test (seff_interpret_1)	0.69
I can check from output whether the assumptions are met to perform a t-test (seff_interpret_2)	0.69
I can check from output the statistical significance of a t-test (seff_interpret_3)	0.74
I can correctly interpret the effect size (seff_interpret_4)	0.79
I can use an objective writing style (seff_writing_1)	0.40
I can integrate statistical results in a consistent way in a report (seff_writing_2)	0.47
I can distinguish between what statistical information is relevant to report and what statistical information is not relevant to report (seff_writing_3)	0.55
I can report statistical information clearly and understandably to a reader with little statistical knowledge (seff_language_1)	0.51
I can use an active writing style (seff_language_2)	0.42
I can write a report without making spelling mistakes (seff_language_3)	0.58
I can write grammatically correct sentences (seff_language_4)	0.55

Below factor loadings are shown of an EFA with oblique rotation assuming one factor. A cut-off of 0.40 (absolute value) is used for item retention (Hair et al., 1998). The item seff_language_4 (I can write grammatically correct sentences) has a negative factor loading smaller than 0.40 and is removed. Next, a new EFA with oblique rotation is performed.

Items	Factor 1	Communalities
I can discuss all the elements that should be included in reporting the results of a t-test (seff_content_1)	0.92	0.84
I can report all essential elements of a t-test in a logical order (seff_content_2)	0.89	0.79
I can correctly interpret the important elements of a t-test (seff_interpret_1)	0.95	0.90
I can check from output whether the assumptions are met to perform a t-test (seff_interpret_2)	0.86	0.73
I can check from output the statistical significance of a t-test (seff_interpret_3)	0.88	0.78
I can correctly interpret the effect size (seff_interpret_4)	0.81	0.65
I can distinguish between what statistical information is relevant to report and what statistical information is not relevant to report (seff_writing_3)	0.62	0.39
I can report statistical information clearly and understandably to a reader with little statistical knowledge (seff_language_1)	0.43	0.19
I can write grammatically correct sentences (seff_language_4)	-0.32	0.10

Second (final) factor analysis
After removal of the item seff_language_4 all criteria point again to one factor.

Now, all items have a factor loading of at least 0.40. These items explain 66% of the total variance. Next, the internal consistency of the items is checked and found to be high (\(\alpha\) = 0.93).

Items	Factor 1	Communalities
I can discuss all the elements that should be included in reporting the results of a t-test (seff_content_1)	0.91	0.84
I can report all essential elements of a t-test in a logical order (seff_content_2)	0.89	0.79
I can correctly interpret the important elements of a t-test (seff_interpret_1)	0.95	0.90
I can check from output whether the assumptions are met to perform a t-test (seff_interpret_2)	0.86	0.74
I can check from output the statistical significance of a t-test (seff_interpret_3)	0.88	0.78
I can correctly interpret the effect size (seff_interpret_4)	0.81	0.65
I can distinguish between what statistical information is relevant to report and what statistical information is not relevant to report (seff_writing_3)	0.63	0.40
I can report statistical information clearly and understandably to a reader with little statistical knowledge (seff_language_1)	0.44	0.19

1.3.3 Create variables

Prior knowledge and self-efficacy
The variable ‘self-efficacy’ is created by summing the 8 self-efficacy items and dividing them by 8 (the total number of items). The scores on the five variables measuring prior knowledge (items prior_know_1 tot prior_know_5) are summed as well. For the purpose of analysis, self-efficacy and prior knowledge are standardized.

pretest_C <- 
  pretest_C %>%
  rowwise() %>%
  mutate(self_efficacy = sum(seff_content_1,   seff_content_2,
                             seff_interpret_1, seff_interpret_2,
                             seff_interpret_3, seff_interpret_4,
                             seff_writing_3,   seff_language_1) / 8,
         prior_knowledge = sum(prior_know_1, prior_know_2, 
                               prior_know_3, prior_know_4, 
                               prior_know_5)) %>%
  ungroup() %>%
  mutate(self_efficacyZ = as.numeric(scale(self_efficacy)),
         prior_knowledgeZ = as.numeric(scale(prior_knowledge)))

feedback_abs_C <- 
  feedback_abs_C %>%
  mutate(pos_contentD   = ifelse(pos_content >=1,   yes = 1, no = 0),
         pos_interpretD = ifelse(pos_interpret >=1, yes = 1, no = 0), 
         pos_styleD     = ifelse(pos_style >=1,     yes = 1, no = 0),
         pos_languageD  = ifelse(pos_language >=1,  yes = 1, no = 0),
         pos_holisticD  = ifelse(pos_holistic >=1,  yes = 1, no = 0),
         pos_lengthD    = ifelse(pos_length >=1,    yes = 1, no = 0),
         pos_otherD     = ifelse(pos_other>=1,      yes = 1, no = 0),
         neg_contentD   = ifelse(neg_content >=1,   yes = 1, no = 0),
         neg_interpretD = ifelse(neg_interpret >=1, yes = 1, no = 0), 
         neg_styleD     = ifelse(neg_style >=1,     yes = 1, no = 0),
         neg_languageD  = ifelse(neg_language >=1,  yes = 1, no = 0),
         neg_holisticD  = ifelse(neg_holistic >=1,  yes = 1, no = 0),
         neg_lengthD    = ifelse(neg_length >=1,    yes = 1, no = 0),
         neg_otherD     = ifelse(neg_other>=1,      yes = 1, no = 0)) %>%
  rowwise() %>%
  mutate(pos_total = sum(pos_contentD,  pos_interpretD, pos_styleD,
                         pos_languageD, pos_holisticD,  pos_lengthD,
                         pos_otherD),
         neg_total = sum(neg_contentD,  neg_interpretD, neg_styleD,
                         neg_languageD, neg_holisticD,  neg_lengthD,
                         neg_otherD),
         total     = pos_total + neg_total) %>%
  ungroup()

First, this is done for the feedback of the absolute condition. Recoding of the dummy variables is checked.

Check recoding of dummy variable 'pos_contentD'

   
     0  1  2  3  4
  0 28  0  0  0  0
  1  0 35 11  2  2

Check recoding of dummy variable 'pos_interpretD'

Check recoding of dummy variable 'pos_styleD'

Check recoding of dummy variable 'pos_languageD'

   
     0  1  2  3  4
  0 58  0  0  0  0
  1  0 11  3  4  2

Check recoding of dummy variable 'pos_holisticD'

   
     0  1  2  3
  0 68  0  0  0
  1  0  7  1  2

Check recoding of dummy variable 'pos_lengthD'

Check recoding of dummy variable 'pos_otherD'

Check recoding of dummy variable 'neg_contentD'

   
     0  1  2  3
  0 28  0  0  0
  1  0 33 11  6

Check recoding of dummy variable 'neg_interpretD'

Check recoding of dummy variable 'neg_styleD'

Check recoding of dummy variable 'neg_languageD'

   
     0  1  2  3
  0 49  0  0  0
  1  0 17 11  1

Check recoding of dummy variable 'neg_holisticD'

Check recoding of dummy variable 'neg_lengthD'

Check recoding of dummy variable 'neg_otherD'

Then, the same is done for the feedback of the students in the criteria condition.

feedback_comp_C <- 
  feedback_comp_C %>%
  mutate(pos_contentD   = ifelse(pos_content >=1,   yes = 1, no = 0),
         pos_interpretD = ifelse(pos_interpret >=1, yes = 1, no = 0), 
         pos_styleD     = ifelse(pos_style >=1,     yes = 1, no = 0),
         pos_languageD  = ifelse(pos_language >=1,  yes = 1, no = 0),
         pos_holisticD  = ifelse(pos_holistic >=1,  yes = 1, no = 0),
         pos_lengthD    = ifelse(pos_length >=1,    yes = 1, no = 0),
         pos_otherD     = ifelse(pos_other>=1,      yes = 1, no = 0),
         neg_contentD   = ifelse(neg_content >=1,   yes = 1, no = 0),
         neg_interpretD = ifelse(neg_interpret >=1, yes = 1, no = 0), 
         neg_styleD     = ifelse(neg_style >=1,     yes = 1, no = 0),
         neg_languageD  = ifelse(neg_language >=1,  yes = 1, no = 0),
         neg_holisticD  = ifelse(neg_holistic >=1,  yes = 1, no = 0),
         neg_lengthD    = ifelse(neg_length >=1,    yes = 1, no = 0),
         neg_otherD     = ifelse(neg_other>=1,      yes = 1, no = 0)) %>%
  rowwise() %>%
  mutate(pos_total = sum(pos_contentD,  pos_interpretD, pos_styleD,
                         pos_languageD, pos_holisticD,  pos_lengthD,
                         pos_otherD),
         neg_total = sum(neg_contentD,  neg_interpretD, neg_styleD,
                         neg_languageD, neg_holisticD,  neg_lengthD,
                         neg_otherD),
         total     = pos_total + neg_total) %>%
  ungroup()

Recoding of the dummy variables is checked.

Check recoding of dummy variable 'pos_contentD'

   
     0  1  2  3
  0 29  0  0  0
  1  0 27  8  2

Check recoding of dummy variable 'pos_interpretD'

Check recoding of dummy variable 'pos_styleD'

Check recoding of dummy variable 'pos_languageD'

Check recoding of dummy variable 'pos_holisticD'

Check recoding of dummy variable 'pos_lengthD'

Check recoding of dummy variable 'pos_otherD'

Check recoding of dummy variable 'neg_contentD'

   
     0  1  2  3  4
  0 34  0  0  0  0
  1  0 20 10  1  1

Check recoding of dummy variable 'neg_interpretD'

Check recoding of dummy variable 'neg_styleD'

Check recoding of dummy variable 'neg_languageD'

Check recoding of dummy variable 'neg_holisticD'

Check recoding of dummy variable 'neg_lengthD'

Check recoding of dummy variable 'neg_otherD'

Then, both files on feedback are merged. The variable condition is recoded into a factor.

feedback_C <- 
  bind_rows(feedback_abs_C, feedback_comp_C) %>%
  mutate(conditionF = factor(condition,
                             levels = c(0, 1),
                             labels = c("absolute", "comparative"),
                             ordered = TRUE))

Check recoding of variable conditionF.

             
               0  1
  absolute    78  0
  comparative  0 66

Writing performance
The variable that maps students’ writing performance results from the assessment set up in Comproved. This variable is also standardised.

posttest_C  <- 
  posttest_C %>%
  mutate(abilityZ = as.numeric(scale(ability)))

1.3.4 Create data for analysis

Two data sets are created (one for each research question).

Research question 1 (RQ1)
For RQ1, this data set should contain the following variables:
- ID of the student (variable ID_student)
- ID of the product (variable ID_product)
- condition (variable condition)
- number of positive feedback statements (variable pos_total)
- number of negative feedback statements (variable neg_total)
- total number of feedback statements (variable total)
- dummy variables on negative feedback per category (variables neg_vocabularyD, neg_contentD, neg_grammarD, neg_syntaxDand neg_spellingD)
- dummy variables on negative feedback per category (variables pos_vocabularyD, pos_contentD, pos_grammarD, pos_syntaxDand pos_spellingD)

data_RQ1_C <- 
  feedback_C %>%
  select(ID_student, contains("product"), contains("condition"),
         pos_total,  neg_total,  total,
         ends_with("D"))

Research question 2 (RQ2)
For RQ2, the data set should contain the following variables:
- ID student (variable ID_student)
- condition (variables condition and conditionF)
- gender (variables gender and genderF) - participation in session (variables session1 and session1F)
- participation in optional writing task 1 (variables writing_task1 and writing_task1F)
- processing of learning materials of session (variables processed and processedF)

self-efficacy at pre-test (variables self_efficacy and self_efficacyZ)
prior knowledge (variables prior_knowledge and prior_knowledgeZ)
writing ability at post-test (variables ability and abilityZ)

data_RQ2_C <- 
  inner_join(pretest_C, posttest_C, by = "ID_student") %>%
  select(ID_student, 
         contains("condition"),
         contains("gender"),
         contains("session"),
         contains("writing_task1"),
         contains("processed"),
         contains("prior_knowledge"), 
         contains("self_efficacy"),
         contains("ability"))

Both data sets are saved. These data sets can be found here.

save(data_RQ1_C, file = here("01_Data", "02_Processed data", "Sample C", 
                             "data_RQ1_C.RData"))
save(data_RQ2_C, file = here("01_Data", "02_Processed data", "Sample C",
                             "data_RQ2_C.RData"))

2 Descriptive statistics

This code was used to create the tables with the descriptive statistics of the independent variables (Table 2), the dependent variables (Table 3 and Table 4) and the plot on the distribution of the amount of arguments per judgement (Figure 2).

2.1 Create Table 4.2

Descriptives of prior knowledge for sample A

#' Create list for sample A with descriptives of prior knowledge
prior_A <- list(min = min(data_RQ2_A$prior_knowledge), 
                max = max(data_RQ2_A$prior_knowledge),
                mean = mean(data_RQ2_A$prior_knowledge),
                SD = sd(data_RQ2_A$prior_knowledge))

Descriptives of self-efficacy for sample A

#' Create list for sample A with descriptives of self-efficacy
self_eff_A <- list(min_mastery = min(data_RQ2_A$self_eff_mastery), 
                   max_mastery = max(data_RQ2_A$self_eff_mastery),
                   mean_mastery = mean(data_RQ2_A$self_eff_mastery),
                   SD_mastery = sd(data_RQ2_A$self_eff_mastery),
                   min_persuasion = min(data_RQ2_A$self_eff_persuasion), 
                   max_persuasion = max(data_RQ2_A$self_eff_persuasion),
                   mean_persuasion = mean(data_RQ2_A$self_eff_persuasion),
                   SD_persuasion = sd(data_RQ2_A$self_eff_persuasion),
                   min_state = min(data_RQ2_A$self_eff_state), 
                   max_state = max(data_RQ2_A$self_eff_state),
                   mean_state = mean(data_RQ2_A$self_eff_state),
                   SD_state = sd(data_RQ2_A$self_eff_state))

Descriptives of performance for sample A

#' Create list for sample A with descriptives of performance
performance_A <- list(min = min(data_RQ2_A$ability), 
                      max = max(data_RQ2_A$ability),
                      mean = mean(data_RQ2_A$ability),
                      SD = sd(data_RQ2_A$ability))

Create and save table 4.2 (table 4.2 is saved as xlsx-file and can be found here)

#' Create table 4.2
table_descript1 <- 
  tibble(variable  = c("Prior knowledge", 
                       "S-E mastery", 
                       "S-E persuasion", 
                       "S-E state",
                       "Performance"),
         min_A = c(prior_A$min, 
                   self_eff_A$min_mastery, self_eff_A$min_persuasion, 
                   self_eff_A$min_state,
                   performance_A$min),
         max_A = c(prior_A$max, 
                   self_eff_A$max_mastery, self_eff_A$max_persuasion, 
                   self_eff_A$max_state,
                   performance_A$max),
         mean_A = c(prior_A$mean, 
                    self_eff_A$mean_mastery, self_eff_A$mean_persuasion, 
                    self_eff_A$mean_state,
                    performance_A$mean),
         sd_A = c(prior_A$SD, 
                  self_eff_A$SD_mastery, self_eff_A$SD_persuasion, 
                  self_eff_A$SD_state,
                  performance_A$SD)) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate_all(str_remove, "NA") 

#' Save table 4.2 
openxlsx::write.xlsx(table_descript1, "04_Tables/Table 4.2.xlsx")

Variable	Sample A (N = 78)
	Min.	Max.	M	SD
Prior knowledge	0.00	8.00	5.42	1.78
S-E mastery	2.25	6.00	4.56	0.73
S-E persuasion	1.00	5.83	2.78	1.11
S-E state	2.00	6.00	5.46	0.72
Performance	7.00	16.00	13.42	1.96

2.2 Create Table 4.3

Descriptives of prior knowledge for samples B and C

#' Create list for sample B and C with descriptives of prior knowledge
prior_B <- list(min = min(data_RQ2_B$prior_knowledge), 
                max = max(data_RQ2_B$prior_knowledge),
                mean = mean(data_RQ2_B$prior_knowledge),
                SD = sd(data_RQ2_B$prior_knowledge))
prior_C <- list(min = min(data_RQ2_C$prior_knowledge), 
                max = max(data_RQ2_C$prior_knowledge),
                mean = mean(data_RQ2_C$prior_knowledge),
                SD = sd(data_RQ2_C$prior_knowledge))

Descriptives of self-efficacy for samples B and C

#' Create list for samples B and C with descriptives of self-efficacy
self_eff_B <- list(min = min(data_RQ2_B$self_efficacy), 
                   max = max(data_RQ2_B$self_efficacy),
                   mean = mean(data_RQ2_B$self_efficacy),
                   SD = sd(data_RQ2_B$self_efficacy))
self_eff_C <- list(min = min(data_RQ2_C$self_efficacy), 
                   max = max(data_RQ2_C$self_efficacy),
                   mean = mean(data_RQ2_C$self_efficacy),
                   SD = sd(data_RQ2_C$self_efficacy))

Descriptives of performance for samples B and C

#' Create list for samples B and C with descriptives of performance
performance_B <- list(min = min(data_RQ2_B$ability), 
                      max = max(data_RQ2_B$ability),
                      mean = mean(data_RQ2_B$ability),
                      SD = sd(data_RQ2_B$ability))
performance_C <- list(min = min(data_RQ2_C$ability), 
                      max = max(data_RQ2_C$ability),
                      mean = mean(data_RQ2_C$ability),
                      SD = sd(data_RQ2_C$ability))

Create and save table 4.3 (table 4.3 is saved as xlsx-file and can be found here)

#' Create table 4.3
table_descript2 <- 
  tibble(variable  = c("Prior knowledge", 
                       "Self-efficacy (S-E)", 
                       "Performance"),
         min_B = c(prior_B$min, self_eff_B$min, performance_B$min),
         max_B = c(prior_B$max, self_eff_B$max, performance_B$max),
         mean_B = c(prior_B$mean, self_eff_B$mean, performance_B$mean),
         sd_B = c(prior_B$SD, self_eff_B$SD, performance_B$SD),
         min_C = c(prior_C$min, self_eff_C$min, performance_C$min),
         max_C = c(prior_C$max, self_eff_C$max, performance_C$max),
         mean_C = c(prior_C$mean, self_eff_C$mean, performance_C$mean),
         sd_C = c(prior_C$SD, self_eff_C$SD, performance_C$SD)) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate_all(str_remove, "NA") 

#' Save table 4.3 
openxlsx::write.xlsx(table_descript2, "04_Tables/Table 4.3.xlsx")

Variable	Sample B (N = 42)				Sample C (N = 22)
	Min.	Max.	M	SD	Min.	Max.	M	SD
Prior knowledge	0.00	7.00	2.62	1.90	0.20	7.75	4.88	1.83
Self-efficacy (S-E)	14.64	90.64	53.50	18.39	10.50	76.75	50.39	19.07
Performance	-3.34	3.75	-0.02	1.85	-5.20	3.24	-0.19	2.00

2.3 Create Table 4.4

Descriptives of amount of feedback for samples B and C

#' Create list for samples B and C with descriptives of total amount of feedback
amount_tot_B <- list(min = min(data_RQ1_B$total), 
                     max = max(data_RQ1_B$total),
                     mean = mean(data_RQ1_B$total),
                     SD = sd(data_RQ1_B$total))
amount_tot_C <- list(min = min(data_RQ1_C$total), 
                     max = max(data_RQ1_C$total),
                     mean = mean(data_RQ1_C$total),
                     SD = sd(data_RQ1_C$total))

Descriptives of amount of positive feedback for samples B and C

#' Create list for samples B and C with descriptives of amount of positive feedback
amount_pos_B <- list(min = min(data_RQ1_B$pos_total), 
                     max = max(data_RQ1_B$pos_total),
                     mean = mean(data_RQ1_B$pos_total),
                     SD = sd(data_RQ1_B$pos_total))
amount_pos_C <- list(min = min(data_RQ1_C$pos_total), 
                     max = max(data_RQ1_C$pos_total),
                     mean = mean(data_RQ1_C$pos_total),
                     SD = sd(data_RQ1_C$pos_total))

Descriptives of total amount of negative feedback for samples B and C

#' Create list for samples B and C with descriptives of amount of negative feedback
amount_neg_B <- list(min = min(data_RQ1_B$neg_total), 
                     max = max(data_RQ1_B$neg_total),
                     mean = mean(data_RQ1_B$neg_total),
                     SD = sd(data_RQ1_B$neg_total))
amount_neg_C <- list(min = min(data_RQ1_C$neg_total), 
                     max = max(data_RQ1_C$neg_total),
                     mean = mean(data_RQ1_C$neg_total),
                     SD = sd(data_RQ1_C$neg_total))

Create and save table 4.4 (table 4.4 is saved as xlsx-file and can be found here)

#' Create table 4.4
table_descript3 <- 
  tibble(variable  = c("Amount of feedback", 
                       "Amount of positive feedback",
                       "Amount of negative feedback"),
         min_B = c(amount_tot_B$min, amount_pos_B$min, amount_neg_B$min),
         max_B = c(amount_tot_B$max, amount_pos_B$max, amount_neg_B$max),
         mean_B = c(amount_tot_B$mean, amount_pos_B$mean, amount_neg_B$mean),
         sd_B = c(amount_tot_B$SD, amount_pos_B$SD, amount_neg_B$SD),
         min_C = c(amount_tot_C$min, amount_pos_C$min, amount_neg_C$min),
         max_C = c(amount_tot_C$max, amount_pos_C$max, amount_neg_C$max),
         mean_C = c(amount_tot_C$mean, amount_pos_C$mean, amount_neg_C$mean),
         sd_C = c(amount_tot_C$SD, amount_pos_C$SD, amount_neg_C$SD)) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate_all(str_remove, "NA") 

#' Save table 4.4  
openxlsx::write.xlsx(table_descript3, "04_Tables/Table 4.4.xlsx")

Variable	Sample B				Sample C
	Min.	Max.	M	SD	Min.	Max.	M	SD
Amount of feedback	0.00	5.00	1.72	1.47	1.00	7.00	3.15	1.21
Amount of positive feedback	0.00	5.00	0.80	1.03	0.00	5.00	1.62	0.88
Amount of negative feedback	0.00	4.00	0.93	0.99	0.00	4.00	1.53	0.95

2.4 Create Table 4.5

Descriptives of content of positive and negative feedback for sample B

#' Create data set per category with absolute and relative frequencies 
content_B <- tibble(N_pos = data_RQ1_B %>% filter(pos_contentD==1) %>%nrow(),
                    rel_pos = round(N_pos / (data_RQ1_B %>% nrow()) *100, 2),
                    N_neg = data_RQ1_B %>% filter(neg_contentD==1) %>%nrow(),
                    rel_neg = round(N_neg / (data_RQ1_B %>% nrow()) *100, 2))
syntax_B <- tibble(N_pos = data_RQ1_B %>% filter(pos_syntaxD==1) %>%nrow(),
                    rel_pos = round(N_pos / (data_RQ1_B %>% nrow()) *100, 2),
                    N_neg = data_RQ1_B %>% filter(neg_syntaxD==1) %>%nrow(),
                    rel_neg = round(N_neg / (data_RQ1_B %>% nrow()) *100, 2))
grammar_B <- tibble(N_pos = data_RQ1_B %>% filter(pos_grammarD==1) %>%nrow(),
                    rel_pos = round(N_pos / (data_RQ1_B %>% nrow()) *100, 2),
                    N_neg = data_RQ1_B %>% filter(neg_grammarD==1) %>%nrow(),
                    rel_neg = round(N_neg / (data_RQ1_B %>% nrow()) *100, 2))
voca_B <- tibble(N_pos = data_RQ1_B %>% filter(pos_vocabularyD==1) %>%nrow(),
                 rel_pos = round(N_pos / (data_RQ1_B %>% nrow()) *100, 2),
                 N_neg = data_RQ1_B %>% filter(neg_vocabularyD==1) %>%nrow(),
                 rel_neg = round(N_neg / (data_RQ1_B %>% nrow()) *100, 2))
spelling_B <- tibble(N_pos = data_RQ1_B %>% filter(pos_spellingD==1) %>%nrow(),
                     rel_pos = round(N_pos / (data_RQ1_B %>% nrow()) *100, 2),
                     N_neg = data_RQ1_B %>% filter(neg_spellingD==1) %>%nrow(),
                     rel_neg = round(N_neg / (data_RQ1_B %>% nrow()) *100, 2))

#' Bind all data sets
descriptives_content_B <- 
  rbind(content_B, syntax_B, grammar_B, voca_B, spelling_B) %>%
  add_case(.after = 5) %>%
  add_case(.after = 5) %>%
  mutate(Variable1 = c("Content", "Syntax", "Grammar", "Vocabulary", "Spelling", " ", " ")) %>%
  relocate(Variable1)

Descriptives of content of positive and negative feedback for sample C

#' Create data set per category with absolute and relative frequencies 
content_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_contentD==1) %>%nrow(),
                    rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                    N_neg = data_RQ1_C %>% filter(neg_contentD==1) %>%nrow(),
                    rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
interpret_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_interpretD==2) %>%nrow(),
                      rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                      N_neg = data_RQ1_C %>% filter(neg_interpretD==1) %>%nrow(),
                      rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
style_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_styleD==1) %>%nrow(),
                  rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                  N_neg = data_RQ1_C %>% filter(neg_styleD==1) %>%nrow(),
                  rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
language_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_languageD==1) %>%nrow(),
                     rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                     N_neg = data_RQ1_C %>% filter(neg_languageD==1) %>%nrow(),
                     rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
holistic_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_holisticD==1) %>%nrow(),
                     rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                     N_neg = data_RQ1_C %>% filter(neg_holisticD==1) %>%nrow(),
                     rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
length_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_lengthD==1) %>%nrow(),
                    rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                    N_neg = data_RQ1_C %>% filter(neg_lengthD==1) %>%nrow(),
                    rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))
other_C <- tibble(N_pos = data_RQ1_C %>% filter(pos_otherD==1) %>%nrow(),
                  rel_pos = round(N_pos / (data_RQ1_C %>% nrow())*100, 2),
                  N_neg = data_RQ1_C %>% filter(neg_otherD==1) %>%nrow(),
                  rel_neg = round(N_neg / (data_RQ1_C %>% nrow())*100, 2))

#' Bind all data sets
descriptives_content_C <- 
  rbind(content_C, interpret_C, style_C, language_C, holistic_C, length_C,
        other_C) %>%
  mutate(Variable2 = c("Content", "Interpretation", "Style", "Language use",
                       "Holistic", "Length", "Other")) %>%
  mutate_all(as.character) %>%
  relocate(Variable2)

Create and save table 4.5 (table 4.5 is saved as xlsx-file and can be found here)

#' Create table 4.5
table_descript4 <- 
  cbind(descriptives_content_B, descriptives_content_C) 

#' Save table 4.5
openxlsx::write.xlsx(table_descript4, "04_Tables/Table 4.5.xlsx")

manual	Sample B				manual	Sample C
Variable	Positive		Negative		Variable	Positive		Negative
	N	%	N	%		N	%	N	%
Content	59	28.10	29	13.81	Content	87	60.42	82	56.94
Syntax	39	18.57	50	23.81	Interpretation	0	0	30	20.83
Grammar	25	11.90	45	21.43	Style	36	25	39	27.08
Vocabulary	31	14.76	33	15.71	Language use	39	27.08	43	29.86
Spelling	10	4.76	32	15.24	Holistic	19	13.19	4	2.78
					Length	21	14.58	15	10.42
					Other	6	4.17	8	5.56

2.5 Create Figure 4.2

Create plot with distribution of number of arguments per judgement for sample B

#' Create labels with relative percentages
text_labels_B <-
  data_RQ1_B %>%
  mutate(totalF = factor(total,
                         levels = c(0:5),
                         labels = c("0", "1", "2", "3", "4", "5"),
                         ordered = TRUE)) %>%
  count(totalF, .drop = FALSE) %>% 
  mutate(rel = paste0(formatC(n / sum(n) * 100, format = "f", digits = 1), "%"),
         position = n + rep(-1.5, 6))


#' Create plot 
plot_total_B <- 
  data_RQ1_B %>%
  mutate(totalF = factor(total,
                         levels = c(0:5),
                         labels = c("0", "1", "2", "3", "4", "5"),
                         ordered = TRUE)) %>%
  count(totalF) %>%
  ggplot(aes(x = totalF, y = n)) +
  geom_col(fill = "#A9A9A9", alpha = .8) +
  geom_text(data = text_labels_B,
            aes(y = position,  label = rel), 
            color = "black", size = 4.5, family = "Arial") +
  scale_x_discrete(position = "top", 
                   name = "Number of arguments per judgement") +
  scale_y_continuous(breaks = seq(0, 50, 10), limits = c(0, 57)) +
  ggtitle("Sample B") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
       plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(.5, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x  = element_text(color = "black", size = 14, face = "bold"),
        axis.text.y  = element_blank(),
        axis.ticks   = element_blank(),
        axis.title.x = element_text(color = "black", size = 14, face = "bold", hjust = 0.3),
        axis.title.y = element_blank(),
        panel.border     = element_blank(),
        panel.spacing.x  = unit(units = "cm", .75),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "none")

Create plot with distribution of number of arguments per judgement for sample C

#' Create labels with relative percentages
text_labels_C <-
  data_RQ1_C %>%
  mutate(totalF = factor(total,
                         levels = c(0:7),
                         labels = c("0", "1", "2", "3", "4", "5", "6", "7"),
                         ordered = TRUE)) %>%
  count(totalF, .drop=F) %>% 
  mutate(rel = paste0(formatC(n / sum(n) * 100, format = "f", digits = 1), "%"),
         position = n + c(-5, rep(-1.5, 6), -1))

#' Create plot
plot_total_C <- 
  data_RQ1_C %>%
  mutate(totalF = factor(total,
                         levels = c(0:7),
                         labels = c("0", "1", "2", "3", "4", "5", "6", "7"),
                         ordered = TRUE)) %>%
  count(totalF, .drop=F) %>%
  ggplot(aes(x = totalF, y = n)) +
  geom_col(fill = "#A9A9A9", alpha = .8) +
  geom_text(data = text_labels_C,
            aes(y = position,  label = rel), 
            color = "black", size = 4.5, family = "Arial") +
  scale_x_discrete(position = "top", name = "Number of arguments per judgement") +
  scale_y_continuous(breaks = seq(0, 50, 10), limits = c(0, 57)) +
  ggtitle("Sample C") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(.5, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x  = element_text(color = "black", size = 14, face = "bold"),
        axis.text.y  = element_blank(),
        axis.ticks   = element_blank(),
        axis.title.x = element_text(color = "black", size = 14,  face = "bold", hjust = 0.15),
        axis.title.y = element_blank(),
        panel.border     = element_blank(),
        panel.spacing.x  = unit(units = "cm", .75),
        panel.grid.major = element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "none")

Assemble and save figure 4.2 (figure 4.2 is saved as png-file and can be found here)

#' Assemble figure 4.2
plot_amount <- 
  plot_total_B + plot_spacer() + plot_total_C + 
  plot_layout(nrow = 1, widths = c(1.1, .025, 1.35))

#' Save figure 4.2
png_file <- here("03_Figures", "Figure 4.2.png")

ggsave(
  png_file,
  width  = 11.6,
  height = 6,
  units  = "in",
  device = "png",
  dpi = 320
)


knitr::plot_crop(png_file)
rm(text_labels_B, text_labels_C, plot_total_B, plot_total_C, png_file)

3 Check randomisation

Before answering both research questions, randomization is checked for each sample.

3.1 Sample A

Randomisation is checked for gender, class, study track, prior knowledge and the four sources of self-efficacy. The table below provides an overview of the results (\(chi2\)-tests and t-tests) for sample A. Only psychological state (self-efficacy) differs between both conditions (d = 0.28) at pre-test with students in the criteria condition scoring lower (M = -0.14; SD = 0.97) than students in the comparative condition (M = 0.14; SD = 1.02).

#' chi2-tests
qual_A <- list(chisq.test(table(x = data_RQ2_A$conditionF, y = data_RQ2_A$genderF)),
               chisq.test(table(x = data_RQ2_A$conditionF, y = data_RQ2_A$ID_class)),
               chisq.test(table(x = data_RQ2_A$conditionF, y = data_RQ2_A$trackF)),
               cramerV(x = data_RQ2_A$conditionF, y = data_RQ2_A$genderF),
               cramerV(x = data_RQ2_A$conditionF, y = data_RQ2_A$ID_class),
               cramerV(x = data_RQ2_A$conditionF, y = data_RQ2_A$trackF))
names(qual_A) <- c("chi.gender", "chi.class", "chi.track", 
                   "V.gender", "V.class", "V.track")

#' t-tests
random_data_A <- data.frame(data_RQ2_A %>% select(conditionF, 
                                                  prior_knowledgeZ,
                                                  self_eff_masteryZ, 
                                                  self_eff_persuasionZ,
                                                  self_eff_stateZ))
quan_A <- list(t.test(prior_knowledgeZ ~ conditionF, data = data_RQ2_A, var.equal=F),
               t.test(self_eff_masteryZ ~ conditionF, data = data_RQ2_A, var.equal=F),
               t.test(self_eff_persuasionZ ~ conditionF, data = data_RQ2_A, var.equal=F),
               t.test(self_eff_stateZ ~ conditionF, data = data_RQ2_A, var.equal=F),
               cohen.d(x = random_data_A, group = "conditionF"))
names(quan_A) <- c("prior", "mastery", "persuasion", "state", "d")

Variable	Type of test	Statistic	df	p-value	Effectsize
Gender	chi²-test	0.05	1.00	0.82	0.05
Class	chi²-test	1.56	3.00	0.67	0.14
Study track	chi²-test	0.94	1.00	0.33	0.14
Prior knowledge	t-test	0.83	75.87	0.41	-0.19
Mastery experiences (self-efficacy)	t-test	0.00	75.96	1.00	0.00
Social persuasion (self-efficacy)	t-test	-0.49	74.69	0.63	0.11
Psychological state (self-efficacy)	t-test	-1.21	75.78	0.23	0.28

3.2 Sample B

In sample B, randomisation is checked for prior knowledge and self-efficacy (both standardized). The table below provides an overview of the results (t-tests) for sample B. Only self-efficacy differs between both conditions at pre-test with students in the criteria condition scoring higher (M = 0.11; SD = 1.04) than students in the comparative condition (M = -0.11; SD = 0.97).

#' t-tests
random_data_B <- data.frame(data_RQ2_B %>% select(conditionF,
                                                  prior_knowledgeZ,
                                                  self_efficacyZ))
quan_B <- list(t.test(prior_knowledgeZ ~ conditionF, data = data_RQ2_B, var.equal=F),
               t.test(self_efficacyZ ~ conditionF, data = data_RQ2_B, var.equal=F),
               cohen.d(x = random_data_B, group = "conditionF"))
names(quan_B) <- c("prior", "efficacy", "d")

Variable	Type of test	Statistic	df	p-value	Effectsize
Prior knowledge	t-test	0.48	39.96	0.63	0
Self-efficacy	t-test	0.70	39.79	0.49	0

3.3 Sample C

Randomisation in sample C is checked for gender, whether or not a student participated in the first statistics session (session1F), whether or not a student handed in the optional writing task (writing_task1F), whether or not the student processed the learning materials covered in the first class (processedF), prior knowledge and self-efficacy. The table below provides an overview of the results (\(chi2\)-tests, t-tests) for sample C. Results indicate that there are no difference between conditions in any of the variables.

#' chi2-tests
qual_C <- list(chisq.test(table(x = data_RQ2_C$conditionF, y = data_RQ2_C$genderF)),
               chisq.test(table(x = data_RQ2_C$conditionF, y = data_RQ2_C$session1F)),
               chisq.test(table(x = data_RQ2_C$conditionF, y = data_RQ2_C$writing_task1F)),
               chisq.test(table(x = data_RQ2_C$conditionF, y = data_RQ2_C$processedF)),
               cramerV(x = data_RQ2_C$conditionF, y = data_RQ2_C$genderF),
               cramerV(x = data_RQ2_C$conditionF, y = data_RQ2_C$session1F),
               cramerV(x = data_RQ2_C$conditionF, y = data_RQ2_C$writing_task1F),
               cramerV(x = data_RQ2_C$conditionF, y = data_RQ2_C$processedF))
names(qual_C) <- c("chi.gender", "chi.session", "chi.task", "chi.processed", 
                   "V.gender", "V.session", "V.task", "V.processed")
#' t-tests
random_data_C <- data.frame(data_RQ2_C %>% select(conditionF,
                                                  prior_knowledgeZ,
                                                  self_efficacyZ))
quan_C <- list(t.test(prior_knowledgeZ ~ conditionF, data = data_RQ2_C, var.equal=F),
               t.test(self_efficacyZ ~ conditionF, data = data_RQ2_C, var.equal=F),
               cohen.d(x = random_data_C, group = "conditionF"))
names(quan_C) <- c("prior", "efficacy", "d")

Variable	Type of test	Statistic	df	p-value	Effectsize
Gender	chi²-test	0.57	1.00	0.45	0.28
Participation in first session	chi²-test	0.37	1.00	0.54	0.29
Handed in optional writing task	chi²-test	0.00	1.00	1.00	0.04
Processed learning materials	chi²-test	0.00	1.00	1.00	0.02
Prior knowledge	t-test	0.43	19.97	0.67	0.00
Self-efficacy	t-test	-0.47	18.58	0.64	0.00

4 Quality of feedback (RQ1)

This section presents the analyses that were performed to answer RQ1 on the effect of peer assessment method on the quantity and content of the feedback.

4.1 Quantity of feedback

To look at the effect on the quantity of the feedback, three generalized mixed-effect models (Poisson-distribution with log-link) are fitted for sample B and sample C (one for the total amount of feedback, one for the amount of positive feedback and one for the amount of negative feedback). Furthermore, 95% bootstrapped confidence intervals and two effect sizes (marginal and conditional \(R^2\)-statistic) are estimated. All fitted models and other R-output (confidence intervals, \(R^2\)-statistics) are saved and can be found here.

4.1.1 Estimate models and 95% CI’s

Estimate mixed-effect models on total amount of feedback provided for sample B and sample C

#' Fit models
amount_B <- glmer(total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                  data = data_RQ1_B,
                  family = poisson(),
                  nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

amount_C <- glmer(total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                  data = data_RQ1_C,
                  family = poisson(),
                  nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

#' Save models
save(amount_B, file = here("02_Models", "Quantity of feedback (total)",
                           "amount_B.RData"))
save(amount_C, file = here("02_Models", "Quantity of feedback (total)",
                           "amount_C.RData"))

Bootstrap 95% confidence intervals for models on total amount of feedback provided for sample B and sample C

#' Bootstrapped 95% CI
CI_amount_B <- confint(amount_B, method = "boot", seed = 1980)
CI_amount_C <- confint(amount_C, method = "boot", seed = 1980)

#' Save CI
save(CI_amount_B, file = here("02_Models", "Quantity of feedback (total)",
                              "CI_amount_B.RData"))
save(CI_amount_C, file = here("02_Models", "Quantity of feedback (total)",
                              "CI_amount_C.RData"))

Estimate mixed-effect models on total amount of positive feedback provided for sample B and sample C

#' Fit models
pos_amount_B <- glmer(pos_total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_B,
                      family = poisson(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

pos_amount_C <- glmer(pos_total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_C,
                      family = poisson(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

#' Save models
save(pos_amount_B, file = here("02_Models", "Quantity of feedback (positive)",
                               "pos_amount_B.RData"))
save(pos_amount_C, file = here("02_Models", "Quantity of feedback (positive)",
                               "pos_amount_C.RData"))

Bootstrap 95% confidence intervals for models on total amount of positive feedback provided for sample B and sample C

#' Bootstrapped 95% CI
CI_pos_amount_B <- confint(pos_amount_B, method = "boot", seed = 1980)
CI_pos_amount_C <- confint(pos_amount_C, method = "boot", seed = 1980)

#' Save CI
save(CI_pos_amount_B, file = here("02_Models", "Quantity of feedback (positive)",
                                  "CI_pos_amount_B.RData"))
save(CI_pos_amount_C, file = here("02_Models", "Quantity of feedback (positive)",
                                  "CI_pos_amount_C.RData"))

Estimate mixed-effect models on total amount of negative feedback provided for sample B and sample C

#' Fit models
neg_amount_B <- glmer(neg_total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_B,
                      family = poisson(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

neg_amount_C <- glmer(neg_total ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_C,
                      family = poisson(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

#' Save models
save(neg_amount_B, file = here("02_Models", "Quantity of feedback (negative)",
                               "neg_amount_B.RData"))
save(neg_amount_C, file = here("02_Models", "Quantity of feedback (negative)",
                               "neg_amount_C.RData"))

Bootstrap 95% confidence intervals for models on total amount of negative feedback provided for sample B and sample C

#' Bootstrapped 95% CI
CI_neg_amount_B <- confint(neg_amount_B, method = "boot", seed = 1980)
CI_neg_amount_C <- confint(neg_amount_C, method = "boot", seed = 1980)

#' Save CI
save(CI_neg_amount_B, file = here("02_Models", "Quantity of feedback (negative)",
                                  "CI_neg_amount_B.RData"))
save(CI_neg_amount_C, file = here("02_Models", "Quantity of feedback (negative)",
                                  "CI_neg_amount_C.RData"))

4.1.2 Estimate R²-statistics

Estimate marginal and condition \(R^2\)-statistics for total amount of feedback provided for sample B and sample C

#' Estimate R2-statistics
R2_amount_B <- r.squaredGLMM(amount_B)
R2_amount_C <- r.squaredGLMM(amount_C)

#' Save R2-statistics
save(R2_amount_B, file = here("02_Models", "Quantity of feedback (total)",
                              "R2_amount_B.RData"))
save(R2_amount_C, file = here("02_Models", "Quantity of feedback (total)",
                              "R2_amount_C.RData"))

Estimate marginal and condition \(R^2\)-statistics for total amount of positive feedback provided for sample B and sample C

#' Estimate R2-statistics
R2_pos_amount_B <- r.squaredGLMM(pos_amount_B)
R2_pos_amount_C <- r.squaredGLMM(pos_amount_C)

#' Save R2-statistics
save(R2_pos_amount_B, file = here("02_Models", "Quantity of feedback (positive)",
                                  "R2_pos_amount_B.RData"))
save(R2_pos_amount_C, file = here("02_Models", "Quantity of feedback (positive)",
                                  "R2_pos_amount_C.RData"))

Estimate marginal and condition \(R^2\)-statistics for total amount of negative feedback provided for sample B and sample C

#' Estimate R2-statistics
R2_neg_amount_B <- r.squaredGLMM(neg_amount_B)
R2_neg_amount_C <- r.squaredGLMM(neg_amount_C)  

#' Save R2-statistics
save(R2_neg_amount_B, file = here("02_Models", "Quantity of feedback (negative)",
                                  "R2_neg_amount_B.RData"))
save(R2_neg_amount_C, file = here("02_Models", "Quantity of feedback (negative)",
                                  "R2_neg_amount_C.RData"))

4.1.3 Create Table 4.10 (appendix)

Results on the total amount of feedback are tidied.

#' Tidy models
tidy_amount_B <- 
  amount_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B") 

tidy_amount_C <- 
  amount_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2)  %>%
  mutate(sample = "C")

#' Tidy 95% CI 
tidy_CI_amount_B <-
  as_tibble(CI_amount_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B")

tidy_CI_amount_C <-
  as_tibble(CI_amount_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C")

#' Tidy R2-statistics
tidy_R2_amount <-
  as_tibble(rbind(R2_amount_B[2,],  R2_amount_C[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = c("B", "C"), type = rep("total", 2))

Results on the total amount of positive feedback are tidied.

#' Tidy models
tidy_pos_amount_B <- 
  pos_amount_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B")

tidy_pos_amount_C <- 
  pos_amount_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2)  %>%
  mutate(sample = "C")

#' Tidy 95% CI
tidy_CI_pos_amount_B <-
  as_tibble(CI_pos_amount_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B")

tidy_CI_pos_amount_C <-
  as_tibble(CI_pos_amount_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C")

#' Tidy R2-statistics
tidy_R2_pos_amount <-
  as_tibble(rbind(R2_pos_amount_B[2,],  R2_pos_amount_C[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = c("B", "C"), type = rep("positive", 2))

Results on the total amount of negative feedback are tidied.

#' Tidy models
tidy_neg_amount_B <- 
  neg_amount_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B")

tidy_neg_amount_C <- 
  neg_amount_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2)  %>%
  mutate(sample = "C")

#' Tidy 95% CI
tidy_CI_neg_amount_B <-
  as_tibble(CI_neg_amount_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B")

tidy_CI_neg_amount_C <-
  as_tibble(CI_neg_amount_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C")

#' Tidy R2-statistics
tidy_R2_neg_amount <-
  as_tibble(rbind(R2_neg_amount_B[2,],  R2_neg_amount_C[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = c("B", "C"), type = rep("negative", 2))

Create and save table 4.10 (table 4.10 is saved as xlsx-file and can be found here)

#' Bind results of sample B and sample C 
part1 <- bind_rows(tidy_amount_B, tidy_amount_C)
part2 <- bind_rows(tidy_CI_amount_B, tidy_CI_amount_C)
part3 <- bind_rows(tidy_pos_amount_B, tidy_pos_amount_C)
part4 <- bind_rows(tidy_CI_pos_amount_B, tidy_CI_pos_amount_C)
part5 <- bind_rows(tidy_neg_amount_B, tidy_neg_amount_C)
part6 <- bind_rows(tidy_CI_neg_amount_B, tidy_CI_neg_amount_C)
part7 <- bind_rows(tidy_R2_amount, tidy_R2_pos_amount, tidy_R2_neg_amount)

#' Bind results across type of feedback and add results on R2-statistics
table_amount <-
  bind_rows(left_join(part1, part2, by = "sample", suffix = c("_est", "_CI")),
            left_join(part3, part4, by = "sample", suffix = c("_est", "_CI")),
            left_join(part5, part6, by = "sample", suffix = c("_est", "_CI"))) %>%
  mutate(type = c("total", "total", "positive", "positive", "negative", "negative")) %>%
  relocate(sample, type,
           Intercept_est,     Intercept_CI, 
           Condition_est,     Condition_CI, 
           `SD students_est`, `SD students_CI`, 
           `SD products_est`, `SD products_CI`) %>%
  left_join(part7, by = c("sample", "type")) %>%
  rename("marg. R2" = "R2m", "cond. R2" = "R2c") %>%
  arrange(sample, desc(type))

#' Save table as xlsx-file
openxlsx::write.xlsx(table_amount, "04_Tables/Table 4.10.xlsx")

Sample	Type feedback	Intercept		Condition^*		Differences between students		Differences between products		Effectsizes
		Est.	95% CI	Est.	95% CI	SD	95% CI	SD	95% CI	marg. R²	cond. R²
B	total	0.10	-0.28\|0.60	0.33	-0.33\|0.93	0.93	0.64\|1.17	0.00	0.00\|0.14	0.02	0.69
B	positive	-1.01	-1.39\|-0.33	0.80	0.02\|1.44	0.99	0.64\|1.23	0.13	0.00\|0.29	0.09	0.63
B	negative	-0.27	-0.61\|0.23	0.06	-0.50\|0.55	0.69	0.43\|0.87	0.30	0.00\|0.50	0.00	0.46
C	total	1.25	1.13\|1.37	-0.23	-0.42\|-0.04	0.07	0.00\|0.18	0.00	0.00\|0.13	0.05	0.06
C	positive	0.52	0.35\|0.69	-0.08	-0.33\|0.18	0.06	0.00\|0.23	0.03	0.00\|0.19	0.00	0.01
C	negative	0.59	0.39\|0.80	-0.40	-0.65\|-0.11	0.00	0.00\|0.24	0.14	0.00\|0.28	0.07	0.10
Note:
^Criteria condition is reference category.*

4.1.4 Create Figure 4.3

Create plot with results on quantity of feedback for sample B

#' Tibble with results
results_amount_B <- 
  tibble(aspect  = factor(rep(c("Total", "Positive", "Negative"), 2),
                          levels = c("Total", "Positive", "Negative"),
                          ordered = TRUE),
         condition = factor(c(rep("Criteria condition", 3), rep("Comparative condition", 3)),
                            levels = c("Criteria condition", "Comparative condition"),
                            ordered = TRUE),
         relative = c(1, .4, 1, 1, .8, 1),  
         position = relative + .1,
         label = c("1", NA, "1", NA, NA, NA))

#' Create plot
plot_results_amount_B <- 
  results_amount_B %>%
  ggplot(aes(x = aspect, y = relative, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge") +
  geom_text(aes(y = position,  label = label, group = condition), 
            color = "black", size = 6, family = "Arial") +
  annotate(geom = "text", label = "0.4", x = 1.75, y = 0.5,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  annotate(geom = "text", label = "0.8", x = 2.25, y = 0.9,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 3.8)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample B") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        plot.margin = margin(3.5, 0, 0, 0, "mm"),
        axis.text.x = element_text(color = "black", size = 16.5, face = "bold"),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "bottom",
        legend.text = element_text(size = 10),
        legend.justification = c(-.075,0),
        legend.title = element_blank())

Create plot with results on quantity of feedback for sample C

#' Tibble with results
results_amount_C <- 
  tibble(aspect  = factor(rep(c("Total", "Positive", "Negative"), 2),
                          levels = c("Total", "Positive", "Negative"),
                          ordered = TRUE),
         condition = factor(c(rep("Criteria condition", 3), rep("Comparative condition", 3)),
                            levels = c("Criteria condition", "Comparative condition"),
                            ordered = TRUE),
         relative = c(3.5, 1.7, 1.8, 2.8, 1.7, 1.2),  
         position = relative + .1,
         label = c(NA, "1.7", NA, NA, "1.7", NA))

#' Create plots
plot_results_amount_C <- 
  results_amount_C %>%
  ggplot(aes(x = aspect, y = relative, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge") +
  geom_text(aes(y = position,  label = label, group = condition), 
            color = "black", size = 6, family = "Arial") +
  annotate(geom = "text", label = "3.5", x = 0.75, y = 3.6,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  annotate(geom = "text", label = "2.8", x = 1.25, y = 2.9,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  annotate(geom = "text", label = "1.8", x = 2.75, y = 1.9,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  annotate(geom = "text", label = "1.2", x = 3.25, y = 1.3,
           color = "black", size = 6, family = "Arial", fontface = "italic") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 3.8)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample C") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        plot.margin = margin(3.5, 0, 0, 0, "mm"),
        axis.text.x = element_textbox(color = "black", size = 16.5, face = "bold"),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "none")

Assemble and save figure 4.3 (figure 4.3 is saved as png-file and can be found here)

#' Assemble figure 4.3
plot_results_amount <- 
  plot_results_amount_B +  plot_spacer() + plot_results_amount_C + 
  plot_layout(nrow = 1, widths = c(1, .1, 1)) 

#' Save figure 3
png_file <- here("03_Figures", "Figure 4.3.png")

ggsave(
  png_file,
  width  = 11.6,
  height = 6,
  units  = "in",
  device = "png",
  dpi = 320
)

knitr::plot_crop(png_file)
rm(results_amount_B, results_amount_C,
   plot_results_amount_B, plot_results_amount_C, png_file)

4.2 Content of positive feedback

To look at the effect on the content of the positive feedback, five generalized mixed-effect models (binomial distribution with logit-link) are fitted for sample B (for content, syntax, grammar, vocabulary and spelling) and 7 models for sample C (for content, interpretation, style, language use, holistic judgement, length and other). Furthermore, 95% bootstrapped confidence intervals and two effect sizes (marginal and conditional \(R^2\)-statistic) are estimated. All fitted models and other R-output (confidence intervals, \(R^2\)-statistics) are saved and can be found here.

4.2.1 Estimate models and 95% CI’s

Estimate mixed-effect models on content of positive feedback for sample B

#' Fit models
pos_content_B <- glmer(pos_contentD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_B,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

pos_syntax_B <- glmer(pos_syntaxD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_B,
                      family = binomial(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

pos_grammar_B <- glmer(pos_grammarD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_B,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

pos_voca_B <- glmer(pos_vocabularyD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                    data = data_RQ1_B,
                    family = binomial(),
                    nAGQ   = 0,  
                    glmerControl(optimizer = "bobyqa",
                                 optCtrl   = list(maxfun = 1000000)))

pos_spelling_B <- glmer(pos_spellingD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_B,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

#' Save models
save(pos_content_B, file = here("02_Models", "Content of feedback (positive)",
                                "pos_content_B.RData"))
save(pos_syntax_B, file = here("02_Models", "Content of feedback (positive)",
                               "pos_syntax_B.RData"))
save(pos_grammar_B, file = here("02_Models", "Content of feedback (positive)",
                                "pos_grammar_B.RData"))
save(pos_voca_B, file = here("02_Models", "Content of feedback (positive)",
                             "pos_voca_B.RData"))
save(pos_spelling_B, file = here("02_Models", "Content of feedback (positive)",
                                 "pos_spelling_B.RData"))

Estimate mixed-effect models on content of positive feedback for sample C

#' Fit models
pos_content_C <- glmer(pos_contentD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_C,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

pos_interpret_C <- glmer(pos_interpretD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                         data = data_RQ1_C,
                         family = binomial(),
                         nAGQ   = 0,  
                         glmerControl(optimizer = "bobyqa",
                                      optCtrl   = list(maxfun = 1000000)))

pos_style_C <- glmer(pos_styleD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                     data = data_RQ1_C,
                     family = binomial(),
                     nAGQ   = 0,  
                     glmerControl(optimizer = "bobyqa",
                                  optCtrl   = list(maxfun = 1000000)))

pos_language_C <- glmer(pos_languageD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_C,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

pos_holistic_C <- glmer(pos_holisticD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_C,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

pos_length_C <- glmer(pos_lengthD ~ 1 + condition +
                          (1|ID_student) + (1|ID_product),
                      data = data_RQ1_C,
                      family = binomial(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

pos_other_C <- glmer(pos_otherD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                     data = data_RQ1_C,
                     family = binomial(),
                     nAGQ   = 0,  
                     glmerControl(optimizer = "bobyqa",
                                  optCtrl   = list(maxfun = 1000000)))

#' Save models
save(pos_content_C, file = here("02_Models", "Content of feedback (positive)",
                                "pos_content_C.RData"))
save(pos_interpret_C, file = here("02_Models", "Content of feedback (positive)",
                                  "pos_interpret_C.RData"))
save(pos_style_C, file = here("02_Models", "Content of feedback (positive)",
                              "pos_style_C.RData"))
save(pos_language_C, file = here("02_Models", "Content of feedback (positive)",
                                 "pos_language_C.RData"))
save(pos_holistic_C, file = here("02_Models", "Content of feedback (positive)",
                                 "pos_holistic_C.RData"))
save(pos_length_C, file = here("02_Models", "Content of feedback (positive)",
                               "pos_length_C.RData"))
save(pos_other_C, file = here("02_Models", "Content of feedback (positive)",
                              "pos_other_C.RData"))

Bootstrap 95% confidence intervals for models on content of positive feedback of sample B.

#' Bootstrapped 95% CI
CI_pos_content_B <- confint(pos_content_B, method = "boot", seed = 1980)
CI_pos_syntax_B <- confint(pos_syntax_B, method = "boot", seed = 1980)
CI_pos_grammar_B <- confint(pos_grammar_B, method = "boot", seed = 1980)
CI_pos_voca_B <- confint(pos_voca_B, method = "boot", seed = 1980)
CI_pos_spelling_B <- confint(pos_spelling_B, method = "boot", seed = 1980)

#' Save CI
save(CI_pos_content_B, file = here("02_Models", "Content of feedback (positive)",
                                   "CI_pos_content_B.RData"))
save(CI_pos_syntax_B, file = here("02_Models", "Content of feedback (positive)",
                                  "CI_pos_syntax_B.RData"))
save(CI_pos_grammar_B, file = here("02_Models", "Content of feedback (positive)",
                                  "CI_pos_grammar_B.RData"))
save(CI_pos_voca_B, file = here("02_Models", "Content of feedback (positive)",
                                "CI_pos_voca_B.RData"))
save(CI_pos_spelling_B, file = here("02_Models", "Content of feedback (positive)",
                                    "CI_pos_spelling_B.RData"))

Bootstrap 95% confidence intervals for models on content of positive feedback of sample C

#' Bootstrapped 95% CI
CI_pos_content_C <- confint(pos_content_C, method = "boot", seed = 1980)
CI_pos_interpret_C <- confint(pos_interpret_C, method = "boot", seed = 1980)
CI_pos_style_C <- confint(pos_style_C, method = "boot", seed = 1980)
CI_pos_language_C <- confint(pos_language_C, method = "boot", seed = 1980)
CI_pos_holistic_C <- confint(pos_holistic_C, method = "boot", seed = 1980)
CI_pos_length_C <- confint(pos_length_C, method = "boot", seed = 1980)
CI_pos_other_C <- confint(pos_other_C, method = "boot", seed = 1980)

#' Save CI
save(CI_pos_content_C, file = here("02_Models", "Content of feedback (positive)",
                                   "CI_pos_content_C.RData"))
save(CI_pos_interpret_C, file = here("02_Models", "Content of feedback (positive)",
                                     "CI_pos_interpret_C.RData"))
save(CI_pos_style_C, file = here("02_Models", "Content of feedback (positive)",
                                 "CI_pos_style_C.RData"))
save(CI_pos_language_C, file = here("02_Models", "Content of feedback (positive)",
                                    "CI_pos_language_C.RData"))
save(CI_pos_holistic_C, file = here("02_Models", "Content of feedback (positive)",
                                    "CI_pos_holistic_C.RData"))
save(CI_pos_length_C, file = here("02_Models", "Content of feedback (positive)",
                                   "CI_pos_length_C.RData"))
save(CI_pos_other_C, file = here("02_Models", "Content of feedback (positive)",
                                 "CI_pos_other_C.RData"))

4.2.2 Estimate R²-statistics

Estimate marginal and condition \(R^2\)-statistics on content of positive feedback for sample B

#' Estimate R2-statistics
R2_pos_content_B <- r.squaredGLMM(pos_content_B)
R2_pos_syntax_B <- r.squaredGLMM(pos_syntax_B)
R2_pos_grammar_B <- r.squaredGLMM(pos_grammar_B)
R2_pos_voca_B <- r.squaredGLMM(pos_voca_B)
R2_pos_spelling_B <- r.squaredGLMM(pos_spelling_B)

#' Save R2-statistics
save(R2_pos_content_B, file = here("02_Models", "Content of feedback (positive)",
                                   "R2_pos_content_B.RData"))
save(R2_pos_syntax_B, file = here("02_Models", "Content of feedback (positive)",
                                   "R2_pos_syntax_B.RData"))
save(R2_pos_grammar_B, file = here("02_Models", "Content of feedback (positive)",
                                   "R2_pos_grammar_B.RData"))
save(R2_pos_voca_B, file = here("02_Models", "Content of feedback (positive)",
                                "R2_pos_voca_B.RData"))
save(R2_pos_spelling_B, file = here("02_Models", "Content of feedback (positive)",
                                    "R2_pos_spelling_B.RData"))

Estimate marginal and condition \(R^2\)-statistics on content of positive feedback for sample C

#' Estimate R2-statistics
R2_pos_content_C <- r.squaredGLMM(pos_content_C)
R2_pos_interpret_C <- r.squaredGLMM(pos_interpret_C)
R2_pos_style_C <- r.squaredGLMM(pos_style_C)
R2_pos_language_C <- r.squaredGLMM(pos_language_C)
R2_pos_holistic_C <- r.squaredGLMM(pos_holistic_C)
R2_pos_length_C <- r.squaredGLMM(pos_length_C)
R2_pos_other_C <- r.squaredGLMM(pos_other_C)

#' Save R2-statistics
save(R2_pos_content_C, file = here("02_Models", "Content of feedback (positive)",
                                   "R2_pos_content_C.RData"))
save(R2_pos_interpret_C, file = here("02_Models", "Content of feedback (positive)",
                                     "R2_pos_interpret_C.RData"))
save(R2_pos_style_C, file = here("02_Models", "Content of feedback (positive)",
                                 "R2_pos_style_C.RData"))
save(R2_pos_language_C, file = here("02_Models", "Content of feedback (positive)",
                                    "R2_pos_language_C.RData"))
save(R2_pos_holistic_C, file = here("02_Models", "Content of feedback (positive)",
                                    "R2_pos_holistic_C.RData"))
save(R2_pos_length_C, file = here("02_Models", "Content of feedback (positive)",
                                   "R2_pos_length_C.RData"))
save(R2_pos_other_C, file = here("02_Models", "Content of feedback (positive)",
                                 "R2_pos_othrt_C.RData"))

4.2.3 Create Table 4.11 (appendix)

Results on the content of positive feedback for model B are tidied.

#' Tidy models
tidy_pos_content_B <- 
  pos_content_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Higher-order", category = "Content")

tidy_pos_syntax_B <- 
  pos_syntax_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Higher-order", category = "Syntax")

tidy_pos_grammar_B <- 
  pos_grammar_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Grammar")

tidy_pos_voca_B <- 
  pos_voca_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Vocabulary")

tidy_pos_spelling_B <- 
  pos_spelling_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Spelling")

#' Tidy 95% CI 
tidy_CI_pos_content_B <-
  as_tibble(CI_pos_content_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Higher-order", category = "Content")

tidy_CI_pos_syntax_B <-
  as_tibble(CI_pos_syntax_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Higher-order", category = "Syntax")

tidy_CI_pos_grammar_B <-
  as_tibble(CI_pos_grammar_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order", category = "Grammar")

tidy_CI_pos_voca_B <-
  as_tibble(CI_pos_voca_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order", category = "Vocabulary")

tidy_CI_pos_spelling_B <-
  as_tibble(CI_pos_spelling_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order" , category = "Spelling")

#' Tidy R2-statistics
tidy_R2_pos_content_B <-
  as_tibble(rbind(R2_pos_content_B[2,], R2_pos_syntax_B[2,],
            R2_pos_grammar_B[2,], R2_pos_voca_B[2,],   R2_pos_spelling_B[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = rep("B", 5), 
         type = c(rep("Higher-order", 2), rep("Lower-order" , 3)),
         category = c("Content", "Syntax", "Grammar", "Vocabulary", "Spelling"))

Results on the content of positive feedback for model C are tidied.

#' Tidy models
tidy_pos_content_C <- 
  pos_content_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Content")

tidy_pos_interpret_C <- 
  pos_interpret_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Interpretation")

tidy_pos_style_C <- 
  pos_style_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Scientific style")

tidy_pos_language_C <- 
  pos_language_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Language use")

tidy_pos_holistic_C <- 
  pos_holistic_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Holistic judgement")

tidy_pos_length_C <- 
  pos_length_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Length")

tidy_pos_other_C <- 
  pos_other_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Other")

#' Tidy 95% CI 
tidy_CI_pos_content_C <-
  as_tibble(CI_pos_content_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Content")

tidy_CI_pos_interpret_C <-
  as_tibble(CI_pos_interpret_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Interpretation")

tidy_CI_pos_style_C <-
  as_tibble(CI_pos_style_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Scientific style")

tidy_CI_pos_language_C <-
  as_tibble(CI_pos_language_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Language use")

tidy_CI_pos_holistic_C <-
  as_tibble(CI_pos_holistic_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Holistic judgement")

tidy_CI_pos_length_C <-
  as_tibble(CI_pos_length_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Length")

tidy_CI_pos_other_C <-
  as_tibble(CI_pos_other_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Other")

#' Tidy R2-statistics
tidy_R2_pos_content_C <-
  as_tibble(rbind(R2_pos_content_C[2,],  R2_pos_interpret_C[2,],
                  R2_pos_style_C[2,],    R2_pos_language_C[2,],  
                  R2_pos_holistic_C[2,], R2_pos_length_C[2,],
                  R2_pos_other_C[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = rep("C", 7), 
         type = c(rep("Higher-order", 3), rep("Lower-order" , 4)),
         category = c("Content", "Interpretation",
                      "Scientific style", "Language use",
                      "Holistic judgement", "Length", "Other"))

Create and save table 4.11 (table 4.11 is saved as xlsx-file and can be found here)

#' Bind results of sample B
part1 <- bind_rows(tidy_pos_content_B,  tidy_pos_syntax_B,
                   tidy_pos_grammar_B,  tidy_pos_voca_B,  
                   tidy_pos_spelling_B,
                   tidy_pos_content_C,  tidy_pos_interpret_C, 
                   tidy_pos_style_C,    tidy_pos_language_C,
                   tidy_pos_holistic_C, tidy_pos_length_C,
                   tidy_pos_other_C)

#' Bind results of sample C
part2 <- bind_rows(tidy_CI_pos_content_B,  tidy_CI_pos_syntax_B,
                   tidy_CI_pos_grammar_B,  tidy_CI_pos_voca_B,    
                   tidy_CI_pos_spelling_B,      
                   tidy_CI_pos_content_C,  tidy_CI_pos_interpret_C,
                   tidy_CI_pos_style_C,    tidy_CI_pos_language_C, 
                   tidy_CI_pos_holistic_C, tidy_CI_pos_length_C,  
                   tidy_CI_pos_other_C)

#' Bind results of sample B and sample C
part3 <-  bind_rows(tidy_R2_pos_content_B, tidy_R2_pos_content_C)


#' Bind results across content of feedback and add results on R2-statistics
table_content <-
  left_join(part1, part2, by = c("sample", "category", "type"),
            suffix = c("_est", "_CI")) %>%
  relocate(sample, type, category,
           Intercept_est,     Intercept_CI, 
           Condition_est,     Condition_CI, 
           `SD students_est`, `SD students_CI`, 
           `SD products_est`, `SD products_CI`) %>%
  left_join(part3, by = c("sample", "category", "type")) %>%
  rename("marg. R2" = "R2m", "cond. R2" = "R2c") %>%
  select(-type)

#' Save table as xlsx-file
openxlsx::write.xlsx(table_content, "04_Tables/Table 4.11.xlsx")

Sample	Content	Intercept		Condition<sup>*</sup>		Differences between students		Differences between products		Effectsizes
		Est.	95% CI	Est.	95% CI	SD	95% CI	SD	95% CI	marg. R²	cond. R²
B	Content	-1.50	-2.08\|-0.58	0.66	-0.53\|1.70	1.52	0.68\|2.24	0.00	0.00\|0.49	0.02	0.34
B	Syntax	-3.12	-4.65\|-1.76	2.17	0.87\|3.53	1.10	0.00\|1.80	0.87	0.00\|1.61	0.13	0.36
B	Grammar	-3.18	-3.77\|-1.53	0.97	-0.66\|2.26	2.16	0.87\|3.15	0.00	0.00\|0.62	0.02	0.43
B	Vocabulary	-2.96	-3.58\|-1.20	0.87	-0.83\|2.27	2.36	1.08\|3.36	0.42	0.00\|0.95	0.02	0.51
B	Spelling	-3.98	-22.57\|-2.64	1.38	-0.07\|19.95	0.95	0.00\|1.90	0.00	0.00\|0.81	0.03	0.08
C	Content	0.68	-0.16\|1.31	-0.41	-1.34\|0.73	1.17	0.30\|1.85	0.00	0.00\|0.59	0.01	0.25
C	Interpretation	-1.15	-1.92\|-0.41	-1.42	-3.13\|-0.22	0.92	0.00\|1.62	0.00	0.00\|0.64	0.07	0.18
C	Scientific style	-1.06	-1.86\|-0.28	-0.25	-1.21\|0.76	0.73	0.00\|1.38	0.49	0.00\|0.91	0.00	0.14
C	Language use	-1.12	-1.86\|-0.42	0.18	-0.70\|1.16	0.77	0.00\|1.28	0.00	0.00\|0.52	0.00	0.11
C	Holistic judgement	-2.11	-3.29\|-1.02	0.07	-1.22\|1.29	1.02	0.00\|1.85	0.66	0.00\|1.32	0.00	0.17
C	Length	-2.37	-3.64\|-1.41	1.00	-0.03\|2.30	0.60	0.00\|1.30	0.50	0.00\|1.12	0.03	0.11
C	Other	-3.34	-22.54\|-2.03	0.20	-18.20\|19.06	0.76	0.00\|1.94	0.72	0.00\|1.90	0.00	0.06
Note:
^Criteria condition is reference category.*

4.2.4 Create Figure 4.4

Create plot with results on content of positive feedback for sample B

#' Tibble with results
results_content_B <- 
  tibble(aspect  = factor(rep(c("Content", "Syntax", "Grammar", 
                                "Vocabu-\nlary", "Spelling"),
                              2),
                          levels = c("Content", "Syntax", "Grammar", 
                                "Vocabu-\nlary", "Spelling"),
                          ordered = TRUE),
         condition = factor(c(rep("Criteria condition", 5), rep("Comparative condition", 5)),
                            levels = c("Criteria condition", "Comparative condition"),
                            ordered = TRUE),
         rel_pos = c(18.2, 4.2, 4.0, 4.9, 1.8, 18.2, 27.9, 4.0, 4.9, 1.8),
         rel_neg = rep(c(9.6, 14.9, 16.5, 16.7, 17.1), 2),
         position_pos = rel_pos + 1.2,
         position_neg = rel_neg + 1.2,
         label_pos = c("18.2", NA, "4.0", "4.9", "1.8", 
                       "18.2", NA, "4.0", "4.9", "1.8"),
         label_neg = as.character(rel_neg))

#' Styling of labels of x-axis
text_face_xB <- ifelse(c("Content", "Syntax", "Grammar", "Vocabu-\nlary", "Spelling") %in%
                         (c("Content", "Syntax")), no = "bold", yes = "bold.italic")

#' Create plot  
plot_results_pos_content_B <- 
  results_content_B %>%
  ggplot(aes(x = aspect, y = rel_pos, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge", width = .85) +
  geom_text(aes(y = position_pos,  label = label_pos, group = condition), 
            color = "black", size = 5.25, family = "Arial") +
  annotate(geom = "text", label = "4.2", x = 1.77, y = 5.1,
           color = "black", size = 5.25, family = "Arial",
           fontface = "italic") +
  annotate(geom = "text", label = "27.9", x = 2.23, y = 28.8,
           color = "black", size = 5.25, family = "Arial",
           fontface = "italic") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 50)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample B") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x = element_text(color = "black", size = 12.5, face = text_face_xB),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.justification = c(-.10,0),
        legend.title = element_blank(),
        legend.text = element_text(size = 10),
        legend.position = "bottom")

Create plot with results on content of positive feedback for sample C

#' Tibble with results
results_content_C <- 
  tibble(aspect  = factor(rep(c("Content",  "Inter-\npretation",  "Style", 
                                "Language\nuse", "Holistic", "Length", 
                                "Other"),
                              2),
                          levels = c("Content",  "Inter-\npretation",  "Style", 
                                     "Language\nuse", "Holistic", "Length",
                                      "Other"),
                          ordered = TRUE),
         condition = factor(c(rep("Criteria condition", 7), rep("Comparative condition", 7)),
                            levels = c("Criteria condition", "Comparative condition"),
                            ordered = TRUE),
         rel_pos = c(50.0, 24.1, 25.7, 24.6, 10.8, 8.6, 3.2,
                     50.0,  7.1, 25.7, 24.6, 10.8, 8.6, 3.2),
         rel_neg = rep(c(50.0, 25.4, 31.0, 50.0, 2.3, 12.9, 4.0), 2),
         position_pos = rel_pos + 1.3,
         position_neg = rel_neg + 1.3,
         label_pos = c("50.0", NA, "25.7", "24.6", "10.8", "8.6", "3.2",
                       "50.0", NA, "25.7", "24.6", "10.8", "8.6", "3.2"),
         label_neg = as.character(rel_neg))

#' Styling of labels of x-axis
text_face_xC <- ifelse(c("Content",  "Inter-\npretation",  "Style", 
                         "Language\nuse", "Holistic", "Length",
                         "Other") %in%
                         (c("Content", "Inter-\npretation", "Style")), 
                       no = "bold", yes = "bold.italic")

#' Create plots
plot_results_pos_content_C <- 
  results_content_C %>%
  ggplot(aes(x = aspect, y = rel_pos, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge", width = .85) +
  geom_text(aes(y = position_pos,  label = label_pos, group = condition), 
            color = "black", size = 5.25, family = "Arial") +
  annotate(geom = "text", label = "24.1", x = 1.77, y = 25.1,
           color = "black", size = 5.25, family = "Arial", fontface = "italic") +
  annotate(geom = "text", label = "7.1", x = 2.23, y = 8.1,
           color = "black", size = 5.25, family = "Arial", fontface = "italic") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 52)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample C") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x = element_text(color = "black", size = 12.5, face = text_face_xC),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "none")

Assemble and save figure 4.4 (figure 4.4 is saved as png-file and can be found here)

#' Assemble figure 4.4
plot_pos_content <- 
  plot_results_pos_content_B +  plot_spacer() + plot_results_pos_content_C + 
  plot_layout(nrow = 1, widths = c(1.1, .05, 1.35))

#' Save figure 4.4
png_file <- here("03_Figures", "Figure 4.4.png")

ggsave(
  png_file,
  width  = 11.6,
  height = 5.64,
  units  = "in",
  device = "png",
  dpi = 320
)

knitr::plot_crop(png_file)
rm(plot_results_pos_content_B, plot_results_pos_content_C, png_file)

4.3 Content of negative feedback

To look at the effect on the content of the negative feedback, five generalized mixed-effect models (binomial distribution with logit-link) are fitted for sample B (for content, syntax, grammar, vocabulary and spelling) and 7 models for sample C (for content, interpretation, style, language use, holistic judgement, length and other). Furthermore, 95% bootstrapped confidence intervals and two effect sizes (marginal and conditional \(R^2\)-statistic) are estimated. All fitted models and other R-output (confidence intervals, \(R^2\)-statistics) are saved and can be found here.

4.3.1 Estimate models and 95% CI’s

Estimate mixed-effect models on content of negative feedback for sample B

#' Fit models
neg_content_B <- glmer(neg_contentD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_B,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

neg_syntax_B <- glmer(neg_syntaxD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                      data = data_RQ1_B,
                      family = binomial(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

neg_grammar_B <- glmer(neg_grammarD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_B,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

neg_voca_B <- glmer(neg_vocabularyD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                    data = data_RQ1_B,
                    family = binomial(),
                    nAGQ   = 0,  
                    glmerControl(optimizer = "bobyqa",
                                 optCtrl   = list(maxfun = 1000000)))

neg_spelling_B <- glmer(neg_spellingD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_B,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

#' Save models
save(neg_content_B, file = here("02_Models", "Content of feedback (negative)",
                                "neg_content_B.RData"))
save(neg_syntax_B, file = here("02_Models", "Content of feedback (negative)",
                               "neg_syntax_B.RData"))
save(neg_grammar_B, file = here("02_Models", "Content of feedback (negative)",
                                "neg_grammar_B.RData"))
save(neg_voca_B, file = here("02_Models", "Content of feedback (negative)",
                             "neg_voca_B.RData"))
save(neg_spelling_B, file = here("02_Models", "Content of feedback (negative)",
                                 "neg_spelling_B.RData"))

Estimate mixed-effect models on content of negative feedback for sample C

#' Fit models
neg_content_C <- glmer(neg_contentD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                       data = data_RQ1_C,
                       family = binomial(),
                       nAGQ   = 0,  
                       glmerControl(optimizer = "bobyqa",
                                    optCtrl   = list(maxfun = 1000000)))

neg_interpret_C <- glmer(neg_interpretD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                         data = data_RQ1_C,
                         family = binomial(),
                         nAGQ   = 0,  
                         glmerControl(optimizer = "bobyqa",
                                      optCtrl   = list(maxfun = 1000000)))

neg_style_C <- glmer(neg_styleD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                     data = data_RQ1_C,
                     family = binomial(),
                     nAGQ   = 0,  
                     glmerControl(optimizer = "bobyqa",
                                  optCtrl   = list(maxfun = 1000000)))

neg_language_C <- glmer(neg_languageD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_C,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

neg_holistic_C <- glmer(neg_holisticD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                        data = data_RQ1_C,
                        family = binomial(),
                        nAGQ   = 0,  
                        glmerControl(optimizer = "bobyqa",
                                     optCtrl   = list(maxfun = 1000000)))

neg_length_C <- glmer(neg_lengthD ~ 1 + condition +
                          (1|ID_student) + (1|ID_product),
                      data = data_RQ1_C,
                      family = binomial(),
                      nAGQ   = 0,  
                      glmerControl(optimizer = "bobyqa",
                                   optCtrl   = list(maxfun = 1000000)))

neg_other_C <- glmer(neg_otherD ~ 1 + condition + (1|ID_student) + (1|ID_product),
                     data = data_RQ1_C,
                     family = binomial(),
                     nAGQ   = 0,  
                     glmerControl(optimizer = "bobyqa",
                                  optCtrl   = list(maxfun = 1000000)))

#' Save models
save(neg_content_C, file = here("02_Models", "Content of feedback (negative)",
                                "neg_content_C.RData"))
save(neg_interpret_C, file = here("02_Models", "Content of feedback (negative)",
                                  "neg_interpret_C.RData"))
save(neg_style_C, file = here("02_Models", "Content of feedback (negative)",
                              "neg_style_C.RData"))
save(neg_language_C, file = here("02_Models", "Content of feedback (negative)",
                                 "neg_language_C.RData"))
save(neg_holistic_C, file = here("02_Models", "Content of feedback (negative)",
                                 "neg_holistic_C.RData"))
save(neg_length_C, file = here("02_Models", "Content of feedback (negative)",
                               "neg_length_C.RData"))
save(neg_other_C, file = here("02_Models", "Content of feedback (negative)",
                              "neg_other_C.RData"))

Bootstrap 95% confidence intervals for models on content of negative feedback of sample B

#' Bootstrapped 95% CI
CI_neg_content_B <- confint(neg_content_B, method = "boot", seed = 1980)
CI_neg_syntax_B <- confint(neg_syntax_B, method = "boot", seed = 1980)
CI_neg_grammar_B <- confint(neg_grammar_B, method = "boot", seed = 1980)
CI_neg_voca_B <- confint(neg_voca_B, method = "boot", seed = 1980)
CI_neg_spelling_B <- confint(neg_spelling_B, method = "boot", seed = 1980)

#' Save CI
save(CI_neg_content_B, file = here("02_Models", "Content of feedback (negative)",
                                   "CI_neg_content_B.RData"))
save(CI_neg_syntax_B, file = here("02_Models", "Content of feedback (negative)",
                                  "CI_neg_syntax_B.RData"))
save(CI_neg_grammar_B, file = here("02_Models", "Content of feedback (negative)",
                                  "CI_neg_grammar_B.RData"))
save(CI_neg_voca_B, file = here("02_Models", "Content of feedback (negative)",
                                "CI_neg_voca_B.RData"))
save(CI_neg_spelling_B, file = here("02_Models", "Content of feedback (negative)",
                                    "CI_neg_spelling_B.RData"))

Bootstrap 95% confidence intervals for models on content of negative feedback of sample C

#' Bootstrapped 95% CI
CI_neg_content_C <- confint(neg_content_C, method = "boot", seed = 1980)
CI_neg_interpret_C <- confint(neg_interpret_C, method = "boot", seed = 1980)
CI_neg_style_C <- confint(neg_style_C, method = "boot", seed = 1980)
CI_neg_language_C <- confint(neg_language_C, method = "boot", seed = 1980)
CI_neg_holistic_C <- confint(neg_holistic_C, method = "boot", seed = 1980)
CI_neg_length_C <- confint(neg_length_C, method = "boot", seed = 1980)
CI_neg_other_C <- confint(neg_other_C, method = "boot", seed = 1980)

#' Save CI
save(CI_neg_content_C, file = here("02_Models", "Content of feedback (negative)",
                                   "CI_neg_content_C.RData"))
save(CI_neg_interpret_C, file = here("02_Models", "Content of feedback (negative)",
                                     "CI_neg_interpret_C.RData"))
save(CI_neg_style_C, file = here("02_Models", "Content of feedback (negative)",
                                 "CI_neg_style_C.RData"))
save(CI_neg_language_C, file = here("02_Models", "Content of feedback (negative)",
                                    "CI_neg_language_C.RData"))
save(CI_neg_holistic_C, file = here("02_Models", "Content of feedback (negative)",
                                    "CI_neg_holistic_C.RData"))
save(CI_neg_length_C, file = here("02_Models", "Content of feedback (negative)",
                                   "CI_neg_length_C.RData"))
save(CI_neg_other_C, file = here("02_Models", "Content of feedback (negative)",
                                 "CI_neg_other_C.RData"))

4.3.2 Estimate R²-statistics

Estimate marginal and condition \(R^2\)-statistics on content of negative feedback for sample B

#' Estimate R2-statistics
R2_neg_content_B <- r.squaredGLMM(neg_content_B)
R2_neg_syntax_B <- r.squaredGLMM(neg_syntax_B)
R2_neg_grammar_B <- r.squaredGLMM(neg_grammar_B)
R2_neg_voca_B <- r.squaredGLMM(neg_voca_B)
R2_neg_spelling_B <- r.squaredGLMM(neg_spelling_B)

#' Save R2-statistics
save(R2_neg_content_B, file = here("02_Models", "Content of feedback (negative)",
                                   "R2_neg_content_B.RData"))
save(R2_neg_syntax_B, file = here("02_Models", "Content of feedback (negative)",
                                   "R2_neg_syntax_B.RData"))
save(R2_neg_grammar_B, file = here("02_Models", "Content of feedback (negative)",
                                   "R2_neg_grammar_B.RData"))
save(R2_neg_voca_B, file = here("02_Models", "Content of feedback (negative)",
                                "R2_neg_voca_B.RData"))
save(R2_neg_spelling_B, file = here("02_Models", "Content of feedback (negative)",
                                    "R2_neg_spelling_B.RData"))

Estimate marginal and condition \(R^2\)-statistics on content of negative feedback for sample C

#' Estimate R2-statistics
R2_neg_content_C <- r.squaredGLMM(neg_content_C)
R2_neg_interpret_C <- r.squaredGLMM(neg_interpret_C)
R2_neg_style_C <- r.squaredGLMM(neg_style_C)
R2_neg_language_C <- r.squaredGLMM(neg_language_C)
R2_neg_holistic_C <- r.squaredGLMM(neg_holistic_C)
R2_neg_length_C <- r.squaredGLMM(neg_length_C)
R2_neg_other_C <- r.squaredGLMM(neg_other_C)

#' Save R2-statistics
save(R2_neg_content_C, file = here("02_Models", "Content of feedback (negative)",
                                   "R2_neg_content_C.RData"))
save(R2_neg_interpret_C, file = here("02_Models", "Content of feedback (negative)",
                                     "R2_neg_interpret_C.RData"))
save(R2_neg_style_C, file = here("02_Models", "Content of feedback (negative)",
                                 "R2_neg_style_C.RData"))
save(R2_neg_language_C, file = here("02_Models", "Content of feedback (negative)",
                                    "R2_neg_language_C.RData"))
save(R2_neg_holistic_C, file = here("02_Models", "Content of feedback (negative)",
                                    "R2_neg_holistic_C.RData"))
save(R2_neg_length_C, file = here("02_Models", "Content of feedback (negative)",
                                   "R2_neg_length_C.RData"))
save(R2_neg_other_C, file = here("02_Models", "Content of feedback (negative)",
                                 "R2_neg_othrt_C.RData"))

4.3.3 Create Table 4.12 (appendix)

Results on the content of negative feedback for model B are tidied.

#' Tidy models
tidy_neg_content_B <- 
  neg_content_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Higher-order", category = "Content")

tidy_neg_syntax_B <- 
  neg_syntax_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Higher-order", category = "Syntax")

tidy_neg_grammar_B <- 
  neg_grammar_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Grammar")

tidy_neg_voca_B <- 
  neg_voca_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Vocabulary")

tidy_neg_spelling_B <- 
  neg_spelling_B %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "B", type = "Lower-order", category = "Spelling")

#' Tidy 95% CI 
tidy_CI_neg_content_B <-
  as_tibble(CI_neg_content_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Higher-order", category = "Content")

tidy_CI_neg_syntax_B <-
  as_tibble(CI_neg_syntax_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Higher-order", category = "Syntax")

tidy_CI_neg_grammar_B <-
  as_tibble(CI_neg_grammar_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order", category = "Grammar")

tidy_CI_neg_voca_B <-
  as_tibble(CI_neg_voca_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order", category = "Vocabulary")

tidy_CI_neg_spelling_B <-
  as_tibble(CI_neg_spelling_B) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "B", type = "Lower-order" , category = "Spelling")

#' Tidy R2-statistics
tidy_R2_neg_content_B <-
  as_tibble(rbind(R2_neg_content_B[2,], R2_neg_syntax_B[2,],
            R2_neg_grammar_B[2,], R2_neg_voca_B[2,],   R2_neg_spelling_B[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = rep("B", 5), 
         type = c(rep("Higher-order", 2), rep("Lower-order" , 3)),
         category = c("Content", "Syntax", "Grammar", "Vocabulary", "Spelling"))

Results on the content of negative feedback for model C are tidied.

#' Tidy models
tidy_neg_content_C <- 
  neg_content_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Content")

tidy_neg_interpret_C <- 
  neg_interpret_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Interpretation")

tidy_neg_style_C <- 
  neg_style_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Higher-order", category = "Scientific style")

tidy_neg_language_C <- 
  neg_language_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Language use")

tidy_neg_holistic_C <- 
  neg_holistic_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Holistic judgement")

tidy_neg_length_C <- 
  neg_length_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Length")

tidy_neg_other_C <- 
  neg_other_C %>% 
  tidy() %>%
  mutate(ID = c("Intercept", "Condition", "SD students", "SD products")) %>%
  select(ID, estimate) %>%
  pivot_wider(names_from = "ID", values_from = "estimate") %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = "C", type = "Lower-order", category = "Other")

#' Tidy 95% CI 
tidy_CI_neg_content_C <-
  as_tibble(CI_neg_content_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Content")

tidy_CI_neg_interpret_C <-
  as_tibble(CI_neg_interpret_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Interpretation")

tidy_CI_neg_style_C <-
  as_tibble(CI_neg_style_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Higher-order", category = "Scientific style")

tidy_CI_neg_language_C <-
  as_tibble(CI_neg_language_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Language use")

tidy_CI_neg_holistic_C <-
  as_tibble(CI_neg_holistic_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Holistic judgement")

tidy_CI_neg_length_C <-
  as_tibble(CI_neg_length_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Length")

tidy_CI_neg_other_C <-
  as_tibble(CI_neg_other_C) %>%
  rename(CI_low = `2.5 %`, CI_high = `97.5 %`) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(ID = c("SD students", "SD products", "Intercept", "Condition")) %>%
  rowwise() %>%
  mutate(CI = paste(CI_low, CI_high, sep = "|")) %>%
  ungroup() %>%
  select(ID, CI) %>%
  pivot_wider(names_from = "ID", values_from = "CI") %>%
  mutate(sample = "C", type = "Lower-order", category = "Other")

#' Tidy R2-statistics
tidy_R2_neg_content_C <-
  as_tibble(rbind(R2_neg_content_C[2,],  R2_neg_interpret_C[2,],
                  R2_neg_style_C[2,],    R2_neg_language_C[2,],  
                  R2_neg_holistic_C[2,], R2_neg_length_C[2,],
                  R2_neg_other_C[2,])) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate(sample = rep("C", 7), 
         type = c(rep("Higher-order", 3), rep("Lower-order" , 4)),
         category = c("Content", "Interpretation",
                      "Scientific style", "Language use",
                      "Holistic judgement", "Length", "Other"))

Create and save table 4.12 (table 4.12 is saved as xlsx-file and can be found here)

#' Bind results of sample B
part1 <- bind_rows(tidy_neg_content_B,  tidy_neg_syntax_B,
                   tidy_neg_grammar_B,  tidy_neg_voca_B,  
                   tidy_neg_spelling_B,
                   tidy_neg_content_C,  tidy_neg_interpret_C, 
                   tidy_neg_style_C,    tidy_neg_language_C,
                   tidy_neg_holistic_C, tidy_neg_length_C,
                   tidy_neg_other_C)

#' Bind results of sample C
part2 <- bind_rows(tidy_CI_neg_content_B,  tidy_CI_neg_syntax_B,
                   tidy_CI_neg_grammar_B,  tidy_CI_neg_voca_B,    
                   tidy_CI_neg_spelling_B,      
                   tidy_CI_neg_content_C,  tidy_CI_neg_interpret_C,
                   tidy_CI_neg_style_C,    tidy_CI_neg_language_C, 
                   tidy_CI_neg_holistic_C, tidy_CI_neg_length_C,  
                   tidy_CI_neg_other_C)

#' Bind results of sample B and sample C
part3 <-  bind_rows(tidy_R2_neg_content_B, tidy_R2_neg_content_C)


#' Bind results across content of feedback and add results on R2-statistics
table_content <-
  left_join(part1, part2, by = c("sample", "category", "type"),
            suffix = c("_est", "_CI")) %>%
  relocate(sample, type, category,
           Intercept_est,     Intercept_CI, 
           Condition_est,     Condition_CI, 
           `SD students_est`, `SD students_CI`, 
           `SD products_est`, `SD products_CI`) %>%
  left_join(part3, by = c("sample", "category", "type")) %>%
  rename("marg. R2" = "R2m", "cond. R2" = "R2c") %>%
  select(-type)

#' Save table as xlsx-file
openxlsx::write.xlsx(table_content, "04_Tables/Table 4.12.xlsx")

Sample	Content	Intercept		Condition<sup>*</sup>		Differences between students		Differences between products		Effectsizes
		Est.	95% CI	Est.	95% CI	SD	95% CI	SD	95% CI	marg. R²	cond. R²
B	Content	-2.24	-3.61\|-1.20	0.27	-0.54\|1.21	0.00	0.00\|0.98	1.00	0.00\|1.91	0.00	0.11
B	Syntax	-1.74	-2.41\|-0.65	0.60	-0.52\|1.46	1.33	0.46\|1.87	0.74	0.00\|1.34	0.01	0.32
B	Grammar	-1.62	-2.25\|-0.64	0.22	-0.76\|1.10	1.16	0.28\|1.72	0.65	0.00\|1.18	0.00	0.25
B	Vocabulary	-1.61	-2.17\|-0.87	-0.35	-1.27\|0.53	0.91	0.00\|1.39	0.33	0.00\|0.71	0.00	0.13
B	Spelling	-1.58	-2.19\|-0.65	-0.74	-1.67\|0.28	1.12	0.00\|1.68	0.50	0.00\|0.95	0.02	0.20
C	Content	0.75	-0.52\|1.95	-0.86	-1.84\|0.32	1.03	0.00\|1.65	1.24	0.14\|2.10	0.03	0.41
C	Interpretation	-1.08	-1.70\|-0.50	-0.65	-1.61\|0.29	0.53	0.00\|1.16	0.00	0.00\|0.61	0.02	0.06
C	Scientific style	-0.80	-1.54\|-0.16	-0.58	-1.45\|0.45	0.80	0.00\|1.37	0.38	0.00\|0.82	0.01	0.15
C	Language use	-0.55	-1.28\|0.08	-0.80	-1.68\|0.17	0.73	0.00\|1.22	0.20	0.00\|0.59	0.03	0.14
C	Holistic judgement	-3.75	-23.57\|-2.19	0.02	-19.88\|19.92	1.27	0.00\|2.60	0.00	0.00\|1.36	0.00	0.06
C	Length	-1.91	-2.92\|-0.95	-1.03	-2.61\|0.32	0.96	0.00\|1.70	0.53	0.00\|1.17	0.03	0.14
C	Other	-3.17	-4.60\|-1.89	0.35	-1.61\|2.36	1.13	0.00\|2.21	0.00	0.00\|0.98	0.00	0.08
Note:
^Criteria condition is reference category.*

4.3.4 Create Figure 4.5

Create plot with results on content of negative feedback for sample B

#' Create plot
plot_results_neg_content_B <- 
  results_content_B %>%
  ggplot(aes(x = aspect, y = rel_neg, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge", width = .85) +
  geom_text(aes(y = position_neg,  label = label_neg, group = condition), 
            color = "black", size = 5.25, family = "Arial") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 50)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample B") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x = element_text(color = "black", size = 12.5, face = text_face_xB),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.justification = c(-.1,0),
        legend.title = element_blank(),
        legend.text = element_text(size = 10),
        legend.position = "bottom")

Create plot with results on content of negative feedback for sample C

#' Create plot
plot_results_neg_content_C <- 
  results_content_C %>%
  ggplot(aes(x = aspect, y = rel_neg, group = condition)) +
  geom_col(aes(fill = condition), alpha = .9, position = "dodge", width = .85) +
  geom_text(aes(y = position_neg,  label = label_neg, group = condition), 
            color = "black", size = 5.25, family = "Arial") +
  scale_x_discrete(expand = expansion(add = c(0,0)), position = "top") +
  scale_y_continuous(limits = c(0, 52)) +
  scale_fill_brewer(type = "qual", palette = 6) +
  ggtitle("Sample C") +
  theme_light() + 
  theme(text = element_text(family = "Arial"),
        plot.title = element_textbox(size = 18, 
                                     r = unit(8, "pt"),
                                     padding = margin(1, 1, .5, 1, "mm"),
                                     margin = margin(0, 0, .5, 0, "cm"),
                                     color = NULL,
                                     linetype = 1,
                                     linewidth = .15, 
                                     fill = "#f6f6f6",
                                     hjust = 0.5),
        axis.text.x = element_text(color = "black", size = 12.5, face = text_face_xC),
        axis.text.y = element_blank(),
        axis.ticks  = element_blank(),
        axis.title  = element_blank(),
        panel.border     = element_blank(),
        panel.grid.major =  element_blank(),
        panel.grid.minor =  element_blank(),
        legend.position = "none")

Assemble and save figure 4.5 (figure 4.5 is saved as png-file and can be found here)

#' Assemble figure 4.5
plot_neg_content <- 
  plot_results_neg_content_B +  plot_spacer() + plot_results_neg_content_C + 
  plot_layout(nrow = 1, widths = c(1.1, .05, 1.35))

#' Save figure 4.5
png_file <- here("03_Figures", "Figure 4.5.png")

ggsave(
  png_file,
  width  = 11.6,
  height = 5.64,
  units  = "in",
  device = "png",
  dpi = 320
)

knitr::plot_crop(png_file)
rm(plot_results_neg_content_B, plot_results_neg_content_C, png_file,
   results_content_B, results_content_C, text_face_xB, text_face_xC)

5 Effect on performance (RQ2)

The effect of the peer assessment methods on the performance of students was assessed using t-tests and regression analysis. Analysis for sample A was done twice: once on the full sample and once on a filtered data set (excluding information of students with a maximum score on prior knowledge at the pre-test).

5.1 T-tests

5.1.1 t-tests per sample

T-test for sample A

#' T-test
ttest1_A <-  
  t.test(abilityZ ~ conditionF, data = data_RQ2_A, var.equal=F) %>%
  tidy() %>%
  select("mean criteria"    = "estimate1",
         "mean comparative" = "estimate2",
         "t"                = "statistic",
         "df"               = "parameter",
         "p"                = "p.value") 

#' Calculate N, mean and SD
SD1_A <- 
  data_RQ2_A %>%
  group_by(conditionF) %>%
  summarise(N = n(),
            mean = mean(abilityZ),
            SD = sd(abilityZ)) %>%
  ungroup() 

#' Calculate cohen's d
d1_A <- as.numeric((SD1_A[1,"mean"] - SD1_A[2,"mean"])/((SD1_A[1,"SD"] + SD1_A[2,"SD"])/2))

#' Merge all results
ttest1_A_full <-
  cbind(ttest1_A, SD1_A[1,"SD"], SD1_A[2,"SD"], d1_A) %>%
  rename("SD criteria" = 6, "SD comparative" = 7, "d" = 8) %>%
  relocate(`mean criteria`, `SD criteria`, `mean comparative`, `SD comparative`,
           t, df, p, d) %>%
  mutate_all(round, digits = 2)

#' Save results
save(ttest1_A_full, file = here("02_Models", "Effect on learning",
                                "ttest1_A_full.RData"))

T-test for sample A excluding data of students who received a maximum score on prior knowledge at pre-test

#' Filter data
data_RQ2_A_filtered <- 
  data_RQ2_A %>% filter(prior_knowledge!=8) 

#' T-test
ttest2_A <-  
  t.test(abilityZ ~ conditionF, data = data_RQ2_A_filtered, var.equal=F) %>%
  tidy() %>%
  select("mean criteria"    = "estimate1",
         "mean comparative" = "estimate2",
         "t"                = "statistic",
         "df"               = "parameter",
         "p"                = "p.value") 

#' Calculate N, mean and SD
SD2_A <- 
  data_RQ2_A_filtered %>%
  group_by(conditionF) %>%
  summarise(N = n(),
            mean = mean(abilityZ),
            SD = sd(abilityZ)) %>%
  ungroup() 

#' Calculate cohen's d
d2_A <- as.numeric((SD2_A[1,"mean"] - SD2_A[2,"mean"])/((SD2_A[1,"SD"] + SD2_A[2,"SD"])/2))

#' Merge all results
ttest2_A_full <-
  cbind(ttest2_A, SD2_A[1,"SD"], SD2_A[2,"SD"], d2_A) %>%
  rename("SD criteria" = 6, "SD comparative" = 7, "d" = 8) %>%
  relocate(`mean criteria`, `SD criteria`, `mean comparative`, `SD comparative`,
           t, df, p, d) %>%
  mutate_all(round, digits = 2)

#' Save results
save(ttest2_A_full, file = here("02_Models", "Effect on learning",
                                "ttest2_A_full.RData"))

T-test for sample B

#' T-test
ttest_B <-  
  t.test(abilityZ ~ conditionF, data = data_RQ2_B, var.equal=F) %>%
  tidy() %>%
  select("mean criteria"    = "estimate1",
         "mean comparative" = "estimate2",
         "t"                = "statistic",
         "df"               = "parameter",
         "p"                = "p.value") 

#' Calculate N, mean and SD
SD_B <- 
  data_RQ2_B %>%
  group_by(conditionF) %>%
  summarise(N = n(),
            mean = mean(abilityZ),
            SD = sd(abilityZ)) %>%
  ungroup() 

#' Calculate cohen's d
d_B <- as.numeric((SD_B[1,"mean"] - SD_B[2,"mean"])/((SD_B[1,"SD"] + SD_B[2,"SD"])/2))

#' Merge all results
ttest_B_full <-
  cbind(ttest_B, SD_B[1,"SD"], SD_B[2,"SD"], d_B) %>%
  rename("SD criteria" = 6, "SD comparative" = 7, "d" = 8) %>%
  relocate(`mean criteria`, `SD criteria`, `mean comparative`, `SD comparative`,
           t, df, p, d) %>%
  mutate_all(round, digits = 2)

#' Save results
save(ttest_B_full, file = here("02_Models", "Effect on learning",
                               "ttest_B_full.RData"))

T-test for sample C

#' T-test
ttest_C <-  
  t.test(abilityZ ~ conditionF, data = data_RQ2_C, var.equal=F) %>%
  tidy() %>%
  select("mean criteria"    = "estimate1",
         "mean comparative" = "estimate2",
         "t"                = "statistic",
         "df"               = "parameter",
         "p"                = "p.value") 

#' Calculate N, mean and SD
SD_C <- 
  data_RQ2_C %>%
  group_by(conditionF) %>%
  summarise(N = n(),
            mean = mean(abilityZ),
            SD = sd(abilityZ)) %>%
  ungroup() 

#' Calculate cohen's d
d_C <- as.numeric((SD_C[1,"mean"] - SD_C[2,"mean"])/((SD_C[1,"SD"] + SD_C[2,"SD"])/2))

#' Merge all results
ttest_C_full <-
  cbind(ttest_C, SD_C[1,"SD"], SD_C[2,"SD"], d_C) %>%
  rename("SD criteria" = 6, "SD comparative" = 7, "d" = 8) %>%
  relocate(`mean criteria`, `SD criteria`, `mean comparative`, `SD comparative`,
           t, df, p, d) %>%
  mutate_all(round, digits = 2)

#' Save results
save(ttest_C_full, file = here("02_Models", "Effect on learning",
                               "ttest_C_full.RData"))

5.1.2 Create Table 4.6

Create and save table 4.6 (table 4.6 is saved as xlsx-file and can be found here)

#' Create table 4.6
table_ttest <- 
  bind_rows(ttest1_A_full, ttest2_A_full,
            ttest_B_full,  ttest_C_full) %>%
  mutate_at(vars(1:7), formatC, format="f", digits=2) %>%
  mutate(sample = c("Sample A (full)", "sample A (filtered)", "Sample B", "Sample C")) %>%
  relocate(sample)

#' Save table 4.6 as xlsx-file
openxlsx::write.xlsx(table_ttest, "04_Tables/Table 4.6.xlsx")

Sample	Criteria condition		Comparative condition		Results of t-test
	M	SD	M	SD	t	df	p	Cohen’s d
Sample A (full)	-0.06	1.15	0.06	0.83	-0.52	69.22	0.61	-0.12
sample A (filtered)	-0.10	1.17	0.05	0.86	-0.62	62.69	0.53	-0.15
Sample B	0.08	0.90	-0.08	1.11	0.51	38.34	0.61	0.16
Sample C	0.03	0.90	-0.04	1.16	0.14	16.76	0.89	0.06

5.2 Regression analysis

5.2.1 Regression analysis per sample

Regression analysis for all samples

#' Fit models
model_regression1_A <- 
  lm(abilityZ ~ condition +  prior_knowledgeZ + self_eff_masteryZ +  
                self_eff_stateZ + self_eff_persuasionZ,
     data = data_RQ2_A) 

model_regression2_A <- 
  lm(abilityZ ~ condition +  prior_knowledgeZ + self_eff_masteryZ +  
                self_eff_stateZ + self_eff_persuasionZ,
     data = data_RQ2_A_filtered) 

model_regression_B <- lm(abilityZ ~ condition + prior_knowledgeZ + self_efficacyZ,
                         data = data_RQ2_B)

model_regression_C <- lm(abilityZ ~ condition + prior_knowledgeZ + self_efficacyZ,
                         data = data_RQ2_C)

#' Save models 
save(model_regression1_A, file = here("02_Models", "Effect on learning",
                                      "model_regression1_A.RData"))
save(model_regression2_A, file = here("02_Models", "Effect on learning",
                                      "model_regression2_A.RData"))
save(model_regression_B, file = here("02_Models", "Effect on learning",
                                     "model_regression_B.RData"))
save(model_regression_C, file = here("02_Models", "Effect on learning",
                                     "model_regression_C.RData"))

Estimate 95% confidence intervals for all samples.

#' Estimate 95% CIs
CI_regression1_A <- as_tibble(confint(model_regression1_A))
CI_regression2_A <- as_tibble(confint(model_regression2_A))
CI_regression_B  <- as_tibble(confint(model_regression_B))
CI_regression_C  <- as_tibble(confint(model_regression_C))

#' Save CI
save(CI_regression1_A, file = here("02_Models", "Effect on learning",
                                   "CI_regression1_A.RData"))
save(CI_regression2_A, file = here("02_Models", "Effect on learning",
                                   "CI_regression2_A.RData"))
save(CI_regression_B, file = here("02_Models", "Effect on learning",
                                  "CI_regression_B.RData"))
save(CI_regression_C, file = here("02_Models", "Effect on learning",
                                  "CI_regression_C.RData"))

Tidy model results for all samples

tidy_model_regression1_A <- model_regression1_A %>% tidy()
tidy_model_regression2_A <- model_regression2_A %>% tidy()
tidy_model_regression_B <- model_regression_B %>% tidy()
tidy_model_regression_C <- model_regression_C %>% tidy()

5.2.2 Create Tables 4.7 and 4.8

Create and save table 4.7 (table 4.7 is saved as xlsx-file and can be found here)

#' Bind results on sample A
part1 <-  bind_cols(tidy_model_regression1_A, CI_regression1_A,
                    tidy_model_regression2_A, CI_regression2_A)

#' Create table 4.7
table_regression1 <- 
  part1 %>%
  select("term" = "term...1",
         "est. (A1)"    = "estimate...2",
         "CI low (A1)"  = "2.5 %...6",
         "CI high (A1)" = "97.5 %...7",
         "est. (A2)"    = "estimate...9",
         "CI low (A2)"  = "2.5 %...13",
         "CI high (A2)" = "97.5 %...14") %>%
  replace_na(replace = list(rep("", 9))) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate_all(str_remove, "NA") %>%
  mutate(`CI (A1)` = ifelse(str_count(`CI low (A1)`) > 2,
                            yes = paste(`CI low (A1)`, `CI high (A1)`, sep = "|"),
                            no = " "),
         `CI (A2)` = ifelse(str_count(`CI low (A2)`) > 2,
                            yes = paste(`CI low (A2)`, `CI high (A2)`, sep = "|"),
                            no = " "),
         term = c("Intercept", "Condition<sup>*</sup>",
                  "Prior knowledge (Z)",
                  "Self-efficacy mastery (Z)",
                  "Self-efficacy state (Z)",
                  "Self-efficacy persuasion (Z)")) %>% 
  select(-c(contains("low"), contains("high"))) %>%
  relocate(1, 2, 4, 3, 5)

#' Save table 7 as xlsx-file
openxlsx::write.xlsx(table_regression1,  "04_Tables/Table 4.7.xlsx")

Variable	Sample A		Sample A (filtered)
	Est.	95% CI	Est.	95% CI
Intercept	-0.03	-0.33\|0.27	-0.04	-0.37\|0.29
Condition^*	0.06	-0.37\|0.49	0.06	-0.41\|0.52
Prior knowledge (Z)	0.11	-0.12\|0.34	0.09	-0.17\|0.35
Self-efficacy mastery (Z)	0.15	-0.09\|0.40	0.13	-0.12\|0.39
Self-efficacy state (Z)	0.31	0.08\|0.55	0.34	0.10\|0.58
Self-efficacy persuasion (Z)	-0.10	-0.32\|0.12	-0.05	-0.30\|0.20
Note:
Criteria condition is reference category.

Create and save table 4.8 (table 4.8 is saved as xlsx-file and can be found here)

#' Bind results on sample B and sample C
part1 <- 
  bind_cols(tidy_model_regression_B, CI_regression_B,
            tidy_model_regression_C, CI_regression_C)

#' Create table 4.8
table_regression2 <- 
  part1 %>%
  select("term" = "term...1",
         "est. (B)"    = "estimate...2",
         "CI low (B)"  = "2.5 %...6",
         "CI high (B)" = "97.5 %...7",
         "est. (C)"    = "estimate...9",
         "CI low (C)"  = "2.5 %...13",
         "CI high (C)" = "97.5 %...14") %>%
  replace_na(replace = list(rep("", 9))) %>%
  mutate_all(formatC, format = "f", digits = 2) %>%
  mutate_all(str_remove, "NA") %>%
  mutate(`CI (B)` = ifelse(str_count(`CI low (B)`) > 2,
                           yes = paste(`CI low (B)`, `CI high (B)`, sep = "|"),
                           no = " "),
         `CI (C)` = ifelse(str_count(`CI low (C)`) > 2,
                           yes = paste(`CI low (C)`, `CI high (C)`, sep = "|"),
                           no = " "),
         term = c("Intercept", 
                  "Condition<sup>*</sup>",
                  "Prior knowledge (Z)",
                  "Self-efficacy (Z)")) %>% 
  select(-c(contains("low"), contains("high"))) %>%
  relocate(1, 2, 4, 3, 5)

#' Save table 8 as xlsx-file
openxlsx::write.xlsx(table_regression2,  "04_Tables/Table 4.8.xlsx")

Variable	Sample B		Sample C
	Est.	95% CI	Est.	95% CI
Intercept	0.03	-0.38\|0.44	-0.06	-0.53\|0.42
Condition^*	-0.06	-0.63\|0.52	0.13	-0.58\|0.84
Prior knowledge (Z)	-0.03	-0.34\|0.29	0.74	0.36\|1.13
Self-efficacy (Z)	0.48	0.16\|0.79	-0.27	-0.66\|0.11
Note:
Criteria condition is reference category.

6 References

Hair, J. F., Anderson, R. F.,Tatbam, R.L. ,& Black, W. C. (1998). Multivariate Data Analysis (5th ed.). Upper Saddle River, NJ: Prentice-Hall.
Henson, R.K. & Roberts, J.K. (2006). Use of Exploratory Factor Analysis in Published Research. Common Errors and Some Comment on Improved Practice. Educational and Psychological Measurement, 66(3), 393-416. 10.1177/0013164405282485
Williams, B., Brown, T., & Onsman, A. (2010). Exploratory factor analysis: A five-step guide for novices. Australasian Journal of Paramedicine, 8(3). http://ro.ecu.edu.au/jephc/vol8/iss3/1

7 Information on packages

Below an overview is printed of all the packages (and version) used.

R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] MuMIn_1.43.17       broom.mixed_0.2.9.4 lme4_1.1-25        
 [4] Matrix_1.2-18       ggtext_0.1.2        kableExtra_1.3.4   
 [7] knitr_1.36          patchwork_1.1.2     rcompanion_2.3.26  
[10] psych_2.0.9         ggcorrplot_0.1.4    here_1.0.1         
[13] forcats_0.5.1       stringr_1.4.0       dplyr_1.0.10       
[16] purrr_0.3.4         readr_2.1.2         tidyr_1.2.1        
[19] tibble_3.1.8        ggplot2_3.4.2       tidyverse_1.3.2    
[22] readxl_1.3.1       

loaded via a namespace (and not attached):
  [1] minqa_1.2.4           TH.data_1.0-10        googledrive_2.0.0    
  [4] colorspace_2.0-3      ellipsis_0.3.2        class_7.3-17         
  [7] modeltools_0.2-23     rprojroot_2.0.2       markdown_1.1         
 [10] fs_1.5.0              gridtext_0.1.4        gld_2.6.2            
 [13] rstudioapi_0.14       farver_2.1.1          listenv_0.8.0        
 [16] furrr_0.2.3           fansi_1.0.3           mvtnorm_1.1-1        
 [19] lubridate_1.8.0       coin_1.3-1            xml2_1.3.3           
 [22] codetools_0.2-16      splines_4.0.3         mnormt_2.0.2         
 [25] rootSolve_1.8.2.1     libcoin_1.0-6         jsonlite_1.7.2       
 [28] nloptr_1.2.2.2        broom_1.0.3           dbplyr_2.2.1         
 [31] compiler_4.0.3        httr_1.4.2            backports_1.2.0      
 [34] assertthat_0.2.1      gargle_1.2.1          cli_3.4.1            
 [37] htmltools_0.5.1.1     tools_4.0.3           gtable_0.3.1         
 [40] glue_1.6.2            lmom_2.8              reshape2_1.4.4       
 [43] Rcpp_1.0.8            cellranger_1.1.0      jquerylib_0.1.4      
 [46] vctrs_0.5.1           svglite_2.1.1         nlme_3.1-149         
 [49] lmtest_0.9-38         xfun_0.29             globals_0.13.1       
 [52] openxlsx_4.2.3        rvest_1.0.3           lifecycle_1.0.3      
 [55] statmod_1.4.35        googlesheets4_1.0.1   future_1.20.1        
 [58] MASS_7.3-53           zoo_1.8-8             scales_1.2.1         
 [61] ragg_1.1.2.9000       hms_1.1.2             parallel_4.0.3       
 [64] sandwich_3.0-0        expm_0.999-5          RColorBrewer_1.1-3   
 [67] yaml_2.2.1            Exact_2.1             sass_0.3.1           
 [70] EMT_1.1               stringi_1.7.6         highr_0.8            
 [73] nortest_1.0-4         e1071_1.7-4           zip_2.1.1            
 [76] boot_1.3-25           rlang_1.1.1           pkgconfig_2.0.3      
 [79] systemfonts_1.0.1     matrixStats_0.57.0    evaluate_0.14        
 [82] lattice_0.20-41       labeling_0.4.2        tidyselect_1.1.2     
 [85] parallelly_1.21.0     plyr_1.8.6            magrittr_2.0.3       
 [88] R6_2.5.1              magick_2.7.2          DescTools_0.99.38    
 [91] generics_0.1.0        multcompView_0.1-8    multcomp_1.4-15      
 [94] DBI_1.1.0             pillar_1.8.1          haven_2.5.1          
 [97] withr_2.5.0           survival_3.2-7        modelr_0.1.8         
[100] crayon_1.5.1          utf8_1.2.2            tmvnsim_1.0-2        
[103] tzdb_0.3.0            rmarkdown_2.11        grid_4.0.3           
[106] webshot_0.5.2         reprex_2.0.2          digest_0.6.27        
[109] GPArotation_2014.11-1 textshaping_0.3.3     stats4_4.0.3         
[112] munsell_0.5.0         viridisLite_0.4.1     bslib_0.2.4

Peer assessment using criteria or comparative judgement? A conceptual replication study on the learning effect of two assessment methods

Tine van Daal, Mike Snajder, Kris Nijs, & Hanna Van Dyck

1 Data preparation

1.1 Sample A

1.1.1 Import and clean data

1.1.2 Factor analysis

1.1.3 Create variables

1.1.4 Create data for analysis

1.2 Sample B

1.2.1 Import and clean data

1.2.2 Factor analysis

1.2.3 Create variables

1.2.4 Create data for analysis

1.3 Sample C

1.3.1 Import and clean data

1.3.2 Factor analysis

1.3.3 Create variables

1.3.4 Create data for analysis

2 Descriptive statistics

2.1 Create Table 4.2

2.2 Create Table 4.3

2.3 Create Table 4.4

2.4 Create Table 4.5

2.5 Create Figure 4.2

3 Check randomisation

3.1 Sample A

3.2 Sample B

3.3 Sample C

4 Quality of feedback (RQ1)

4.1 Quantity of feedback

4.1.1 Estimate models and 95% CI’s

4.1.2 Estimate R2-statistics

4.1.3 Create Table 4.10 (appendix)

4.1.4 Create Figure 4.3

4.2 Content of positive feedback

4.2.1 Estimate models and 95% CI’s

4.2.2 Estimate R2-statistics

4.2.3 Create Table 4.11 (appendix)

4.2.4 Create Figure 4.4

4.3 Content of negative feedback

4.3.1 Estimate models and 95% CI’s

4.3.2 Estimate R2-statistics

4.3.3 Create Table 4.12 (appendix)

4.3.4 Create Figure 4.5

5 Effect on performance (RQ2)

5.1 T-tests

5.1.1 t-tests per sample

5.1.2 Create Table 4.6

5.2 Regression analysis

5.2.1 Regression analysis per sample

5.2.2 Create Tables 4.7 and 4.8

6 References

7 Information on packages

4.1.2 Estimate R²-statistics

4.2.2 Estimate R²-statistics

4.3.2 Estimate R²-statistics