Before you begin building your experiment, here's a quick reminder to reach out to email@example.com if you have any questions!
As you get started, here are a few things to have on hand:
- An experiment title and description
- A PDF of the experiment’s Informed Consent statement (must be smaller than 10MB)
- Different versions of each assignment for each experimental condition
- A calendar with the open, due, and close dates for each assignment
Table of Contents
Terracotta will walk you through setting up your experiments. Begin by clicking Terracotta in the left navigation bar of your Canvas course site, and then click "Create your first experiment."
Then, give your experiment a descriptive title (i.e., “My First Terracotta Experiment” or “Nudging Interventions in High School Math”) and click Next.
Describe your experiment to give you a way to differentiate it from others and click Next.
Name your conditions (any experiment is an attempt to determine the difference between 2 different conditions. In this one, we’re going to try to determine the effects of equations that are written consistent with mathematical order of operations), and click Save & Exit.
Now decide how you would like participants to be exposed to those conditions. Choosing “one condition” means that participants will be exposed to one condition (in this example, either “Consistent” or “Inconsistent”). Choosing “all conditions” means that all participants will be exposed to both conditions (in this example, both “Consistent” and “Inconsistent”), and you will have to create at least two assignments in order to balance the experiment.
Make your choice, then click Select.
Then choose your default condition (this should be the condition that is closest to the classroom’s normal instructional practice). Click next.
Now you’ve completed the first part of the experiment set-up! Click Continue to Next Section.
Create your Informed Consent assignment and upload your PDF. Terracotta automatically gives students 100% completion credit on this assignment, but you can change the point value in your Canvas settings.
You have three participation options:
- Students will be invited to consent - this option will create a consent assignment within Canvas.
- Teacher will manually decide - this option allows teachers to opt minor students in after collecting parental consent.
- Automatically include all students - this option requires no consent.
Once you have made your choice, click Select.
Click Continue, and we’ll walk through the process of setting up a consent assignment.
Give your assignment a title, then click Next.
Upload your Informed Consent PDF, then click Next. Note that Terracotta offers a link to download an informed consent template if you’re not familiar with their contents.
Now decide how you’d like your participants distributed. You have three options:
- Even - this option will distribute consenting students across all conditions. Note: if you have chosen “All Conditions,” even distribution will be your only option.
- Custom - this option lets you choose the percent of participants distributed to each condition
- Manual - this option allows you to choose the students distributed to each condition(s)
Once you've made your choice, click Select.
Great work! You set up your Participation process! Now click Continue to Next Section.
Create your different exposure sets and different assignment versions; Terracotta will walk you through balancing them.
In this experiment, our “consistent” condition will use regular spacing in math equations; our “inconsistent” condition will use irregular spacing. We want all participants to experience both conditions, so we will need to make two assignments for two exposure sets (four total assignments). To get started, you’ll click Continue, and then again on the next page, and then on the page after that once you read through the pages.
Start creating your first assignment by giving it a title. This title will appear in Canvas, and your students will see it (once you publish the assignment). You also have the option to opt into and out of assignment-level settings here. Click Continue when you're ready to proceed.
On the next page, click the arrow to reveal the drop down menu, then click Create.
Give your treatment a name (this will not show in Canvas), and instructions (these will show in Canvas). Click Add Question to reveal the types of questions you can add. For this example, we’ll use multiple choice.
Decide how many multiple choice questions you want to put into this assignment, and add them by clicking Add Question as many times as you need. We’ll use 3 questions in this assignment.
Then, click each question to drop down its fields and fill out those fields for each one.
Start by populating the question field and point value. Note: questions can be worth 0 points while still giving students credit for completing the assignment.
Then, add answer options by clicking the Option field and adding a value.
Once you’ve added your answer choices, be sure to click the checkmark for the correct answer (this allows Canvas to grade the assignment automatically).
Decide if you want to randomize the order in which the options appear by clicking the button (it will turn blue when activated).
You can also click the meatball menu (3 horizontal dots) to expose “Add page break” and “Delete question” actions.
Then, click into the next question (2, in this example), and repeat the process for your remaining questions. Once they’re all populated, scroll to the bottom of the page and click continue. That click will take you back to the assignments screen where you’ll see the number of assignments you have created for each treatment.
Click Exposure 1 to create your assignment for the “Inconsistent” treatment. Then, repeat the process until you have created two versions of Assignment 1. Terracotta will show you that you’ve created 2/2 treatments; now you’re ready to balance your experiment by creating two assignments for Exposure 2.
When the experiment is balanced, it will look like this:
You will also see that your assignments and exposures are balanced in the assignments tab:
Now, go into your Canvas site and click Assignments in the left navigation bar. Then, find the assignments you just created (don’t forget your Informed Consent assignment!) and click Edit Assignment Settings. Set the assignments’ open and close dates, and publish them. You’re ready to start your experiment!
To students, Terracotta assignments look like any other Canvas assignments in that they appear in assignment and grades lists alongside all their class assignments. We’ll go through the flow of the assignments created in the section above so you can see what they look like from the student perspective.
After opening the Informed Consent assignment, your student will see a screen containing instructions and the document.
The student scrolls to read the document, and is then asked to agree or not agree to participate in the experiment.
Students who agree become part of the study; students who do not are placed in the “business as usual” condition, and their data will not be included in any exports (this guide shows the flow when students agree). When the students click submit, they will see a confirmation screen.
Note: After the student has submitted an assignment, even if it does not require manual grading, it will appear as “Not Yet Graded” until the instructor releases the grade in Canvas. Once you release the grade, students who have completed it will receive full credit regardless of their responses.
The student opens the assignment, answers the questions, and clicks submit.
A message asking the student to confirm that they want to submit the assignment “as is” will come up, and the student will have the option to confirm, or to cancel and go back. Once the student confirms, Terracotta also confirms submission.
As in Canvas, the instructor will need to release the grades before the student can see them.
Once your students have completed the assignment, you can link the questions to outcomes. Start by navigating to your experiment’s status tab, which shows the different parts of your experiment including Informed Consent, if applicable.
Click into the exposure to which you’d like to add the outcome, then click Add Outcome, and decide whether you’d like to select a gradebook item to serve as an outcome measure or add scores manually.
To select an item from your gradebook, choose that option, and then check the box (or boxes) for the items you’d like to add. When you’re done, click Save & Exit in the top right corner.
To verify that your items have been added as outcome measures, open the Status tab; you should see the item listed below the Outcomes header.
To enter scores manually, choose that option, and give the outcome a name in the next screen.
This option is useful for entering participants’ demographic information, like their status as major or nonmajor in the course’s subject matter, or their year in the program.
Once your experiment is complete and you’ve added all your outcome measures, it’s time to export your data.
Exporting and Analyzing Data
Exporting your data
When you’re ready to export your data, open your experiment in Terracotta and click the blue Export Data button in the upper right hand corner.
This will download a zip file to your computer.
The zip file contains several csv files that contain the data for your experiment.
Analyzing your data
Once you've downloaded the data from Terracotta, it's time to analyze your findings. Here is an R script that provides some examples of how one might analyze the data exported from an experiment in Terracotta.