Task Achievement
This feature is only available for Premium plan customers. Check out Speaking Test Pricing Plans for details.
Task achievement questions help you assess a broad spectrum of language competencies by giving the test taker a more nuanced language task that Speechace can evaluate against your own custom criteria.
There are two kinds of task-achievement questions:
Short Task Achievement (fluency)
Long Task Achievement
Short Task Achievement (fluency)
Short task achievement questions are great for testing pronunciation, fluency, and comprehension, based on responses shorter than 30 seconds.
These questions award half the grade for pronunciation and fluency, and the other half the grade for comprehension.
Using short task achievement, you can prescribe questions such as below: Question: Why did Harry Potter hate Voldemort?
Question Context: Harry Potter books
Sample Answer: This is because Voldemort killed Harry Potter's parents.
Score: 100%
Sample Answer: I have no idea.
Score: 0%
Long Task Achievement
Long task achievement questions are great for testing pronunciation, fluency, vocabulary, grammar, and comprehension, based on responses longer than 30 seconds.
These questions award half the grade to pronunciation, fluency, vocab, and grammar, and the other half of the grade to comprehension.
Using long task achievement, you can prescribe questions such as below:
Instructions: Compare and contrast two consumer products you particularly like. Justify your answers.
Question: Compare and contrast two consumer products you particularly like. Response must be comprehensive and at least 100 words.
Context: Product Comparison
Interface difference
On the test-administrator side, the two question types have only one difference: you can choose short tasks to be assessed in several languages other than English:
Apart from this difference, the two interfaces are identical, with exactly the same elements:
Getting started
To get started, do the following:
Select the appropriate question type for your question to display the associated configuration panel
Select the language you want to assess the test taker's abilities in
Type the instructions for the test taker either in the Question field on the left or the Instruction field on the right.
Configuration panel
The configuration panel for the Task Achievement question types is composed of the following elements.
Instruction field
The first field in the configuration panel is where you issue the instructions to the test taker.
This could be a standalone instruction (just like in Open-ended questions), or an instruction to perform some task based around 1 or 2 cue cards (see below).
Here are some examples:
Ask a question about the provided ferry timetable.
Describe what happened in the attached 4-panel comic strip.
⭑ Note: The instruction line is something that you provide. It is not used in scoring. As such, you are free to put anything you want here to best communicate instructions to the test taker. The place to provide your grading scheme is in the "Task achivement" section (below).
Cue cards upload section
The "Cue cards" field is recommended, but optional. It allows you to create a task around an obvious visual artifact:
You can upload a photo/illustration, or a screenshot of text with context beyond the main instructions.
For example, you could upload a timetable and ask the test taker to ask a question about it:
Or you can show a wordless comic strip to the test taker, and ask them to tell a story based on it.
See examples below for an elaboration on these.
Task achievement section
The input in the "Task achievement" section of the configuration panel is what the test taker's response is measured against. Speechace AI uses the content of the Question and Question Context fields to determine the quality of the response.
⭑ Note: This is arguably the most important and nuanced section, and one you should invest the most effort into testing and making it just right.
How it works
In the Instruction field at the top of the configuration panel, you tell the test taker:
"Perform the following task in such and such a way."
For example: "Why does Harry Potter hate Voldemort?"
Meanwhile, in the Task achievement section, you provide the scoring AI with the following information:
"Here is the task I am giving to the test taker."
"Here's how to score the different ways the test taker might respond."
"Here's the expected answer; or where to find the answer; or any additional knowledge that might be needed to tell whether the test taker's answer is correct."
Fill out the Task Achievement Question field with #1 and #2. For #2, you can have different categories of scores, anything from level of politeness or verbosity, to how much information the test taker provides, for example.
Fill out the Question Context field with #3.
Side Note: Task Achievement Question vs. Instruction
There is a subtle, but important distinction between the question/task you describe to AI in #1, and the task you ask the test taker to perform in the Instruction field (earlier, at the top of the configuration panel).
They can be written the exact same way, but they don't have to be. If they ever end up different from each other, it is because they have different motivations. One tells a human what to do, one describes the same thing to a robot.
For example, you could write "Why did Harry Potter hate Voldemort?" in both fields because in both cases they're self-documenting: it's obvious to both the test taker and the AI that, in this case, the assignment is for the test taker to answer the provided question. The only thing you could add is some context for the AI: if it doesn't already know about Harry Potter's relationship with Voldemort, the phrase "Harry Potter books" is enough to provide that context (via the Question Context field).
Example 1
To use the examples from Cue cards (above), you could upload a timetable and ask the test taker to come up with a question that someone could ask about it.
Hypothetical examples the test taker might come up with:
"When does the last ferry arrive?"
"How much does the first ferry cost?"
In the Instruction field at the top of the configuration panel, write the following:
Ask a question about this timetable.
In the Task Achievement section, in the Question field, write the following:
Ask a question based on the information written in the Context. Score 100 if Answer is a question based on the information in the Context. Score 0 if Answer is a question that cannot be answered based on the exact information in the Context.
In the Task Achievement section, in the Question Context field, write the following:
The image shows information on the Treasure Island Ferry schedule. There are 3 Treasure Island ferries that depart and arrive at different times. The first Treasure Island ferry departs at 10:00am and arrives at 11:00am and has a ticket price of 20 dollars. The second Treasure Island ferry departs at 1:00pm and arrives at 2:00pm and has a ticket price of 20 dollars. The third and last Treasure Island ferry departs at 4:00pm and arrives at 5:00pm and has a ticket price of 20 dollars.
Example 2
Show a wordless comic strip to the test taker, and ask them to tell a story based on it.
In the Instruction field at the top of the configuration panel, write the following:
Describe the story shown in the image.
In the Task Achievement section, in the Question field, write the following:
Write the Context in your own words. You must describe each individual sentence in the Context. Score 25 if the Answer is 1 sentence written in the Context. Score 50 if the Answer is 2 sentences written in the Context. Score 75 if the Answer is 3 sentences written in the Context. Score 100 if the Answer is 4 sentences written in the Context. Score 0 if the Answer has no sentences written in the Context.
In the Task Achievement section, in the Question Context field, write the following:
A boy was walking on the street with some shops looking at his cellphone. A woman stepped out of a store looking back and talking to the salesperson in the store. The boy collided with the woman and the woman dropped her shopping bag. The boy picked up the woman's shopping bag and made her happy.
Testing and refining task achievement assessments
After filling out all the fields in the Task Achievement configuration panel, click Test:
On the Test Task Achievement pop-up, you see your drafts of the Question and Question Context forms, as well as a field to type potential student responses:
Here we strongly recommend spending as much time as possible to familiarize yourself with how the AI processes your inputs.
You can do this by trying out potential student responses to see how they would score against your Question and Question Context.
You can change those as well (saving along the way), until it feels like it's doing the right thing.
⭑ Note: Be sure to test the complete spectrum of respoonses, all the way from 100% correct to wildly off-base.
Last updated
