Speechace speaking test documentation

The Speechace speaking test is a fully automated speech recognition-based test that evaluates a candidate's ability to speak English through a short 10 minute simulated interview. The test uses a virtual avatar driven interface wherein the avatar provides instructions to the student while taking the test and at the end of the test, the student is given a comprehensive score on their pronunciation, vocabulary, grammar and fluency as below:

The score can be instantly converted in to different rubrics such as CEFR, IELTS, PTE, TOEFL and TOEIC.

Our primary goal in developing the test was to automatically evaluate a user’s spoken language skills through a short and engaging 1-on-1 conversation, just as a qualified human evaluator would.

In the paragraphs below, we outline a few salient features of the test:

1. Engaging avatar based test examiners

We took advantage of some of the most cutting edge developments in avatar technology to develop an avatar led virtual interview styled test. We made sure that the avatars were realistic and had sufficient gender and ethnic diversity. Check out our test collection to review the types of avatars we used or try out a specific test on College admissions. Note that the avatars videos can be easily replaced by real human videos for additional authenticity.

1. Holistic evaluation of speaking skills

We married some of the best work in speech technology with modern natural language processing techniques to develop algorithms that provide a holistic assessment of key speaking skills such as vocabulary, grammar, pronunciation, coherence, fluency, and relevance  of the response to test prompt. All our algorithms are designed to be robust enough to cope with transcription errors. Here is a sample score summary that the user sees after taking the test:

All responses are first passed through our custom designed relevance algorithm that rejects responses that are either irrelevant or insufficient. This ensures that no test taker can game the system. Further note that we provide an easy slider interface to convert the Speechace score to other standardized scores such as CEFR, TOEFL, PTE, IELTS and TOEIC.

1. Questions that make you think

Our test questions are designed to be contextual and make the candidate think. We provide 30 seconds to prepare for the question and then ask users to freely (spontaneously) speak for a minute. Based on our studies, we found that up to 3 such questions are sufficient to assess a candidate’s spoken language proficiency. Consider a speaking test we designed for recruiting call center workers – This test has the following 3 questions:

a. Why do you want to work in a call center?

b. How will you provide high quality customer service to callers?

c. How will you handle a call from an angry customer?

The beauty of such a test is that not only does it measure a user’s speaking skills but it also serves as a screening interview for domain knowledge and key job competencies.

1. A customizable domain specific speaking test

All aspects of our test are fully customizable. We can vary the type of avatars, the instructions spoken by avatars and the number of questions asked by avatars. We have developed the necessary infrastructure to quickly create custom tests to the needs of our customers. As of today, we have tests on a wide range of topics including college admissions, call center recruiting, aviation recruiting, nursing and more. With our test composition infrastructure we can create a test on any topic and of any length in a matter of minutes.

Furthermore, we can apply branding to avatars to personalize the test for a school or corporation. Consider the test we created for recruiting British Airways cabin crew members. You will observe that not only the avatars quiz users on topics relevant to being a cabin crew member but the avatars plug in branding specific to British Airways.

1. A transparent interactive report

At the end of the test, the user receives a detailed report with overall descriptive feedback and targeted feedback for each question on the test. Checkout a sample report for our candidate Jeevan Chopra.

The report provides feedback on the user’s response to each question including the response transcript, the rate at which they speak, any hesitation or pauses they have during speaking and a detailed pronunciation score. Additionally the report provides full audio playback of the user response along with playback of pronunciations of individual words:

Notice that currently we do not provide granular feedback on vocabulary and grammar but we are working on adding these details and hope to have them in a later release of the test.