# Test development The Speechace test is made fully automatic through extensive use of natural processing techniques. The test is capable of rating students within ±0.54 point of qualified IELTS examiners. This incredible breakthrough was achieved through painstaking data gathering, manual rating, machine learning modeling and user testing over an 18 month period. The test was originally introduced as a practice activity in Speechace’s IELTSAce () app. The app is geared towards helping IELTS students prepare for the speaking section on the IELTS exam. The app is available for both Android and iOS devices and has been downloaded by over 1 millon students as evident below:
Audio samples collected from the app were transcribed samples using industry leading speech recognition and then manually graded by 3 qualified IELTS examiners on a variety of parameters including pronunciation, vocabulary, grammar, coherence and relevance. If 2 IELTS raters had more than one point difference in rating then the 3rd rater was asked to arbitrate. Here are the key statistics observed with regard to inter-rater agreement: % of items on which raters gave exactly the same grade = 21.8% \ % of items on which raters were within 0.5 IELTS points = 72.7% \ % of items on which raters were within 1 IELTS points = 98.2% \ Cohen’s kappa = 0.794 \ Pearson’s correlation between raters = 0.883 \ RMSE = 0.674 Once the data was graded, we ran algorithms that evaluated thousands of English syntax and sematic rules to determine which rules mattered the most for assessing pronunciation, fluency, vocabulary, grammar, cohesion and relevance. We then built deep learning machine learning models on the filtered set of rules to accurately predict IELTS scores on a 9 point scale for any arbitrary audio sample. Note that test re-test reliability of our models is found to be 0.82. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://speaking-test-docs.speechace.com/speechace-speaking-test/speechace-speaking-test-documentation/test-development.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.