RES framework :

remote-based text-to-speech modules' evaluation framework

The key element of the evaluation framework is based on the partition of a text-to-speech synthesis system into web-distributed modules. This split into several modules has the advantage that the developers of an institution active in ECESS can concentrate its efforts on specific module and test its performance in a complete TTS system using missing modules from the developers of other institutions or withinin existing test beds. The evaluation platform – the Remote Evaluation System (RES)  based on the existing internet infrastructure -, makes it possible to evaluate particular modules and to connect TTS modules efficiently.

Configurations for using RES system.

The RES is based on client-server architecture. It consists of several RES clients, several RES module servers (which encapsulate the modules of the developers), and a RES server, which connects all RES components together and organizes the flow of text or audio information. The RES may also be used for selecting the necessary module, that is encapsulated within the RES system, from the internet, which contains a possibly missing TTS module, needed to test and improve the performance of the evaluated TTS modules.

RES system client/server architecture.

The RES allows for the evaluation of TTS modules running at different institutions worldwide. By using RES client, the institution performing the evaluation is able to set-up and to perform various evaluation tasks by sending test data via the RES client and receiving results from selected RES module servers. ELDA www.elda.org is currently setting-up an evaluation using the RES client, which will then be extended to an evaluation client specializing in the envisaged evaluation tasks. 

Current RES system's tasks.

The advantage of RES evaluation platform is that developers run their TTS modules locally. The same is true for used language resources. Then, developers don’t need to be involved in the evaluation procedure, preparation, and running of evaluation packages and they don’t know anything about the evaluation data (very important). Evaluators on the other hand don’t need to contact developers to evaluate their TTS modules, don’t need to prepare evaluation packages for them, and don’t need to offer them any support. The evaluator can decide when to perform evaluation, by just running the RES client. The RES client provides him/her information about all the available TTS modules and the corresponding developers who are prepared to be involved in an on-going evaluation campaign. The RES client obtains this information from the RES server.  Then the evaluator just sequentially selects available TTS modules, enters the test data, and evaluates the returned results.