Measuring the Performance and Intelligence of Systems: Proceedings of the 2001 PerMIS Workshop
Abstract
Testing of performance pertains to evaluation of the potential and actual capabilities of a system to satisfy the expectations of the designer and the users via exploration of its functioning. This includes determining how well the system performs its declared "job," how efficiently and effectively it does so, how robust it is, and so forth. The "job" and expected performance must therefore be defined at the outset. Efficiency is defined as how well the system does things right, effectiveness is defined as how well the system does the right thing, and robustness is defined as "the degree to which a system can function correctly in the presence of invalid inputs or stressful environmental conditions." [Finklestein, 00] Furthermore, the tests under consideration are not meant to be broad-based general evaluations of the system's knowledge or the full spectrum of its capabilities. In particular, we are not striving to ascertain whether a system has common-sense generic knowledge applicable to general-purpose problem solving. The system being evaluated has a given sphere of responsibility and known abilities and tasks that it is able to undertake under its specifications. Comments regarding the testing of intelligent versus non-intelligent systems are not meant to underestimate the difficulty of testing non-intelligent systems. Testing robustness, efficiency, and even functionality of non-intelligent software systems is difficult enough, e.g., see [Mukherjee 97]. Since the software execution can follow a myriad of combinations of paths through the code, it is impossible, in typical practice to exhaustively test all the possible combinations. In non-deterministic real-time systems, the problem is compounded by the uncertainty in the execution times of various processes, the sequence of events, asynchronous interrupts, etc [Butler, 93]. In general, the evaluation of intelligent systems (IS's) is broader than testing of non-intelligent systems (NIS).
Document Details
- Document Type
- Technical Report
- Publication Date
- Sep 04, 2001
- Accession Number
- ADA515942
Entities
People
- A. M. Meystel
- E. R. Messina
Organizations
- National Institute of Standards and Technology