Abstract
Knowledge is definedoperationally by the ability to make predictions. In multiple choice tests one has to predict which alternative will be scored as the correct one. In this context a testee is called a predictor. A prediction is defined as the assignment of probabilities to mutually exclusive events. The “true” probabilities are dependent on the knowledgeability of the predictor, on the Information he has: all probabilities are personal probabilities. Probabilities are called “true” if for each probability the proportion of occurrence of events approaches \the probability assigned to those events. A measure of “realism” indicating if reported probabilities are an overestimation or an underestimation of the true probabilities will be derived as will be the scoring rule to evaluate predictors.
Organisms do adapt to their environment in such a way that the payoff from that environment to their behavior is maximized. So do human predictors. To shape the behavior of an organism one should shape it environment. Important characteristics of environments that grow trustworthy predictors (telling that they don’t know for sure if they don’t) can be derived mathematically and show interesting relationships with Information theory. Paradoxically, the best measure of the predictors’ knowledgeability is not the optimal scoring rule.
It is quite feasible to build such optimal predictor environments with Computers. The essence is that the Utilities for the alternatives are variable: they have to be calibrated by the predictor. Examples using multiple choice tests and tests with open questions will be demonstrated.
Many-predictor Systems are discussed as are the information-theoretic measures on the relevancy of questions asked in such Systems. Differences with item analysis of educational tests are mentioned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bernardo, J.M.Expected Information as Expected Utility. Annals of Statistics 1979,7,686–690.
de Groot, A.D. Methodologie. The Hague. 1960, Mouton.de Groot, A.D. Research as a LearningProcess and Policy as a Learning Environment SVO Workshop on Educational Research and Public Policymaking, 1981, The Hague.
De Finetti, B.Logical Foundations and the measurement of Subjective Probabilities. Acta Psych.1970,34, 129–145.
Dirkzwager, A.Computer-based Testing with automatic scoring based on Subjective Probabilities. Computers in Education, eds. Lecarne, O.Lewis, R.; IFIP North-Holland Publ. Company 1975, 305–311.
Dirkzwager, A. Eendemocratische en objectieve beoordeling van de wetenschappelijke betekenis van een experiment. Congres van Psychologen1981 Nijmegen, Intern rapport vakgroepFunctieleer en Methodenleer, Amsterdam 1981, Vrije Universiteit.
Dirkzwager, A.Education for Knowledge Evaluation and Information Assessment. 2ndInt. Conf. on “Children in the Information Age”, 1987, Sofia.
Dirkzwager, A.Personal Probability Testing and the Assessment of Knowledge, a new pardigma: Multiple Evaluation. Europ. Meeting of thePsychometric Society, 1987, Enschede.
Hofstee,W.K.B. De Empirische Discussie, Meppel 1980, Boom.Hosseini, J. Ferrell, W.R. Detectability of Correctness:a Measure of Knowing that one Knows. Instr. Science 1982, 11, 113–127.
Leclercq, D.Confidence Marking: its use in Testing. Evaluation in Education, 1982, 6,161–287.
Lichtenstein, S. Fischhoff, B. Do those whoknow more also know more about how much they know. Org. Beh. and Human Performance 1977,20,159–183.
Lichtenstein, S.Fischhoff, B. Training for Calibration. Org. Beh. and Human Performance 1980,26,149–171.
Savage, L.J. TheFoundations of Statistics. New York 1951,
Wiley.Shuford, E. Brown, T.A. Elicitation of PersonalProbabilities and their Assessment. Instr. Science1975,4, 137–188.
Staël von Holstein, C A.S. Measurement of SubjectiveProbability. Acta Psych. 1970, 34, 146–159.
Terlouw, P.Subjective Probability Distributions: a Psychometric Approach. Groningen Univ.1989, dissertation.
van Lenthe, J. ELI:An Interactive Elicitation Technique for Subjective Probability Distributions. In: Org. Beh. and Human DecisionProcesses, december 1992.
van Lenthe, J.Scoring-rule Feedforward and the Elicitation of Subjective Probability Distributions. Groningen University, 1992: submitted forpublication.
van Lenthe, J. TheDevelopment and Evaluation of-ELI, an Interactive Elicitation Technique for Subjective Probability Distributions. In:D. Leclercq, J. Bruno (eds.): Item Banking:Interactive Testing and Seif-Assessment. NATO ASI Series F, Vol. 112. Berlin 1993,Springer-Verlag (this volume).
von Winterfeldt, D.Edwards, W. Decision Analysis and Behavioral Research. Cambridge 1986,Cambridge University Press.
Wallsten, T.S.Budescu, D.V. Encoding Subjective Probabilities: a psychological and psychometricreview. Management Science 1983, 29, 151–173.
Weltner, K. TheMeasurement of Verbal Information in Psychology and Education. Berlin 1973, Springer-Verlag.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1993 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Dirkzwager, A. (1993). A Computer Environment to Develop Valid and Realistic Predictions and Self-Assesment of Knowledge with Personal Probabilities. In: Leclercq, D.A., Bruno, J.E. (eds) Item Banking: Interactive Testing and Self-Assessment. NATO ASI Series, vol 112. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-58033-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-58033-8_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-63444-4
Online ISBN: 978-3-642-58033-8
eBook Packages: Springer Book Archive