Abstract
In this paper, we try to find empirically the optimal dimensionality in data-driven models, Latent Semantic Analysis (LSA) model and Probabilistic Latent Semantic Analysis (PLSA) model. These models are used for building linguistic semantic knowledge which could be used in estimating contextual semantic similarity for the target word selection in English-Korean machine translation. We also facilitate k-Nearest Neighbor learning algorithm. We diversify our experiments by analyzing the covariance between the value of k in k-NN learning and accuracy of selection, in addition to that between the dimensionality and the accuracy. While we could not find regular tendency of relationship between the dimensionality and the accuracy, however, we could find the optimal dimensionality having the most sound distribution of data during experiments.
This work was supported by the Korea Ministry of Science and Technology under the BrainTech Project
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bain, L., M. Engelhardt, “Introduction to Probability and Mathematical Statistics,” PWS publishers, pp. 179, 190, 1987.
Berry., M., T. Do, G. O’Brien, V. Krishna, and S. Varadhan, “SVDPACKC: Version 1.0 User’s Guide,” University of Tennessee Technical Report, CS-93-194, 1993.
Cover, T., and P. Hart, “Nearest Neighbor Pattern Classification,” IEEE Transactions on Information Theory, 13, pp. 21–27, 1967.
Hofmann, T., “Probabilistic latent semantic indexing,” Proceedings of the 22th Annual International ACM SIGIR conference on Research and Developement in Information Retrieval (SIGIR99), pp. 50–57, 1999.
Kim, Y., B. Zhang and Y. Kim, “Collocation Dictionary Optimization using WordNet and k-Nearest Neighbor Learning,” Machine Translation 16(2), pp. 89–108, 2001.
Kim, Y., J. Chang, and B. Zhang, “A comparative evaluation of data-driven models in translation selection of machine translation,” Proceedings of the 19th Internation Conference on Computational Linguistics (COLING-2002), Taipei, Taiwan, pp. 453–459, 2002.
Landauer, T. K., P. W. Foltz, and D. Laham, “An Introduction to Latent Semantic Analysis,” Discourse Processes, 25, pp. 259–284, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kim, YS., Chang, JH., Zhang, BT. (2003). An Empirical Study on Dimensionality Optimization in Text Mining for Linguistic Knowledge Acquisition. In: Whang, KY., Jeon, J., Shim, K., Srivastava, J. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2003. Lecture Notes in Computer Science(), vol 2637. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36175-8_11
Download citation
DOI: https://doi.org/10.1007/3-540-36175-8_11
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-04760-5
Online ISBN: 978-3-540-36175-6
eBook Packages: Springer Book Archive