Optimization of the Morpher Morphology Engine Using Knowledge Base Reduction Techniques

Authors

  • Gábor Szabó Institute of Information Technology, University of Miskolc, Miskolc-Egyetemváros, H 3515, Hungary
  • László Kovács Institute of Information Technology, University of Miskolc, Miskolc-Egyetemváros, H 3515, Hungary

DOI:

https://doi.org/10.31577/cai_2019_4_963

Keywords:

Machine learning, natural language processing, inflection, lemmatization, agglutination, morphology, optimization, rule base reduction

Abstract

Morpher is a novel morphological rule induction engine designed and developed for agglutinative languages. The Morpher engine models inflection using general string-based transformation rules and it can learn multiple arbitrary affix types, too. In order to scale the engine to training sets containing millions of examples, we need an efficient management of the generated rule base. In this paper we investigate and present several optimization techniques using rule elimination based on context length, support and cardinality parameters. The performed evaluation tests show that using the proposed optimization techniques, we can reduce the average inflection time to 0.52 %, the average lemmatization time to 2.59 % and the number of rules to 2.25 % of the original values, while retaining a high correctness ratio of 98 %. The optimized model can execute inflection and lemmatization in acceptable time after training millions of items, unlike other existing methods like Morfessor, MORSEL or MorphoChain.

Downloads

Download data is not yet available.

Author Biographies

Gábor Szabó, Institute of Information Technology, University of Miskolc, Miskolc-Egyetemváros, H 3515, Hungary

PhD student, Institute of Information Technology

László Kovács, Institute of Information Technology, University of Miskolc, Miskolc-Egyetemváros, H 3515, Hungary

Head of Institute of Information Tech- nology

Downloads

Published

2019-12-30

How to Cite

Szabó, G., & Kovács, L. (2019). Optimization of the Morpher Morphology Engine Using Knowledge Base Reduction Techniques. Computing and Informatics, 38(4), 963–985. https://doi.org/10.31577/cai_2019_4_963