Optimization of the Morpher Morphology Engine Using Knowledge Base Reduction Techniques
DOI:
https://doi.org/10.31577/cai_2019_4_963Keywords:
Machine learning, natural language processing, inflection, lemmatization, agglutination, morphology, optimization, rule base reductionAbstract
Morpher is a novel morphological rule induction engine designed and developed for agglutinative languages. The Morpher engine models inflection using general string-based transformation rules and it can learn multiple arbitrary affix types, too. In order to scale the engine to training sets containing millions of examples, we need an efficient management of the generated rule base. In this paper we investigate and present several optimization techniques using rule elimination based on context length, support and cardinality parameters. The performed evaluation tests show that using the proposed optimization techniques, we can reduce the average inflection time to 0.52 %, the average lemmatization time to 2.59 % and the number of rules to 2.25 % of the original values, while retaining a high correctness ratio of 98 %. The optimized model can execute inflection and lemmatization in acceptable time after training millions of items, unlike other existing methods like Morfessor, MORSEL or MorphoChain.Downloads
Download data is not yet available.
Downloads
Published
2019-12-30
How to Cite
Szabó, G., & Kovács, L. (2019). Optimization of the Morpher Morphology Engine Using Knowledge Base Reduction Techniques. Computing and Informatics, 38(4), 963–985. https://doi.org/10.31577/cai_2019_4_963
Issue
Section
Articles