A mathematical model for the automatic extraction of idioms in Turkic languages

Source abstract

This paper presents a complete mathematical model for the automatic extraction of phraseological units (idioms) from texts in Turkic languages, with a focus on Uzbek. The task is formulated as a binary classification problem and is solved by a three-stage pipeline. At the first stage, a set of idiom candidates is generated by a linguistic filter based on verb-headed grammatical templates. At the second stage, each candidate is mapped to a feature vector that combines statistical association (pointwise mutual information), syntactic fixedness (a flexibility index), semantic non-compositionality, and template-lexical indicators. At the third stage, a decision function is trained using four classification algorithms, namely Naive Bayes, logistic regression, support vector machines (SVM), and gradient boosting. The model was evaluated on a combined Uzbek corpus of 1.07 million lemma-tokens, supported by a phraseological ontology of 1,362 units and a manually annotated, stratified gold-standard set of 400 sentences. The best overall result was achieved by the SVM (????1 = 0.795; recall = 0.906), followed closely by gradient boosting (????1 = 0.791), whereas the lexical string-matching baseline proved nearly ineffective (????1 = 0.100) and ontology-based matching provided the highest precision (???? = 0.854). Stratified analysis showed that the main advantage of the model lies in recognizing morphologically transformed forms of idioms, where the learning-based model reached a recall of 0.919 against 0.041 for the lexical baseline. The role of each method within the processing pipeline and its most effective area of application are discussed.