Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Frequency consolidation among word N-grams: a practical procedure

Buerki, Andreas ORCID: https://orcid.org/0000-0003-2151-3246 2017. Frequency consolidation among word N-grams: a practical procedure. Mitkov, Ruslan, ed. Computational and Corpus-Based Phraseology, Lecture notes in Computer Science, vol. 10596. Cham: Springer, pp. 432-446. (10.1007/978-3-319-69805-2)

[thumbnail of 10.1007-978-3-319-69805-2_30.pdf]
Preview
PDF - Published Version
Download (365kB) | Preview

Abstract

This paper considers the issue of frequency consolidation in lists of different length word n-grams (i.e. recurrent word sequences) extracted from the same underlying corpus. A simple algorithm – enhanced by a preparatory stage – is proposed which allows the con- solidation of frequencies among lists of different length n-grams, from 2-grams to 6-grams and beyond. The consolidation adjusts the frequency count of each n-gram to the number of its occurrences minus its occur- rences as part of longer n-grams. Among other uses, such a procedure aids linguistic analysis and allows the non-inflationary counting of word tokens that are part of frequent n-grams of various lengths, which in turn allows an assessment of the proportion of running text made up of recurring chunks. The proposed procedure delivers frequency consolida- tion and substring reduction among word n-grams and is independent of any particular method of n-gram extraction and filtering, making it applicable also in situations where full access to underlying corpora is unavailable.

Item Type: Book Section
Date Type: Publication
Status: Published
Schools: English, Communication and Philosophy
Subjects: P Language and Literature > P Philology. Linguistics
Publisher: Springer
ISBN: 9783319698045
Date of First Compliant Deposit: 9 November 2017
Last Modified: 03 Nov 2022 09:54
URI: https://orca.cardiff.ac.uk/id/eprint/106389

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics