|
Buerki, Andreas
2017.
Frequency consolidation among word N-grams: a practical procedure.
In: Mitkov, Ruslan ed.
Computational and Corpus-Based Phraseology,
Lecture notes in Computer Science,
vol. 10596.
Cham:
Springer,
pp. 432-446.
(10.1007/978-3-319-69805-2)
|
Abstract
This paper considers the issue of frequency consolidation in lists of different length word n-grams (i.e. recurrent word sequences) extracted from the same underlying corpus. A simple algorithm – enhanced by a preparatory stage – is proposed which allows the con- solidation of frequencies among lists of different length n-grams, from 2-grams to 6-grams and beyond. The consolidation adjusts the frequency count of each n-gram to the number of its occurrences minus its occur- rences as part of longer n-grams. Among other uses, such a procedure aids linguistic analysis and allows the non-inflationary counting of word tokens that are part of frequent n-grams of various lengths, which in turn allows an assessment of the proportion of running text made up of recurring chunks. The proposed procedure delivers frequency consolida- tion and substring reduction among word n-grams and is independent of any particular method of n-gram extraction and filtering, making it applicable also in situations where full access to underlying corpora is unavailable.
| Item Type: |
Book Section
|
| Date Type: |
Publication |
| Status: |
Published |
| Schools: |
English, Communication and Philosophy |
| Subjects: |
P Language and Literature > P Philology. Linguistics |
| Publisher: |
Springer |
| ISBN: |
9783319698045 |
| Date of First Compliant Deposit: |
9 November 2017 |
| Last Modified: |
08 Jan 2018 09:54 |
| URI: |
http://orca.cf.ac.uk/id/eprint/106389 |
Actions (repository staff only)
 |
Edit Item |
Altmetric
Altmetric
Cardiff University Information Services