
Preparing and cleaning a corpus
Written by Mohammed Saqr and Sonsoles López-Pernas
Source:vignettes/articles/preparing-corpus.Rmd
preparing-corpus.RmdA sentence encoder represents everything in a text, including what an analyst would not count as content. A copyright notice at the end of an abstract, a list marker, a URL or a broken character moves the text’s embedding, and when the same noise recurs across many texts it can draw them together into a topic of its own, grouped by publisher or formatting rather than by meaning. Repetition has a similar effect: a text that occurs two hundred times enters k-means two hundred times and pulls a center toward itself. Corpus preparation therefore precedes encoding, and every step changes the data that are modeled, so each one should be stated and its effect counted.
This guide prepares the covid abstracts in the order the
steps should be applied: removing records without text, removing
duplicates, repairing and cleaning the text, and checking how much of
each unit is prose. Every step runs without a model. Choosing the words
that describe topics, which is done after fitting and does not change
the text, is covered at the end.
library(sbert)
#> sbert 0.6.0: Sentence-BERT embeddings and topic models in R.
#> 14 pinned models (see models()); default: all-MiniLM-L6-v2.
#> A model is downloaded only on request, once, and verified by SHA-256.Data
covid holds 4,170 records of research on COVID-19 and
education, children, schools and society, published from 2020 to
2024.
-
Abstract: the text of the abstract; the text to be modeled. -
Year: the year of publication; carried along with each abstract.
The feedback messages of feedback_translations are used
where a step is better shown on short texts.
Which records have no text?
Bibliographic exports fill a missing abstract with a placeholder. In
covid it is the string
"[No abstract available]":
323 of the 4,170 records have no abstract. The placeholder is a well-formed English phrase, so no cleaning step would remove it, and kept, it would become 323 copies of one text, a topic about nothing. It is removed first:
How much of the corpus repeats?
dedupe() keeps the first occurrence of every distinct
text and counts its occurrences in a new column, n. Given a
data frame and the name of its text column, it keeps all columns of the
first occurrence:
abstracts <- dedupe(abstracts, column = "Abstract")
nrow(abstracts)
#> [1] 3670
with(abstracts, table(n))
#> n
#> 1 2 4
#> 3495 174 1The 3,847 records hold 3,670 distinct abstracts: 3,495 occur once,
174 twice, and one four times, records indexed more than once. Had the
placeholder been kept, dedupe() would have reduced it to a
single row carrying its count:
Given a character vector, dedupe() returns the columns
text and n. Repetition is far heavier in the
feedback messages, which a generator writes from templates:
messages <- with(feedback_translations, dedupe(translation))
nrow(feedback_translations)
#> [1] 8757
nrow(messages)
#> [1] 8005
head(messages, 3)
#> text n
#> 1 Do you know what “one more” means? 2
#> 2 There should be two more in the red than in the blue. 3
#> 3 If it’s not a person, what could it be? 2The 8,757 messages hold 8,005 distinct texts. Modeling the distinct
texts keeps repeated templates from dragging the topic centers, and the
counts are not lost: after fitting,
topic_sizes(weights = "n") reports every topic’s share of
all occurrences beside its share of distinct texts. On the first 600
distinct messages, whose embeddings are bundled:
corpus <- head(dedupe(feedback_translations, column = "translation"), 600)
topic_model <- topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings)
topic_sizes(topic_model, weights = "n")
#> topic label n_documents
#> 1 1 Where is the cow and the cake? Choose all the pictures... 223
#> 2 2 All the ones between the cat and the strawberry. 133
#> 3 3 What does “all the ones between the leaf and the... 131
#> 4 4 Are you sure you have fewer pictures in the red box than... 49
#> 5 5 In the top middle box, it means something is in the... 37
#> 6 6 It is not a person, does not have a hat, and is sitting. 27
#> proportion n_weighted weighted_share
#> 1 0.3717 267 0.3608
#> 2 0.2217 162 0.2189
#> 3 0.2183 138 0.1865
#> 4 0.0817 76 0.1027
#> 5 0.0617 57 0.0770
#> 6 0.0450 40 0.0541n_documents and proportion count distinct
messages; n_weighted and weighted_share count
occurrences. The topic of corrections (topic 4) holds 8.2% of the
distinct messages and 10.3% of the occurrences, because its messages
were repeated more often than the others.
What does clean_corpus() repair?
clean_corpus() gathers the text-level cleaning that
should happen before encoding, so that the same clean text feeds the
encoder, the topic terms and the representative texts. Some repairs are
always applied: HTML tags and entities are decoded or removed; URLs,
DOIs, e-mail addresses and bracketed numeric citations such as
[12] are removed; curly quotes, dashes and bullets are
normalized; and control, zero-width and replacement characters are
dropped. Removing list and section numbering is on by default and can be
switched off. Four invented texts show the effect:
examples <- c("<p>Teachers & parents</p> reported higher stress [12].",
"Data are available at https://doi.org/10.1000/xyz123 for reuse.",
"1. Aims. 2. Methods. 3. Results.",
"Section 2.3.1 describes the sample.")
clean_corpus(examples)
#> [1] "Teachers & parents reported higher stress."
#> [2] "Data are available at for reuse."
#> [3] "Aims. Methods. Results."
#> [4] "Section describes the sample."The markup, the citation, the URL, the list markers and the section number are gone, and the words are kept.
Abstracts also carry text that the generic repairs cannot recognize.
In covid almost every abstract ends with a publisher’s
copyright notice:
3,655 of the 3,670 abstracts contain the copyright sign. The
remove argument takes regular expressions, matched
case-insensitively, whose matches are deleted before the other steps.
The pattern below deletes a notice at the end of an abstract that starts
with “©”, “Copyright ©” or “Copyright” and a year, and leaves abstracts
that discuss copyright as a subject untouched. Given a data frame and a
column, clean_corpus() cleans that column and
keeps the other columns aligned:
cleaned <- clean_corpus(abstracts, column = "Abstract",
remove = "(copyright:?\\s*(©|\\d{4})|©).{0,1000}$")
nrow(cleaned)
#> [1] 3670
with(cleaned, sum(grepl("©", Abstract)))
#> [1] 0
with(cleaned, sum(grepl("copyright", Abstract, ignore.case = TRUE)))
#> [1] 2All 3,670 abstracts are kept, none still contains the copyright sign, and the two that still contain the word “copyright” use it as a subject (accessibility and copyright in online teaching; open educational resources free from copyright issues). The cleaning shortens the abstracts:
with(abstracts, summary(nchar(Abstract)))
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 171 1022 1315 1352 1640 5285
with(cleaned, summary(nchar(Abstract)))
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 117 975 1261 1300 1583 5243The median abstract loses 54 characters, about the length of a copyright line.
Should numbers be removed from the text?
numbers = TRUE and roman_numerals = TRUE
remove stand-alone numeric and Roman-numeral tokens from the text. Both
are off by default, because the result is no longer a well-formed
sentence:
clean_corpus("In 2020, 45 schools closed during phase ii.",
numbers = TRUE, roman_numerals = TRUE)
#> [1] "In, schools closed during phase."The sentence loses its year, its count and its phase, and the encoder
sees a fragment. Numbers rarely decide what a text is about, but they do
clutter the words that describe topics. That is better handled after
fitting, with the numbers = "remove" and
roman_numerals = "remove" arguments of
topics() and terms(), which filter the terms
and leave the text and its embedding intact (see Describing and naming topics).
Which list markers are removed?
strip_list_markers(), which clean_corpus()
applies by default, removes enumeration markers that stand alone:
1., (i), (a), (1)
and Roman-numeral section markers such as II.. It keeps
numbers and parentheses that carry content:
strip_list_markers(c("The aims were (i) access and (ii) equity.",
"1. Background 2. Methods 3. Results",
"The student ranks in the top 20. Good work.",
"Smith v. Jones, cited by V. Nabokov in (2020), with (n=100)."))
#> [1] "The aims were access and equity."
#> [2] "Background Methods Results"
#> [3] "The student ranks in the top 20. Good work."
#> [4] "Smith v. Jones, cited by V. Nabokov in (2020), with (n=100)."The markers of the first two texts are removed. A number that ends a
sentence (“top 20.”), a year or a count in parentheses, a lower-case
“v.” and a mid-sentence initial are kept. In the abstracts,
parenthesized markers such as (i), (1) or
(a) occur in 193 texts before cleaning and in none
after:
How much of each unit is prose?
content_ratio() is the share of a text’s non-space
characters that are letters. Prose scores near 1; references, page
ranges, tables of numbers and identifiers score low:
content_ratio(c("A real sentence here.",
"OJ No L 297, 24.11.1979, p. 1.",
"doi:10.1007/s10639-021-10557-6"))
#> [1] 0.944 0.250 0.133
with(cleaned, summary(content_ratio(Abstract)))
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 0.794 0.962 0.969 0.966 0.975 0.992Every abstract is mostly prose: the lowest ratio is 0.794. The check
matters for smaller units. Split into sentences with
segment(), the abstracts yield fragments that are not prose
at all:
sentences <- with(cleaned, segment(Abstract, level = "sentence"))
nrow(sentences)
#> [1] 28515
knitr::kable(subset(sentences, content_ratio(text) < 0.5,
select = c(document_id, text)))| document_id | text | |
|---|---|---|
| 319 | 44 | 2006. |
| 4374 | 550 | 1999. |
| 4379 | 550 | Ylä-Kotola, 679 - 718. |
| 10130 | 1220 | 2021;60(9):534-537.]. |
| 16883 | 2048 | 26.. |
| 17050 | 2070 | The significantly predicted equations were as follows: In term of raw scores: Y/ = 75.425 + (-5.180) (X4) + 3.816 (X3) + (-3.465) (X5) + (-2.689) (X7) In term of standard scores: Z/Y = (-0.324) (Z X 4) + 0.280 (Z X 3) + (-0.225) (Z X3) + (-0.165) Z X 7). |
| 19107 | 2330 | 2021 |
| 19139 | 2333 | 2021 |
Eight of the 28,515 sentences fall below one half: years, page
ranges, the remains of a citation, and a regression equation. Each would
be embedded as a unit and could join a topic. min_content
drops them: clean_corpus(min_content = ) drops whole
documents below the ratio, and segment(min_content = ) and
topics(min_content = ) drop segments. Splitting documents
is covered in Segmenting long
documents.
Which words are ignored when topics are described?
Stop words do not change the text or the embeddings. They are removed
only when the words that describe each topic are computed.
stop_words() returns the default English list, and
add and remove adapt it to a corpus:
length(stop_words())
#> [1] 125
head(stop_words(), 10)
#> [1] "a" "about" "after" "again" "against" "all" "am" "an"
#> [9] "and" "any"
covid_stop_words <- stop_words(add = c("covid", "pandemic"))
length(covid_stop_words)
#> [1] 127The default list has 125 words. In a corpus selected for COVID-19,
the words covid and pandemic occur in most texts and
describe no topic, so they are added. The list is passed as
topics(stop_words = ) or
terms(stop_words = ).
The preparation in one sequence
The full preparation of the abstracts, in order, ends with an encoding step that needs a downloaded model and is not run here:
abstracts <- subset(covid, Abstract != "[No abstract available]")
abstracts <- dedupe(abstracts, column = "Abstract")
abstracts <- clean_corpus(abstracts, column = "Abstract",
remove = "(copyright:?\\s*(©|\\d{4})|©).{0,1000}$",
min_content = 0.5)
model_download()
topic_model <- topics(abstracts, n_topics = 10, column = "Abstract",
stop_words = stop_words(add = c("covid", "pandemic")),
numbers = "remove")
topic_sizes(topic_model, weights = "n")| Problem | Verb |
|---|---|
| Placeholder or empty records |
subset() before anything else |
| Repeated texts |
dedupe(), then
topic_sizes(weights = "n")
|
| Markup, URLs, citations, broken characters | clean_corpus() |
| Boilerplate such as copyright notices | clean_corpus(remove = ) |
| List and section numbering |
clean_corpus(), strip_list_markers()
|
| Units that are not prose |
content_ratio(), min_content =
|
| Uninformative words in topic terms |
stop_words(add = ),
numbers = "remove"
|
The prepared abstracts are modeled in Case study: COVID-19 research abstracts, and choosing the unit of analysis is discussed in Choosing the number of topics and the granularity.