
Sentence embeddings without Python
Written by Mohammed Saqr and Sonsoles López-Pernas
Source:vignettes/embeddings.Rmd
embeddings.RmdA sentence embedding is a fixed-length vector that represents the meaning of a whole text. Sentence-BERT (Reimers & Gurevych, 2019) obtains such vectors from a pretrained transformer that is fine-tuned in a siamese arrangement: pairs of sentences pass through the same network, and the network is trained so that the cosine similarity of their pooled vectors tracks how similar the sentences are in meaning. The result is a space in which “Choose the pictures between the cat and the dog” and “Pick all the images that lie between them” are neighbors although they share few words, so that similarity search, clustering and topic modeling can operate on meaning rather than on vocabulary.
In R, sentence embeddings have usually been computed by calling the
Python sentence-transformers library, which brings a second
language runtime, a deep-learning framework and its version conflicts
into an R analysis. The sbert package computes the same
embeddings in R. It tokenizes with each model’s own Hugging Face
tokenizer through the tok package, runs the transformer as
an ONNX graph through the onnxr package, and pools and
normalizes the result in base R. Models are pinned to fixed revisions
and verified by checksum, and nothing is downloaded unless it is
requested.
Computing an embedding needs a downloaded model, so the encoding steps below are shown and not run. Everything that runs uses the embeddings bundled with the package, which were computed by those same steps.
library(sbert)
#> sbert 0.6.0: Sentence-BERT embeddings and topic models in R.
#> 14 pinned models (see models()); default: all-MiniLM-L6-v2.
#> A model is downloaded only on request, once, and verified by SHA-256.Data
-
feedback_translations: AI-generated feedback messages from a mathematics learning application, with the source-language text infeedbackand the English translation intranslation. -
feedback_embeddings: the embeddings of the first 600 distinct translations, computed with the pinnedall-MiniLM-L6-v2model. Row i is the embedding of row i ofcorpusbelow.
corpus <- head(dedupe(feedback_translations, column = "translation"), 600)
dim(feedback_embeddings)
#> [1] 600 384Each of the 600 messages is a vector of 384 numbers, one per dimension of the model.
How is a sentence turned into a vector?
encode() takes a character vector and returns a matrix
with one row per text. Between the two, each batch of texts passes
through four steps:
-
Tokenization. The model’s
tokenizer.jsonis loaded withtokand splits each text into subword tokens, adds the model’s special tokens, truncates at the model’s maximum length, and pads the batch to its longest member. It returns the token ids and an attention mask that is 1 for real tokens and 0 for padding. -
Inference. The ONNX graph, run by
onnxr, maps the token ids to one contextual vector per token: an array of texts by tokens by dimensions. - Pooling. The token vectors of each text are combined into one vector. Mean pooling averages the vectors of the real tokens, , where is the vector of token and its mask value; CLS pooling takes the vector of the first token. Each model uses the pooling it was trained with.
- Normalization. Each vector is divided by its Euclidean length, so that the dot product of two embeddings is their cosine similarity.
Steps 3 and 4 are plain R arithmetic, exported as
pool(). A small example with two texts, three token
positions and two dimensions shows what they compute. In the mask below,
the third token position of the first text is padding:
token_vectors <- array(1:12, dim = c(2, 3, 2))
mask <- matrix(c(1, 1, 0,
1, 1, 1), nrow = 2, byrow = TRUE)
pool(token_vectors, mask, normalize = FALSE)
#> [,1] [,2]
#> [1,] 2 8
#> [2,] 4 10The first text has the token vectors (1, 7), (3, 9) and (5, 11), and its third token is masked, so its mean is (2, 8). The second text has three real tokens, (2, 8), (4, 10) and (6, 12), with mean (4, 10). The masked token is excluded from the average, which is why padding a batch to its longest member does not change any text’s embedding. CLS pooling returns the first token of each text, and normalization rescales each row to length 1:
pool(token_vectors, mask, normalize = FALSE, method = "cls")
#> [,1] [,2]
#> [1,] 1 7
#> [2,] 2 8
pool(token_vectors, mask)
#> [,1] [,2]
#> [1,] 0.243 0.970
#> [2,] 0.371 0.928The bundled embeddings were normalized in the same way; their row lengths are 1 up to the four-decimal rounding of the stored values:
Which models can be used?
models() lists the pinned models. It reads the registry
inside the package and needs no network:
models()
#> model dimensions max_tokens languages size_mb
#> 1 all-MiniLM-L6-v2 384 256 English 90.9
#> 2 all-MiniLM-L12-v2 384 128 English 133.6
#> 3 paraphrase-MiniLM-L3-v2 384 128 English 69.5
#> 4 multi-qa-MiniLM-L6-cos-v1 384 512 English 90.9
#> 5 paraphrase-multilingual-MiniLM-L12-v2 384 128 50+ languages 479.4
#> 6 all-mpnet-base-v2 768 384 English 436.3
#> 7 paraphrase-multilingual-mpnet-base-v2 768 128 50+ languages 1119.2
#> 8 bge-small-en-v1.5 384 512 English 133.8
#> 9 bge-base-en-v1.5 768 512 English 436.5
#> 10 multilingual-e5-small 384 512 100+ languages 487.4
#> 11 nomic-embed-text-v1.5 768 8192 English 548.0
#> 12 jina-embeddings-v2-small-en 512 8192 English 130.5
#> 13 mxbai-embed-large-v1 1024 512 English 1337.6
#> 14 potion-base-8M 256 1000000 English 30.9The registry holds 14 models, from 30.9 MB to 1,337.6 MB on disk,
with 256 to 1,024 dimensions. Most are English; two
paraphrase-multilingual models cover more than 50 languages
and multilingual-e5-small more than 100. The default,
all-MiniLM-L6-v2, is the model of the bundled embeddings.
detail = TRUE adds the settings that encode()
applies for each model:
subset(models(detail = TRUE), select = c(model, pooling, prefix, engine))
#> model pooling prefix engine
#> 1 all-MiniLM-L6-v2 mean onnx
#> 2 all-MiniLM-L12-v2 mean onnx
#> 3 paraphrase-MiniLM-L3-v2 mean onnx
#> 4 multi-qa-MiniLM-L6-cos-v1 mean onnx
#> 5 paraphrase-multilingual-MiniLM-L12-v2 mean onnx
#> 6 all-mpnet-base-v2 mean onnx
#> 7 paraphrase-multilingual-mpnet-base-v2 mean onnx
#> 8 bge-small-en-v1.5 cls onnx
#> 9 bge-base-en-v1.5 cls onnx
#> 10 multilingual-e5-small mean query: onnx
#> 11 nomic-embed-text-v1.5 mean search_document: onnx
#> 12 jina-embeddings-v2-small-en mean onnx
#> 13 mxbai-embed-large-v1 cls onnx
#> 14 potion-base-8M mean staticThree models use CLS pooling and the rest mean pooling. Two require a
text prefix, which encode() adds automatically
("query: " for multilingual-e5-small and
"search_document: " for
nomic-embed-text-v1.5). potion-base-8M is a
static model: its embedding is a lookup and average of token vectors in
R, with no ONNX Runtime involved, intended for corpora too large to
encode with a transformer.
Where are models stored, and when are they downloaded?
Models are stored in a per-user cache directory, located through
tools::R_user_dir(). cache_dir() returns that
path and creates nothing:
cache_dir()
#> [1] "/home/runner/.cache/R/sbert"model_status() reports, for each file of a model,
whether it is in the cache and whether it passes its pinned byte-size
and SHA-256 checks:
status <- model_status()
subset(status, select = c(file, exists, valid, expected_bytes))
#> file exists valid expected_bytes
#> 1 model.onnx TRUE TRUE 90405214
#> 2 tokenizer.json TRUE TRUE 466247On the machine that built this page, the default model is installed
and valid. The two files a transformer model needs are the ONNX graph
(90,405,214 bytes for all-MiniLM-L6-v2) and the
tokenizer.
Nothing is downloaded when the package is installed, loaded, tested
or when this vignette is built. Two downloads are explicit and happen
once: the native ONNX Runtime used by onnxr, and each
model. The chunk below is not run here:
install_runtime()
model_download("all-MiniLM-L6-v2")After that, encode() computes embeddings. The bundled
matrix was computed this way from the 600 messages of
corpus and then rounded to four decimals. The first use of
a model that is not yet installed asks before downloading it in an
interactive session; scripts opt in with
options(sbert.download = TRUE) or call
model_download() first. load_model() loads a
model explicitly, for example to choose the number of inference threads,
and the loaded model is passed to encode():
embeddings <- with(corpus, encode(translation))
model <- load_model("all-MiniLM-L6-v2", threads = 4)
embeddings <- with(corpus, encode(translation, model = model))How similar are two messages?
Because the rows are normalized, the cosine similarity of two
messages is the dot product of their embeddings.
topic_similarity() computes it for every pair of rows, and
tidy = TRUE returns one row per pair. The first four
messages of the corpus are:
head(corpus, 4)
#> feedback
#> 1 Víš, co znamená o jeden více?
#> 2 V červeném má být o dva více než v modrém.
#> 3 Když to není člověk, tak co to může být?
#> 4 Co znamená všechny za prvním?
#> translation n
#> 1 Do you know what “one more” means? 2
#> 2 There should be two more in the red than in the blue. 3
#> 3 If it’s not a person, what could it be? 2
#> 4 What does “everyone after the first one” mean? 1Their pairwise similarities, each pair once:
pairs <- topic_similarity(head(feedback_embeddings, 4), tidy = TRUE)
subset(pairs, from < to)
#> from to similarity
#> 5 1 2 0.3613
#> 9 1 3 0.0954
#> 10 2 3 0.0935
#> 13 1 4 0.4894
#> 14 2 4 0.2236
#> 15 3 4 0.2177Messages 1 and 4 are the most similar pair (0.489): both ask the learner what a phrase means (“one more”, “everyone after the first one”). Message 3, about whether something is a person, is the least similar to messages 1 and 2 (0.095 and 0.094), while messages 1 and 2, both about a difference in quantity, reach 0.361. Similarity follows what a message does and what it is about, not the words it shares.
Which messages are closest to a given message?
Similarity search scores every text of a corpus against one query. The query here is the second message, a correction about the counts in two boxes. Its embedding is taken from the bundled matrix, its similarity to every message is added to the corpus as a column, and the messages above 0.8 are kept:
query_text <- "There should be two more in the red than in the blue."
query <- subset(feedback_embeddings, with(corpus, translation == query_text))
neighbors <- transform(corpus,
similarity = topic_similarity(feedback_embeddings, query))
subset(neighbors, similarity > 0.8, select = c(translation, similarity))
#> translation
#> 2 There should be two more in the red than in the blue.
#> 9 If there are supposed to be two more in the blue one, how many have to be in the red one?
#> 10 There should be two fewer in the red one than in the blue one.
#> 32 There should be more in blue than in red.
#> 96 If there should be more in the blue one, how many should there be in the red one?
#> 456 There should be fewer in the red one than in the blue one.
#> 469 There should be two more in the red box than in the blue box.
#> 475 There should be two fewer in the blue one than in the red one.
#> similarity
#> 2 1.000
#> 9 0.870
#> 10 0.898
#> 32 0.859
#> 96 0.825
#> 456 0.843
#> 469 0.860
#> 475 0.898The query matches itself with similarity 1, and the closest other messages are corrections of the same exercise, phrased with “more” or “fewer”, “box” or “one”, and as statements or questions. The list also shows what embeddings do not separate well: “There should be two fewer in the red one than in the blue one” scores 0.898, as high as any other message, although it asks for the opposite arrangement, and “There should be more in blue than in red” scores 0.859. Sentence embeddings capture topic and structure more reliably than the direction of a comparison or a negation, so a search that must respect such distinctions needs a check on the text itself.
To search with a new text rather than a message of the corpus, the query is encoded with the same model (not run here):
query <- encode("Count the pictures in each box again.")
neighbors <- transform(corpus,
similarity = topic_similarity(feedback_embeddings, query))When to use which
| Task | Function |
|---|---|
| List the pinned models and their settings |
models(), models(detail = TRUE)
|
| Install the runtime and a model, once |
install_runtime(), model_download()
|
| Check or locate the cache |
model_status(), cache_dir(),
cache_size(), model_remove()
|
| Load a model with chosen threads or backend | load_model() |
| Load an ONNX model outside the registry | load_custom() |
| Compute embeddings | encode() |
| Pool token vectors yourself | pool() |
| Compare embeddings | topic_similarity() |
| Group texts into topics |
topics(), compare_topics()
|
The main vignette fits topic models on the bundled embeddings, and the guide to models, the cache, and custom models covers the cache, custom models and inference settings in detail. Verification: Python parity and determinism compares the embeddings with those of the Python library.
References
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019, 3982–3992. https://doi.org/10.18653/v1/D19-1410