
Verification: Python parity and determinism
Written by Mohammed Saqr and Sonsoles López-Pernas
Source:vignettes/articles/verification.Rmd
verification.RmdA reimplementation of a published method is trustworthy only to the
extent that it has been checked against the original. sbert
computes Sentence-BERT embeddings (Reimers & Gurevych, 2019) without
Python: a Hugging Face tokenizer, an ONNX export of the network, and
pooling and normalization in base R. Each of these steps can drift from
the reference Python implementation, SentenceTransformers, in ways that
are invisible in a single result: a tokenizer that splits accented text
differently, a pooling step that counts padding tokens, a normalization
applied twice. Topic models built on the embeddings add a second
requirement. An analysis reported in a paper must give the same topics
when it is run again, by the authors or by a reviewer, which rules out
any hidden dependence on random seeds, processor count or the order in
which parallel work finishes.
This guide states what sbert claims on both counts and
how each claim is checked: whether a fit is reproducible, how the
package’s tests lock the modeled output, how embeddings are compared
with SentenceTransformers, how a fit that did not converge is reported,
and how errors and warnings can be caught by class.
library(sbert)
#> sbert 0.6.0: Sentence-BERT embeddings and topic models in R.
#> 14 pinned models (see models()); default: all-MiniLM-L6-v2.
#> A model is downloaded only on request, once, and verified by SHA-256.Data
The checks that need no model run on the bundled feedback messages.
-
corpus: the first 600 distinct English feedback messages (translation), with their source-language text (feedback) and number of occurrences (n). -
feedback_embeddings: theirall-MiniLM-L6-v2embeddings, a 600 × 384 matrix with one row per message, computed once with the package and bundled so that the topic model can be fitted without a model.
Is a topic model reproducible?
The topic model is deterministic by construction. The k-means starting centers are chosen by a farthest-point rule from the data, not drawn at random, and no other step of the fit uses random numbers. The claim is checked by fitting the same model twice.
first_fit <- topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings)
second_fit <- topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings)
identical(first_fit, second_fit)
#> [1] TRUE
identical(as.data.frame(first_fit), as.data.frame(second_fit))
#> [1] TRUEThe two fits are identical as whole objects, including the document
table, the terms, the representatives and the topic centers, not merely
equal within a tolerance. No set.seed() was needed. The
same holds for the number of cores: the guide to
parallel topic modeling shows compare_topics() with one
and with two cores giving identical results.
What this check establishes is reproducibility on one machine with one build of R. Floating-point sums can differ in their last bits between linear algebra libraries (the reference BLAS, OpenBLAS, Apple’s Accelerate), so the distances and centers of a fit computed on another platform may differ at the level of 1e-15. The tests described next allow for this: they hold the topic assignments to exact equality and the numbers to a small tolerance.
How do the package tests lock the output?
The determinism check above is one of two in the package’s test file
tests/testthat/test-reproducibility.R. The other compares a
fresh fit with a golden fixture, a fitted model saved once in
tests/testthat/fixtures/topic_model_golden.rds.
-
Rerun test. Two fits of six topics to the bundled
feedback embeddings must be
identical(): documents, topics, terms, representatives and centers. It runs everywhere, including on CRAN. - Golden test. A fit of six topics to the same embeddings must match the fixture: the topic of every document exactly, and the distances, topic table, terms, representatives and centers to a tolerance of 1e-10. The tolerance allows for the platform differences above, which the test file records as about 2e-15 in relative terms, and is still far too small for any change to the method to pass. It is skipped on CRAN, whose machines differ from the one that wrote the fixture.
The rerun test proves that a fit does not depend on chance; the golden test proves that it does not depend on the version of the package. A change to the code that alters any topic assignment fails the golden test, and if the change is intended, the fixture must be regenerated on purpose and the change noted in the package news.
Do the embeddings match SentenceTransformers?
Parity with the reference implementation is checked for the default
model, all-MiniLM-L6-v2, in
tests/testthat/test-integration.R. The test needs the
model’s files, so it runs only when the environment variable
SBERT_TEST_MODEL_DIR names a directory that holds
model.onnx and tokenizer.json, and when ONNX
Runtime is installed; otherwise it is skipped. It encodes four sentences
chosen to exercise the tokenizer:
-
"This is an example sentence"and"Each sentence is converted.", the standard SentenceTransformers examples; -
"", an empty string, which must still give a valid vector; -
"Café déjà vu — 東京", accented Latin, an em dash and Japanese characters.
The test checks, in order:
- the token ids of all four sentences equal reference ids stored in the test, exactly;
- the embeddings form a 4 × 384 matrix with no missing or infinite values;
- every row has unit length, to a tolerance of 1e-6;
- the first six values of each embedding match reference values stored in the test, with a largest absolute difference below 1e-6;
- if
SBERT_PYTHON_REFERENCEnames a CSV file of the full Python embeddings (no header, one row per sentence, 384 columns), the whole matrix matches it with a largest absolute difference below 1e-6.
The Python reference can be produced with a script along these lines
(not run here), which pins the same revision as the package (see
models(detail = TRUE)):
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2",
revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
)
sentences = ["This is an example sentence", "Each sentence is converted.",
"", "Café déjà vu — 東京"]
embeddings = model.encode(sentences, normalize_embeddings=True)
np.savetxt("reference.csv", embeddings, delimiter=",")The same comparison can be made by hand in R once the model is installed. It needs the model, so it is not run here.
model <- load_model("all-MiniLM-L6-v2")
sentences <- c("This is an example sentence", "Each sentence is converted.",
"", "Café déjà vu — 東京")
r_embeddings <- encode(sentences, model)
python_embeddings <- as.matrix(read.csv("reference.csv", header = FALSE))
max(abs(r_embeddings - python_embeddings))A largest difference below 1e-6, the tolerance the test uses, means that the R and Python embeddings agree to the precision of the single-precision arithmetic in which the network runs. To run the package’s own test against a local copy of the model, from the package source:
Sys.setenv(SBERT_TEST_MODEL_DIR = "path/to/all-MiniLM-L6-v2",
SBERT_PYTHON_REFERENCE = "path/to/reference.csv")
testthat::test_file("tests/testthat/test-integration.R")Parity is claimed for the pinned models of the registry, whose
settings (pooling, prefix, maximum length) are fixed by the package. A
model loaded with load_custom() carries whatever
configuration its repository declares, and the package makes no parity
claim for it; the comparison above, with the custom model’s id, is how
to establish one (see models, the cache,
and custom models).
Did the clustering converge?
k-means is iterative. It stops when no document changes topic, or
when it reaches iter_max iterations (default 100). A fit
stopped by the limit is a valid partition but not a converged one, and
reporting it as a result would be an error the reader cannot see.
sbert therefore never stops silently: a fit that reaches
iter_max raises a warning of class
sbert_no_converge and records the fact in the model. Two
iterations are too few for the feedback messages:
short_fit <- topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings, iter_max = 2)
#> Warning: k-means did not converge in 2 iterations; the topics are returned as they stood.
#> Raise `iter_max` to continue the fit.
summary(short_fit)
#> Semantic topic model summary
#> documents: 600
#> topics: 6
#> model: precomputed embeddings
#> between/total SS: 17.0%
#> mean npmi coherence: -0.1476
#> topic diversity: 0.883 (top 10 terms)
#> k-means: did not converge (raise iter_max)
#>
#> topic label n_documents proportion
#> 1 Where is the cow and the cake? Choose all the pictures... 322 0.53667
#> 2 What does “all the ones between the leaf and the... 141 0.23500
#> 3 All the ones between the deer and the cat. 57 0.09500
#> 4 There should be two fewer in the blue one than in the... 33 0.05500
#> 5 It is not a person, does not have a hat, and is sitting. 24 0.04000
#> 6 What does it mean to be on the grass? 23 0.03833
#> coherence
#> -0.3377
#> -0.5833
#> -0.2617
#> 0.2728
#> 0.1849
#> -0.1606The warning names the limit that was reached, and the fit is still returned so that it can be inspected. The summary repeats the problem in its overview (“did not converge (raise iter_max)”), so it is not lost when the warning scrolls away or the model is saved and reloaded. With the default limit the same model converges, and its summary carries no such line. The difference is not cosmetic: stopped after two iterations, the topics hold 17.0% of the embedding variance between them, against 22.6% for the converged fit below, and the largest topic has 322 messages instead of 223.
summary(first_fit)
#> Semantic topic model summary
#> documents: 600
#> topics: 6
#> model: precomputed embeddings
#> between/total SS: 22.6%
#> mean npmi coherence: -0.1044
#> topic diversity: 0.917 (top 10 terms)
#>
#> topic label n_documents proportion
#> 1 Where is the cow and the cake? Choose all the pictures... 223 0.37167
#> 2 All the ones between the cat and the strawberry. 133 0.22167
#> 3 What does “all the ones between the leaf and the... 131 0.21833
#> 4 Are you sure you have fewer pictures in the red box than... 49 0.08167
#> 5 In the top middle box, it means something is in the... 37 0.06167
#> 6 It is not a person, does not have a hat, and is sitting. 27 0.04500
#> coherence
#> -0.2368
#> -0.4150
#> -0.2564
#> 0.3011
#> -0.2043
#> 0.1849In compare_topics(), where candidates may be fitted in
parallel worker processes that cannot pass warnings back, the same
information is the converged column of the comparison
table, one value per candidate.
Catching errors and warnings by class
Every error that sbert raises on purpose has a class
that names the problem, and every such class inherits from
sbert_error; warnings inherit from
sbert_warning. A script can therefore react to a specific
problem without matching the text of a message, which may change between
versions. The classes include sbert_bad_input,
sbert_no_embeddings, sbert_dimension_mismatch,
sbert_model_not_installed,
sbert_runtime_missing, sbert_unknown_model,
sbert_cache_invalid, sbert_download_failed,
sbert_model_error, sbert_no_terms,
sbert_clustering_failed and
sbert_missing_package for errors, and
sbert_no_converge, sbert_few_representatives
and sbert_cache_rebuilt for warnings. The help page of each
function lists the conditions it raises.
A model fitted with keep_embeddings = FALSE is smaller,
but verbs that need the embeddings must then be given them again.
topic_membership() without them raises
sbert_no_embeddings, which a handler for that class can
catch:
lean_model <- topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings, keep_embeddings = FALSE)
tryCatch(topic_membership(lean_model),
sbert_no_embeddings = function(condition) conditionMessage(condition))
#> [1] "The model stores no embeddings. Refit with keep_embeddings = TRUE or supply embeddings."
tryCatch(topic_membership(lean_model),
sbert_error = function(condition) class(condition))
#> [1] "sbert_no_embeddings" "sbert_error" "error"
#> [4] "condition"The first handler catches the specific class and returns the message;
the second catches the parent class sbert_error and shows
the full class vector, from the specific class to R’s generic
error and condition. Supplying the embeddings
resolves the problem:
head(topic_membership(lean_model, embeddings = feedback_embeddings), 3)
#> document_id topic probability rank
#> 1 1 3 0.8222 1
#> 2 1 5 0.0924 2
#> 3 1 4 0.0766 3Warnings are caught the same way. A pipeline that must not continue
with an unconverged model can turn sbert_no_converge into a
stop, here by returning the class of the warning instead of the
model:
tryCatch(
topics(corpus, n_topics = 6, column = "translation",
embeddings = feedback_embeddings, iter_max = 2),
sbert_no_converge = function(condition) class(condition)
)
#> [1] "sbert_no_converge" "sbert_warning" "warning" "condition"tryCatch() stops the fit at the warning, so no model is
returned. To log the warning and keep the model,
withCallingHandlers() with the same class is the tool.
What is verified, and where
- Same input, same topics on one machine: the rerun check above and the rerun test, at zero tolerance.
- Same topics across package versions: the golden test, exact for assignments and to 1e-10 for every number.
- Same topics with any number of cores: see parallel topic modeling.
- Same embeddings as SentenceTransformers for
all-MiniLM-L6-v2: the integration test, to 1e-6, when the model is supplied. - A fit that did not converge: a
sbert_no_convergewarning, and a line insummary(). - Programmatic handling: the classed errors and warnings above.
References
Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019, 3982–3992. https://doi.org/10.18653/v1/D19-1410