ML PROJECT · PYTHON

do machines dream
of unseen labels?

Restricted Boltzmann Machines under extreme label scarcity

Overview

Can unsupervised pretraining help when only a handful of diagnostic labels are available for training? In this experiment, a Restricted Boltzmann Machine (RBM) first learns from unlabelled morphology profiles. Its representation is then used for classification with only a few benign and malignant examples.

With two to three labels per class, the RBM is slightly ahead of direct logistic regression on average. With five to ten labels, this advantage is no longer observed. This result applies to this dataset and the model configurations examined here.

01

Results

With two labelled cases per class, the RBM representation achieves a mean AUROC of 0.9691; direct logistic regression achieves 0.9582. The difference is 0.0109. As the number of labels increases, direct logistic regression improves in particular, while the RBM curve remains largely flat.

Figure 1. Performance by number of labels
Fig. 1. Means from ten stratified splits with 50 label draws each. Lines connect separately evaluated label budgets; they do not represent progressive annotation of the same cases. Optional: ± one standard deviation of the ten split means for the RBM and direct logistic regression. This variability is not a confidence interval. The y-axis shows a range from 0.70 to 1.00.
Table 1. Mean results for the main comparison
Labels
per class
AUROC Average Precision
Logistic RBM Logistic RBM
2 0.9582 0.9691 0.9392 0.9616
3 0.9673 0.9729 0.9512 0.9646
5 0.9758 0.9729 0.9647 0.9646
10 0.9807 0.9729 0.9730 0.9646
Figure 2. Differences by split

AUROC: RBM minus direct logistic regression

Fig. 2. Each point is the mean difference for one split across 50 paired label draws. The diamond marks the overall mean. Positive values favour the RBM. This display is descriptive as well.
Full results table: all five models
Alle Label-Budgets und Modelle
Labels/class Representation + logistic regression AUROC AP
2 Direct logistic regression 0.9582 0.9392
2 Binned logistic regression 0.9106 0.8773
2 SVD-32 0.7733 0.6965
2 Autoencoder-32 0.8953 0.8540
2 RBM-32 0.9691 0.9616
3 Direct logistic regression 0.9673 0.9512
3 Binned logistic regression 0.9386 0.9145
3 SVD-32 0.8186 0.7539
3 Autoencoder-32 0.9276 0.8982
3 RBM-32 0.9729 0.9646
5 Direct logistic regression 0.9758 0.9647
5 Binned logistic regression 0.9598 0.9451
5 SVD-32 0.8725 0.8272
5 Autoencoder-32 0.9518 0.9321
5 RBM-32 0.9729 0.9646
10 Direct logistic regression 0.9807 0.9730
10 Binned logistic regression 0.9721 0.9626
10 SVD-32 0.9283 0.9035
10 Autoencoder-32 0.9667 0.9546
10 RBM-32 0.9729 0.9646
02

Methods

The Wisconsin Diagnostic Breast Cancer dataset (WDBC) contains 569 cases with 30 numeric cell-nucleus features. These features were computed from digitised fine-needle aspirates of breast lesions. This analysis uses the tabular features, not the original images. 1

For each split, 426 cases form the training pool and 143 are held out as test cases. Scaling, discretisation and representation learning are fitted only within the training pool. The test cases are also excluded from unsupervised pretraining.

30 morphology features Quantile encoding RBM with 32 units Logistic regression

Continuous features are divided into up to five quantile bins and binarised. A Bernoulli RBM learns joint patterns in these inputs without using diagnostic labels. Its activation probabilities are then used as 32 features for a regularised logistic regression. 2

h j (x) = P(H j = 1 | x) = σ(b j + w j T x)

The classifier receives two, three, five or ten examples of each class. Each configuration is evaluated with 50 random label draws per split. Comparisons include direct continuous logistic regression, binned logistic regression, SVD, autoencoder and RBM. SVD and the autoencoder use 32 latent dimensions, as does the RBM.

Experimental protocol and model parameters
Data split
Ten stratified random 75/25 splits, seeds 1000–1009. Each contains 426 training and 143 test cases; each test set has 90 benign and 53 malignant cases. No case identifier is used as an input.
Label selection
50 balanced draws per label budget and split, with no duplicate cases within a draw. Paired across models, not nested across label budgets.
Direct logistic regression
StandardScaler on all unlabelled training cases. 30 continuous features.
Discretisation
KBinsDiscretizer(n_bins=5, strategy='quantile'), one-hot encoding; fitted exclusively on the training pool.
SVD
32 components on the binary input; subsequently standardised using the training pool.
RBM
32 hidden units, learning rate 0.05, batch size 32, 50 epochs, seeds 2000–2009. Persistent Contrastive Divergence in scikit-learn. 2
Autoencoder
Sigmoid encoder with 32 dimensions and a sigmoid decoder. Binary cross-entropy, Adam, learning rate 0.01, weight decay 10 −5, 250 full-batch epochs. One fixed configuration; no comprehensive architecture comparison.
Classifier
All variants use LogisticRegression(C=0.1, solver='liblinear', max_iter=1000). RBM and autoencoder activations receive no further standardisation. The same C value does not imply the same effective regularisation when feature scales differ.
Evaluation
10,000 stored model runs: five models × four label budgets × ten splits × 50 draws. Average Precision was calculated with average_precision_score; the historical export column is named auprc. 3
03

Interpretation and limitations

The result is consistent with a benefit from unsupervised pretraining under extreme label scarcity. The effect size is limited, however. From five labels per class onward, direct logistic regression already has a higher mean AUROC than the RBM. The study therefore does not establish a general RBM advantage or superiority over autoencoders.

Label scarcity is simulated. To draw a balanced labelled set, candidate class identities must already be known. Four released labels therefore do not automatically correspond to the practical effort of diagnosing four arbitrary cases. No actual reduction in annotation time was measured.

WDBC is a small, historical sample. Repeated splits of the same cases are not a substitute for external validation. Discretisation, scaling and fixed regularisation also affect the model comparison. Transfer to other laboratories, specimens or tumour types was not studied.

The dataset was selected as a positive example after several exploratory dataset and task trials. This selection increases the risk of optimistic findings.

The basic idea is not new either: RBM-based methods for learning from few labels in breast-imaging diagnosis have been studied previously. The purpose of this small project is a transparent comparison on tabular data, including the conditions under which the observed advantage disappears. 4

04

Sources

References

  1. Wolberg W., Mangasarian O., Street N., Street W. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1993.
  2. scikit-learn. BernoulliRBM.
  3. scikit-learn. Average Precision.
  4. Cao et al. RBM-based oversampling and semi-supervised learning for breast CAD, 2015.

The WDBC dataset is provided under CC BY 4.0.

Back