ML PROJECT · PYTHON
do machines dream
of
unseen labels?
Restricted Boltzmann Machines under extreme label scarcity
Overview
Can unsupervised pretraining help when only a handful of diagnostic labels are available for training? In this experiment, a Restricted Boltzmann Machine (RBM) first learns from unlabelled morphology profiles. Its representation is then used for classification with only a few benign and malignant examples.
With two to three labels per class, the RBM is slightly ahead of direct logistic regression on average. With five to ten labels, this advantage is no longer observed. This result applies to this dataset and the model configurations examined here.
Results
With two labelled cases per class, the RBM representation achieves a mean AUROC of 0.9691; direct logistic regression achieves 0.9582. The difference is 0.0109. As the number of labels increases, direct logistic regression improves in particular, while the RBM curve remains largely flat.
|
Labels
per class |
AUROC | Average Precision | ||
|---|---|---|---|---|
| Logistic | RBM | Logistic | RBM | |
| 2 | 0.9582 | 0.9691 | 0.9392 | 0.9616 |
| 3 | 0.9673 | 0.9729 | 0.9512 | 0.9646 |
| 5 | 0.9758 | 0.9729 | 0.9647 | 0.9646 |
| 10 | 0.9807 | 0.9729 | 0.9730 | 0.9646 |
AUROC: RBM minus direct logistic regression
Full results table: all five models
| Labels/class | Representation + logistic regression | AUROC | AP |
|---|---|---|---|
| 2 | Direct logistic regression | 0.9582 | 0.9392 |
| 2 | Binned logistic regression | 0.9106 | 0.8773 |
| 2 | SVD-32 | 0.7733 | 0.6965 |
| 2 | Autoencoder-32 | 0.8953 | 0.8540 |
| 2 | RBM-32 | 0.9691 | 0.9616 |
| 3 | Direct logistic regression | 0.9673 | 0.9512 |
| 3 | Binned logistic regression | 0.9386 | 0.9145 |
| 3 | SVD-32 | 0.8186 | 0.7539 |
| 3 | Autoencoder-32 | 0.9276 | 0.8982 |
| 3 | RBM-32 | 0.9729 | 0.9646 |
| 5 | Direct logistic regression | 0.9758 | 0.9647 |
| 5 | Binned logistic regression | 0.9598 | 0.9451 |
| 5 | SVD-32 | 0.8725 | 0.8272 |
| 5 | Autoencoder-32 | 0.9518 | 0.9321 |
| 5 | RBM-32 | 0.9729 | 0.9646 |
| 10 | Direct logistic regression | 0.9807 | 0.9730 |
| 10 | Binned logistic regression | 0.9721 | 0.9626 |
| 10 | SVD-32 | 0.9283 | 0.9035 |
| 10 | Autoencoder-32 | 0.9667 | 0.9546 |
| 10 | RBM-32 | 0.9729 | 0.9646 |
Methods
The Wisconsin Diagnostic Breast Cancer dataset (WDBC) contains 569 cases with 30 numeric cell-nucleus features. These features were computed from digitised fine-needle aspirates of breast lesions. This analysis uses the tabular features, not the original images. 1
For each split, 426 cases form the training pool and 143 are held out as test cases. Scaling, discretisation and representation learning are fitted only within the training pool. The test cases are also excluded from unsupervised pretraining.
Continuous features are divided into up to five quantile bins and binarised. A Bernoulli RBM learns joint patterns in these inputs without using diagnostic labels. Its activation probabilities are then used as 32 features for a regularised logistic regression. 2
h j (x) = P(H j = 1 | x) = σ(b j + w j T x)
The classifier receives two, three, five or ten examples of each class. Each configuration is evaluated with 50 random label draws per split. Comparisons include direct continuous logistic regression, binned logistic regression, SVD, autoencoder and RBM. SVD and the autoencoder use 32 latent dimensions, as does the RBM.
Experimental protocol and model parameters
- Data split
- Ten stratified random 75/25 splits, seeds 1000–1009. Each contains 426 training and 143 test cases; each test set has 90 benign and 53 malignant cases. No case identifier is used as an input.
- Label selection
- 50 balanced draws per label budget and split, with no duplicate cases within a draw. Paired across models, not nested across label budgets.
- Direct logistic regression
-
StandardScaleron all unlabelled training cases. 30 continuous features. - Discretisation
-
KBinsDiscretizer(n_bins=5, strategy='quantile'), one-hot encoding; fitted exclusively on the training pool. - SVD
- 32 components on the binary input; subsequently standardised using the training pool.
- RBM
- 32 hidden units, learning rate 0.05, batch size 32, 50 epochs, seeds 2000–2009. Persistent Contrastive Divergence in scikit-learn. 2
- Autoencoder
- Sigmoid encoder with 32 dimensions and a sigmoid decoder. Binary cross-entropy, Adam, learning rate 0.01, weight decay 10 −5, 250 full-batch epochs. One fixed configuration; no comprehensive architecture comparison.
- Classifier
-
All variants use
LogisticRegression(C=0.1, solver='liblinear', max_iter=1000). RBM and autoencoder activations receive no further standardisation. The same C value does not imply the same effective regularisation when feature scales differ. - Evaluation
-
10,000 stored model runs: five models × four label budgets × ten splits × 50 draws. Average Precision was calculated with
average_precision_score; the historical export column is namedauprc. 3
Interpretation and limitations
The result is consistent with a benefit from unsupervised pretraining under extreme label scarcity. The effect size is limited, however. From five labels per class onward, direct logistic regression already has a higher mean AUROC than the RBM. The study therefore does not establish a general RBM advantage or superiority over autoencoders.
Label scarcity is simulated. To draw a balanced labelled set, candidate class identities must already be known. Four released labels therefore do not automatically correspond to the practical effort of diagnosing four arbitrary cases. No actual reduction in annotation time was measured.
WDBC is a small, historical sample. Repeated splits of the same cases are not a substitute for external validation. Discretisation, scaling and fixed regularisation also affect the model comparison. Transfer to other laboratories, specimens or tumour types was not studied.
The dataset was selected as a positive example after several exploratory dataset and task trials. This selection increases the risk of optimistic findings.
The basic idea is not new either: RBM-based methods for learning from few labels in breast-imaging diagnosis have been studied previously. The purpose of this small project is a transparent comparison on tabular data, including the conditions under which the observed advantage disappears. 4
Sources
References
- Wolberg W., Mangasarian O., Street N., Street W. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1993.
- scikit-learn. BernoulliRBM.
- scikit-learn. Average Precision.
- Cao et al. RBM-based oversampling and semi-supervised learning for breast CAD, 2015.
The WDBC dataset is provided under CC BY 4.0.