In natural language processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we had two aims. (1) We described our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, the Histopathology data). (2) Since the Histopathology data had no classification labels, we gathered public Finnish datasets as proxy data to analyze whether the signals observed in (1) were associated with downstream classification gains. We observed that CPT train-time loss curves differed strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlated with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.
Citation: Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö. Continued pretraining of FinBERT on Finnish histopathological reports: train-time signals and proxy downstream correlations[J]. Applied Computing and Intelligence, 2026, 6(2): 245-264. doi: 10.3934/aci.2026014
In natural language processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we had two aims. (1) We described our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, the Histopathology data). (2) Since the Histopathology data had no classification labels, we gathered public Finnish datasets as proxy data to analyze whether the signals observed in (1) were associated with downstream classification gains. We observed that CPT train-time loss curves differed strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlated with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.
| [1] |
A. Ajanki, Lauseluokitteludatasetti: ministerien vastaukset kirjallisiin kysymyksiin, GitHub, 2019. Available from: https://github.com/aajanki/eduskunta-vkk. |
| [2] |
E. Alsentzer, J. Murphy, W. Boag, W. H. Weng, D. Jindi, T. Naumann, et al., Publicly available clinical bert embeddings, Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, 72–78. https://doi.org/10.18653/v1/W19-1909 |
| [3] |
W. Bani Issa, I. Al Akour, A. Ibrahim, A. Almarzouqi, S. Abbas, F. Hisham, et al., Privacy, confidentiality, security and patient safety concerns about electronic health records, Int. Nurs. Rev., 67 (2020), 218–230. https://doi.org/10.1111/inr.12585 doi: 10.1111/inr.12585
|
| [4] |
Y. Bo, A. Soni, S. Srivastava, M. Khosla, Evaluating representational similarity measures from the lens of functional correspondence, arXiv: 2411.14633. https://doi.org/10.48550/arXiv.2411.14633 |
| [5] |
D. Brandfonbrener, N. Anand, N. Vyas, E. Malach, S. M. Kakade, Loss-to-loss prediction: Scaling laws for all datasets, arXiv: 2411.12925. https://doi.org/10.48550/arXiv.2411.12925 |
| [6] |
N. Bui, G. Nguyen, N. Nguyen, B. Vo, L. Vo, T. Huynh, et al., Fine-tuning large language models for improved health communication in low-resource languages, Comput. Meth. Prog. Bio., 263 (2025), 108655. https://doi.org/10.1016/j.cmpb.2025.108655 doi: 10.1016/j.cmpb.2025.108655
|
| [7] |
A. M. Dai, Q. V. Le, Semi-supervised sequence learning, Proceedings of the 29th International Conference on Neural Information Processing Systems, 2015, 3079–3087. |
| [8] |
J. Devlin, M. W. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, 4171–4186. https://doi.org/10.18653/v1/N19-1423 |
| [9] |
F. Ding, J. S. Denain, J. Steinhardt, Grounding representation similarity through statistical testing, Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021, 1556–1568. |
| [10] |
K. Dwivedi, G. Roig, Representation similarity analysis for efficient task taxonomy & transfer learning, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 12387–12396. https://doi.org/10.1109/CVPR.2019.01267 |
| [11] |
G. Eilertsen, D. Jönsson, T. Ropinski, J. Unger, A. Ynnerman, Classifying the classifier: Dissecting the weight space of neural networks, Proceedings of the 24th European Conference on Artificial Intelligence, 2020, 1119–1126. https://doi.org/10.3233/FAIA200209 |
| [12] |
S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, et al., Don't stop pretraining: adapt language models to domains and tasks, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, 8342–8360. https://doi.org/10.18653/v1/2020.acl-main.740 |
| [13] |
J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, et al., A common, high-dimensional model of the representational space in human ventral temporal cortex, Neuron, 72 (2011), 404–416. https://doi.org/10.1016/j.neuron.2011.08.026 doi: 10.1016/j.neuron.2011.08.026
|
| [14] |
D. Hernandez, J. Kaplan, T. Henighan, S. McCandlish, Scaling laws for transfer, arXiv: 2102.01293. https://doi.org/10.48550/arXiv.2102.01293 |
| [15] |
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, et al., Training compute-optimal large language models, Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, 30016–30030. |
| [16] |
J. Howard, S. Ruder, Universal language model fine-tuning for text classification, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018,328–339. https://doi.org/10.18653/v1/P18-1031 |
| [17] |
L. Hubert, P. Arabie, Comparing partitions, J. Classif., 2 (1985), 193–218. https://doi.org/10.1007/BF01908075 doi: 10.1007/BF01908075
|
| [18] |
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, et al., Scaling laws for neural language models, arXiv: 2001.08361. https://doi.org/10.48550/arXiv.2001.08361 |
| [19] |
I. Keshta, A. Odeh, Security and privacy of electronic health records: concerns and challenges, Egypt. Inform. J., 22 (2021), 177–183. https://doi.org/10.1016/j.eij.2020.07.003 doi: 10.1016/j.eij.2020.07.003
|
| [20] |
S. Kornblith, M. Norouzi, H. Lee, G. Hinton, Similarity of neural network representations revisited, Proceedings of the 36th International Conference on Machine Learning, 2019, 3519–3529. |
| [21] |
N. Kriegeskorte, M. Mur, P. A. Bandettini, Representational similarity analysis-connecting the branches of systems neuroscience, Front. Syst. Neurosci., 2 (2008), 249. https://doi.org/10.3389/neuro.06.004.2008 doi: 10.3389/neuro.06.004.2008
|
| [22] |
J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, et al., BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36 (2020), 1234–1240. https://doi.org/10.1093/bioinformatics/btz682 doi: 10.1093/bioinformatics/btz682
|
| [23] |
B. Li, H. Zhou, J. He, M. Wang, Y. Yang, L. Li, On the sentence embeddings from pre-trained language models, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, 9119–9130. https://doi.org/10.18653/v1/2020.emnlp-main.733 |
| [24] |
P. Lison, J. Tiedemann, OpenSubtitles2016: extracting large parallel corpora from movie and TV subtitles, Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), 2016,923–929. |
| [25] |
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, et al., Roberta: a robustly optimized bert pretraining approach, arXiv: 1907.11692. https://doi.org/10.48550/arXiv.1907.11692 |
| [26] |
J. Luotolahti, J. Kanerva, V. Laippala, S. Pyysalo, F. Ginter, Towards universal web parsebanks, Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), 2015,211–220. |
| [27] |
D. J. MacKay, Information theory, inference and learning algorithms, Cambridge: Cambridge University Press, 2003. \newpage |
| [28] |
N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, D. Sussillo, Universality and individuality in neural dynamics across large populations of recurrent networks, Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, 15629–15641. |
| [29] |
Ministry of Justice, Finland}, Finlex open data—{F}innish legislation and treaties, Finlex Avoin Data portal, 2025. Available from: https://www.finlex.fi/fi/avoin-data/lataa-aineistoja. |
| [30] |
A. Morcos, M. Raghu, S. Bengio, Insights on representational similarity in neural networks with canonical correlation, Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018, 5732–5741. |
| [31] |
J. Mu, P. Viswanath, All-but-the-top: simple and effective postprocessing for word representations, Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018, 1–25. |
| [32] |
E. Myllylä, P. Siirtola, A. Isosalo, J. Reponen, S. Tamminen, O. Laatikainen, Extracting information from unstructured medical reports written in minority languages: a case study of finnish, Data, 10 (2025), 104. https://doi.org/10.3390/data10070104 doi: 10.3390/data10070104
|
| [33] |
K. Nagatsuka, C. Broni-Bediako, M. Atsumi, Length-based curriculum learning for efficient pre-training of language models, New Gener. Comput., 41 (2023), 109–134. https://doi.org/10.1007/s00354-022-00198-8 doi: 10.1007/s00354-022-00198-8
|
| [34] |
M. Nunes, J. Boné, J. C. Ferreira, P. Chaves, L. B. Elvas, Medialbertina: an european portuguese medical language model, Comput. Biol. Med., 182 (2024), 109233. https://doi.org/10.1016/j.compbiomed.2024.109233 doi: 10.1016/j.compbiomed.2024.109233
|
| [35] |
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, et al., Deep contextualized word representations, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018, 2227–2237. https://doi.org/10.18653/v1/N18-1202 |
| [36] |
J. Phang, H. Liu, S. Bowman, Fine-tuned transformers show clusters of similar representations across layers, Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2021,529–538. https://doi.org/10.18653/v1/2021.blackboxnlp-1.42 |
| [37] |
S. J. Prince, Understanding deep learning, Cambridge: MIT press, 2023. |
| [38] |
{Project Gutenberg}, Project gutenberg electronic book collection—{F}innish subset, GitHub, Inc., 2025. Available from: https://gutendex.com. |
| [39] |
H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu, Y. Ma, et al., {D-CPT} law: domain-specific continual pre-training scaling law for large language models, Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024, 90318–90354. |
| [40] |
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding by generative pre-training, Technical report, 2018. |
| [41] |
M. Raghu, J. Gilmer, J. Yosinski, J. Sohl-Dickstein, SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability, Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, 6078–6087. |
| [42] |
A. Rogers, O. Kovaleva, A. Rumshisky, A primer in BERTology: what we know about how BERT works, Transactions of the Association for Computational Linguistics, 8 (2020), 842–866. https://doi.org/10.1162/tacl_a_00349 doi: 10.1162/tacl_a_00349
|
| [43] |
P. J. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J. Comput. Appl. Math., 20 (1987), 53–65. https://doi.org/10.1016/0377-0427(87)90125-7 doi: 10.1016/0377-0427(87)90125-7
|
| [44] |
O. Roy, M. Vetterli, The effective rank: a measure of effective dimensionality, Proceedings of the 15th European Signal Processing Conference, 2007,606–610. |
| [45] |
W. Rudman, N. Gillman, T. Rayne, C. Eickhoff, Isoscore: measuring the uniformity of embedding space utilization, Proceedings of the Findings of the Association for Computational Linguistics: ACL 2022, 2022, 3325–3339. https://doi.org/10.18653/v1/2022.findings-acl.262 |
| [46] |
S. L. Smith, D. H. P. Turban, S. Hamblin, N. Y. Hammerla, Offline bilingual word vectors, orthogonal transformations and the inverted softmax, Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017, 1–10. |
| [47] |
A. Strehl, J. Ghosh, Cluster ensembles—a knowledge reuse framework for combining multiple partitions, J. Mach. Learn. Res., 3 (2003), 583–617. https://doi.org/10.1162/153244303321897735 doi: 10.1162/153244303321897735
|
| [48] |
R. Toivanen, A. Tanskanen, T. Vehviläinen, {Finnish-NLP/mc4\_fi\_cleaned, Hugging Face, 2022. Available from: https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned. |
| [49] |
H. Türkmen, O. Dikenelli, C. Eraslan, M. C. Įallı, S. S. Özbek, BioBERTurk: exploring turkish biomedical language model development strategies in low-resource setting, J. Healthc. Inform. Res., 7 (2023), 433–446. https://doi.org/10.1007/s41666-023-00140-7 doi: 10.1007/s41666-023-00140-7
|
| [50] |
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, 6000–6010. |
| [51] |
A. Virtanen, J. Kanerva, R. Ilo, J. Luoma, J. Luotolahti, T. Salakoski, et al., Multilingual is not enough: BERT for finnish, arXiv: 1912.07076. https://doi.org/10.48550/arXiv.1912.07076 |
| [52] |
B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, 2526–2547. https://doi.org/10.18653/v1/2025.acl-long.127 |
| [53] |
Wikimedia Foundation, Wikimedia downloads, Wikipedia, 2023. Available from: https://dumps.wikimedia.org. |
| [54] |
A. H. Williams, E. Kunz, S. Kornblith, S. Linderman, Generalized shape metrics on neural representations, Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021, 4738–4750. |
| [55] |
J. Wu, S. Saha, Y. Bo, M. Khosla, Measuring the measures: discriminative capacity of representational similarity metrics across model families, arXiv: 2509.04622. https://doi.org/10.48550/arXiv.2509.04622 |
| [56] |
Yleisradio, Yle {F}innish news archive 2011–2018, source version, Kielipankki, 2017. Available from: http://urn.fi/urn:nbn:fi:lb-2017070501. |
Training loss during continued pretraining of FinBERT on various datasets it is more or less "familiar" with. Markers represent epoch boundaries
CKA similarity between the base and CPT models across FinBERT layers, computed on the $\texttt{[CLS]}$ embeddings of set B; higher values indicate greater similarity. Layer 0 (the context-free token embeddings) is omitted as the CKA scores were degenerate
Macro-F1 of the base and CPT models for the logistic regression classifier across the labeled dataset variants, trained on 500 labeled samples
Subset of the correlation heatmap between CPT features and classification task targets across labeled dataset variants
Example correlations between CPT signals and classification improvement. (a) Train-loss relative improvement vs. the logistic-regression macro-F1 error-reduction rate (500 training samples). (b) Isotropy effective-rank delta vs.\ the $ k $-NN accuracy logit delta (1 000 training samples). The vertical dashed lines mark the Histopathology dataset's feature values; they are feature-side references only, as no classification labels exist for that dataset