Research article

Continued pretraining of FinBERT on Finnish histopathological reports: train-time signals and proxy downstream correlations


  • Published: 06 November 2026
  • In natural language processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we had two aims. (1) We described our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, the Histopathology data). (2) Since the Histopathology data had no classification labels, we gathered public Finnish datasets as proxy data to analyze whether the signals observed in (1) were associated with downstream classification gains. We observed that CPT train-time loss curves differed strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlated with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.

    Citation: Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö. Continued pretraining of FinBERT on Finnish histopathological reports: train-time signals and proxy downstream correlations[J]. Applied Computing and Intelligence, 2026, 6(2): 245-264. doi: 10.3934/aci.2026014

    Related Papers:

  • In natural language processing (NLP) classification tasks where a lack of labeled data is an issue, continued pretraining (CPT) of transformer models on unlabeled data is an established approach. In this paper, we had two aims. (1) We described our observations from continued pretraining of the Finnish BERT transformer model (FinBERT) on a Finnish histopathological dataset (below, the Histopathology data). (2) Since the Histopathology data had no classification labels, we gathered public Finnish datasets as proxy data to analyze whether the signals observed in (1) were associated with downstream classification gains. We observed that CPT train-time loss curves differed strongly by domain, and that, in an exploratory analysis, certain CPT-derived features correlated with proxy classification improvement. In particular, this report contributes to the limited literature on NLP for Finnish healthcare data.



    加载中


    [1]

    A. Ajanki, Lauseluokitteludatasetti: ministerien vastaukset kirjallisiin kysymyksiin, GitHub, 2019. Available from: https://github.com/aajanki/eduskunta-vkk.

    [2]

    E. Alsentzer, J. Murphy, W. Boag, W. H. Weng, D. Jindi, T. Naumann, et al., Publicly available clinical bert embeddings, Proceedings of the 2nd Clinical Natural Language Processing Workshop, 2019, 72–78. https://doi.org/10.18653/v1/W19-1909

    [3]

    W. Bani Issa, I. Al Akour, A. Ibrahim, A. Almarzouqi, S. Abbas, F. Hisham, et al., Privacy, confidentiality, security and patient safety concerns about electronic health records, Int. Nurs. Rev., 67 (2020), 218–230. https://doi.org/10.1111/inr.12585

    doi: 10.1111/inr.12585
    [4]

    Y. Bo, A. Soni, S. Srivastava, M. Khosla, Evaluating representational similarity measures from the lens of functional correspondence, arXiv: 2411.14633. https://doi.org/10.48550/arXiv.2411.14633

    [5]

    D. Brandfonbrener, N. Anand, N. Vyas, E. Malach, S. M. Kakade, Loss-to-loss prediction: Scaling laws for all datasets, arXiv: 2411.12925. https://doi.org/10.48550/arXiv.2411.12925

    [6]

    N. Bui, G. Nguyen, N. Nguyen, B. Vo, L. Vo, T. Huynh, et al., Fine-tuning large language models for improved health communication in low-resource languages, Comput. Meth. Prog. Bio., 263 (2025), 108655. https://doi.org/10.1016/j.cmpb.2025.108655

    doi: 10.1016/j.cmpb.2025.108655
    [7]

    A. M. Dai, Q. V. Le, Semi-supervised sequence learning, Proceedings of the 29th International Conference on Neural Information Processing Systems, 2015, 3079–3087.

    [8]

    J. Devlin, M. W. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, 4171–4186. https://doi.org/10.18653/v1/N19-1423

    [9]

    F. Ding, J. S. Denain, J. Steinhardt, Grounding representation similarity through statistical testing, Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021, 1556–1568.

    [10]

    K. Dwivedi, G. Roig, Representation similarity analysis for efficient task taxonomy & transfer learning, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 12387–12396. https://doi.org/10.1109/CVPR.2019.01267

    [11]

    G. Eilertsen, D. Jönsson, T. Ropinski, J. Unger, A. Ynnerman, Classifying the classifier: Dissecting the weight space of neural networks, Proceedings of the 24th European Conference on Artificial Intelligence, 2020, 1119–1126. https://doi.org/10.3233/FAIA200209

    [12]

    S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, et al., Don't stop pretraining: adapt language models to domains and tasks, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, 8342–8360. https://doi.org/10.18653/v1/2020.acl-main.740

    [13]

    J. V. Haxby, J. S. Guntupalli, A. C. Connolly, Y. O. Halchenko, B. R. Conroy, M. I. Gobbini, et al., A common, high-dimensional model of the representational space in human ventral temporal cortex, Neuron, 72 (2011), 404–416. https://doi.org/10.1016/j.neuron.2011.08.026

    doi: 10.1016/j.neuron.2011.08.026
    [14]

    D. Hernandez, J. Kaplan, T. Henighan, S. McCandlish, Scaling laws for transfer, arXiv: 2102.01293. https://doi.org/10.48550/arXiv.2102.01293

    [15]

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, et al., Training compute-optimal large language models, Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, 30016–30030.

    [16]

    J. Howard, S. Ruder, Universal language model fine-tuning for text classification, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018,328–339. https://doi.org/10.18653/v1/P18-1031

    [17]

    L. Hubert, P. Arabie, Comparing partitions, J. Classif., 2 (1985), 193–218. https://doi.org/10.1007/BF01908075

    doi: 10.1007/BF01908075
    [18]

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, et al., Scaling laws for neural language models, arXiv: 2001.08361. https://doi.org/10.48550/arXiv.2001.08361

    [19]

    I. Keshta, A. Odeh, Security and privacy of electronic health records: concerns and challenges, Egypt. Inform. J., 22 (2021), 177–183. https://doi.org/10.1016/j.eij.2020.07.003

    doi: 10.1016/j.eij.2020.07.003
    [20]

    S. Kornblith, M. Norouzi, H. Lee, G. Hinton, Similarity of neural network representations revisited, Proceedings of the 36th International Conference on Machine Learning, 2019, 3519–3529.

    [21]

    N. Kriegeskorte, M. Mur, P. A. Bandettini, Representational similarity analysis-connecting the branches of systems neuroscience, Front. Syst. Neurosci., 2 (2008), 249. https://doi.org/10.3389/neuro.06.004.2008

    doi: 10.3389/neuro.06.004.2008
    [22]

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, et al., BioBERT: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics, 36 (2020), 1234–1240. https://doi.org/10.1093/bioinformatics/btz682

    doi: 10.1093/bioinformatics/btz682
    [23]

    B. Li, H. Zhou, J. He, M. Wang, Y. Yang, L. Li, On the sentence embeddings from pre-trained language models, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, 9119–9130. https://doi.org/10.18653/v1/2020.emnlp-main.733

    [24]

    P. Lison, J. Tiedemann, OpenSubtitles2016: extracting large parallel corpora from movie and TV subtitles, Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), 2016,923–929.

    [25]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, et al., Roberta: a robustly optimized bert pretraining approach, arXiv: 1907.11692. https://doi.org/10.48550/arXiv.1907.11692

    [26]

    J. Luotolahti, J. Kanerva, V. Laippala, S. Pyysalo, F. Ginter, Towards universal web parsebanks, Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), 2015,211–220.

    [27]

    D. J. MacKay, Information theory, inference and learning algorithms, Cambridge: Cambridge University Press, 2003. \newpage

    [28]

    N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, D. Sussillo, Universality and individuality in neural dynamics across large populations of recurrent networks, Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, 15629–15641.

    [29]

    Ministry of Justice, Finland}, Finlex open data—{F}innish legislation and treaties, Finlex Avoin Data portal, 2025. Available from: https://www.finlex.fi/fi/avoin-data/lataa-aineistoja.

    [30]

    A. Morcos, M. Raghu, S. Bengio, Insights on representational similarity in neural networks with canonical correlation, Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018, 5732–5741.

    [31]

    J. Mu, P. Viswanath, All-but-the-top: simple and effective postprocessing for word representations, Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018, 1–25.

    [32]

    E. Myllylä, P. Siirtola, A. Isosalo, J. Reponen, S. Tamminen, O. Laatikainen, Extracting information from unstructured medical reports written in minority languages: a case study of finnish, Data, 10 (2025), 104. https://doi.org/10.3390/data10070104

    doi: 10.3390/data10070104
    [33]

    K. Nagatsuka, C. Broni-Bediako, M. Atsumi, Length-based curriculum learning for efficient pre-training of language models, New Gener. Comput., 41 (2023), 109–134. https://doi.org/10.1007/s00354-022-00198-8

    doi: 10.1007/s00354-022-00198-8
    [34]

    M. Nunes, J. Boné, J. C. Ferreira, P. Chaves, L. B. Elvas, Medialbertina: an european portuguese medical language model, Comput. Biol. Med., 182 (2024), 109233. https://doi.org/10.1016/j.compbiomed.2024.109233

    doi: 10.1016/j.compbiomed.2024.109233
    [35]

    M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, et al., Deep contextualized word representations, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018, 2227–2237. https://doi.org/10.18653/v1/N18-1202

    [36]

    J. Phang, H. Liu, S. Bowman, Fine-tuned transformers show clusters of similar representations across layers, Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2021,529–538. https://doi.org/10.18653/v1/2021.blackboxnlp-1.42

    [37]

    S. J. Prince, Understanding deep learning, Cambridge: MIT press, 2023.

    [38]

    {Project Gutenberg}, Project gutenberg electronic book collection—{F}innish subset, GitHub, Inc., 2025. Available from: https://gutendex.com.

    [39]

    H. Que, J. Liu, G. Zhang, C. Zhang, X. Qu, Y. Ma, et al., {D-CPT} law: domain-specific continual pre-training scaling law for large language models, Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024, 90318–90354.

    [40]

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding by generative pre-training, Technical report, 2018.

    [41]

    M. Raghu, J. Gilmer, J. Yosinski, J. Sohl-Dickstein, SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability, Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, 6078–6087.

    [42]

    A. Rogers, O. Kovaleva, A. Rumshisky, A primer in BERTology: what we know about how BERT works, Transactions of the Association for Computational Linguistics, 8 (2020), 842–866. https://doi.org/10.1162/tacl_a_00349

    doi: 10.1162/tacl_a_00349
    [43]

    P. J. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J. Comput. Appl. Math., 20 (1987), 53–65. https://doi.org/10.1016/0377-0427(87)90125-7

    doi: 10.1016/0377-0427(87)90125-7
    [44]

    O. Roy, M. Vetterli, The effective rank: a measure of effective dimensionality, Proceedings of the 15th European Signal Processing Conference, 2007,606–610.

    [45]

    W. Rudman, N. Gillman, T. Rayne, C. Eickhoff, Isoscore: measuring the uniformity of embedding space utilization, Proceedings of the Findings of the Association for Computational Linguistics: ACL 2022, 2022, 3325–3339. https://doi.org/10.18653/v1/2022.findings-acl.262

    [46]

    S. L. Smith, D. H. P. Turban, S. Hamblin, N. Y. Hammerla, Offline bilingual word vectors, orthogonal transformations and the inverted softmax, Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017, 1–10.

    [47]

    A. Strehl, J. Ghosh, Cluster ensembles—a knowledge reuse framework for combining multiple partitions, J. Mach. Learn. Res., 3 (2003), 583–617. https://doi.org/10.1162/153244303321897735

    doi: 10.1162/153244303321897735
    [48]

    R. Toivanen, A. Tanskanen, T. Vehviläinen, {Finnish-NLP/mc4\_fi\_cleaned, Hugging Face, 2022. Available from: https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned.

    [49]

    H. Türkmen, O. Dikenelli, C. Eraslan, M. C. Įallı, S. S. Özbek, BioBERTurk: exploring turkish biomedical language model development strategies in low-resource setting, J. Healthc. Inform. Res., 7 (2023), 433–446. https://doi.org/10.1007/s41666-023-00140-7

    doi: 10.1007/s41666-023-00140-7
    [50]

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, 6000–6010.

    [51]

    A. Virtanen, J. Kanerva, R. Ilo, J. Luoma, J. Luotolahti, T. Salakoski, et al., Multilingual is not enough: BERT for finnish, arXiv: 1912.07076. https://doi.org/10.48550/arXiv.1912.07076

    [52]

    B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, 2526–2547. https://doi.org/10.18653/v1/2025.acl-long.127

    [53]

    Wikimedia Foundation, Wikimedia downloads, Wikipedia, 2023. Available from: https://dumps.wikimedia.org.

    [54]

    A. H. Williams, E. Kunz, S. Kornblith, S. Linderman, Generalized shape metrics on neural representations, Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021, 4738–4750.

    [55]

    J. Wu, S. Saha, Y. Bo, M. Khosla, Measuring the measures: discriminative capacity of representational similarity metrics across model families, arXiv: 2509.04622. https://doi.org/10.48550/arXiv.2509.04622

    [56]

    Yleisradio, Yle {F}innish news archive 2011–2018, source version, Kielipankki, 2017. Available from: http://urn.fi/urn:nbn:fi:lb-2017070501.

  • Reader Comments
  • © 2026 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)

Metrics

Article views(120) PDF downloads(6) Cited by(0)

Article outline

Figures and Tables

Figures(5)  /  Tables(4)

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog