Are Natural Language Processing methods applicable to EPS forecasting in Poland?

Abbreviations: MAAPE: mean arctangent absolute percentage error; NLP: natural language processing; SRW: seasonal random walk

1.   Introduction

The valuation of company stocks critically depends on multiplying the earnings per share (EPS) by the price to earnings ratio, a fundamental aspect of investment analysis. Precise forecasting of these elements, particularly EPS, is crucial, as they provide key numerical insights into a company's future direction, offering valuable data on potential market valuation and guiding auditing expectations. While extensive financial analyst coverage is common for companies in developed markets, companies from emerging markets receive much less attention.

The Polish stock market, following its integration into the European Union after 2004, has demonstrated significant growth and development. As of the end of 2021, the market capitalization reached ＄197 billion, with 774 companies listed on the Warsaw Stock Exchange (WSE). However, despite this substantial market size, analyst coverage for Polish stocks remains limited compared with more developed markets like the United States or Western Europe. Approximately 20% of the 711 companies listed in 2019 received analyst attention, highlighting a significant gap in market coverage. This limited analyst coverage underscores the need for alternative methods, such as statistical or machine learning models, to forecast crucial financial metrics like EPS. Accurate EPS predictions are vital for investors, as they provide insights into a company's future performance and guide investment decisions. In the absence of extensive analyst coverage, the application of natural language processing (NLP) techniques to publicly available financial reports offer a promising avenue for improving EPS forecasting in the Polish market.

The Polish stock market's unique characteristics, including its size, structure, and limited analyst coverage, make it an ideal candidate for exploring the potential of NLP in EPS forecasting. The market's rapid growth and increasing importance within the European Union further emphasize the need for accurate and reliable forecasting methods. By leveraging the vast amount of textual data available in financial reports, NLP techniques can extract valuable insights and improve the accuracy of EPS predictions. Moreover, the successful application of NLP in EPS forecasting for the Polish market could have significant implications for other emerging markets with similar characteristics. Many developing countries face challenges related to limited analyst coverage, and the insights gained from this study could provide a framework for improving financial forecasting in these markets. Additionally, the findings of this research may also be relevant for smaller companies in developed markets that receive less analyst attention.

This article undertakes a comparative analysis of various models using a comprehensive set of explanatory variables derived from two different NLP text vectorization methods, namely FastText and FinBERT, applied to companies' public reports, employing the gradient-boosting decision tree (XGBoost) machine learning algorithm. In recent years, NLP techniques have been employed to derive new numerical features from textual documents in the fields of accounting and finance. These textual features have proven effective in predicting financial variables (Rawte et al., 2021; Medya et al., 2022), and models based on them can achieve substantial accuracy (Rao et al., 2022). However, existing research has not focused on EPS prediction using NLP tools but rather on predicting price movements. Closely related research by Ishikawa et al. (2020) applied NLP tools to forecast analysts' net income estimates, while Medya et al. (2022) demonstrated that the semantic features of financial texts were more predictive of stock price movements than traditional hard data, such as sales and EPS, in most cases. This study covers quarterly EPS data for 267 companies listed on the Polish stock exchange from the 2008–2009 financial crisis through to the 2020 pandemic.

Instead of solely using the conventional mean absolute percentage error, which can be distorted by small denominators, this study utilized an alternative measure, the mean arctangent absolute percentage error (MAAPE), as proposed by Kim and Kim (2016).

In summary, this article pursues multiple objectives: evaluating the performance of an advanced machine learning technique of gradient-boosted trees, using NLP-extracted features in EPS prediction for the Polish market, and applying diverse error metrics, varying timeframes, and statistical tests to validate the experimental outcomes. Additionally, it aimed to clarify the practical implications of these findings for investment strategies in Polish stocks.

2.   Literature review

Algorithmic forecasting of EPS began in the 1960s, leading to scholarly investigation centered primarily on autoregressive integrated moving average (ARIMA) models (Ball and Watts, 1972; Watts, 1975; Griffin, 1977; Foster, 1977; Brown and Rozeff, 1979). These models were the primary focus, with the research findings showing variability: some studies advocated for the simplicity of the basic random walk model, suggesting that more complex models did not consistently outperform it, while others reported divergent conclusions. A similar examination of the Polish market was conducted by Kuryłek (2023a).

Over time, a preference emerged for ARIMA-type models due to their typically accurate forecasts (Lorek, 1979; Bathke and Lorek, 1984). This preference remained until the late 1980s, when a prevailing belief emerged that financial analysts' forecasts surpassed those generated by time series models (Brown et al., 1987). Conroy and Harris (1987) noted the superior performance of analysts' forecasts for short-term horizons, which diminished over longer periods. This view persisted until recent years, when the superiority of analysts over time series models was once again questioned (Lacina et al., 2011; Bradshaw et al., 2012; Pagach and Warr, 2020; Gatsios et al., 2021).

Since the late 1960s, various approaches employing exponential smoothing for EPS prediction have also been explored (Elton and Gruber, 1972; Ball and Watts, 1972; Johnson and Schmitt, 1974; Brooks and Buckmaster, 1976; Ruland, 1980; Brandon et al., 1987; Jarrett, 2008), producing mixed results. Research for the Polish market was undertaken by Kuryłek (2023b).

Lorek and Willinger (1996) demonstrated the superiority of multivariate cross-sectional models over firm-specific and common-structure ARIMA models. Lev and Thiagarajan (1993) identified 12 fundamental signals from financial ratios, subsequently utilized by Abarbanell and Bushee (1997) for EPS forecasting. Similar fundamental variables were employed by Cao and Gan (2009), Cao and Parry (2009), Etemadi et al. (2015), and Ball and Ghysels (2017) for multivariate EPS forecasting using neural networks, confirming their effectiveness. Ohlson (1995, 2001) formulated a residual earnings model, while Pope and Wang (2005, 2014) established theoretical frameworks linking earnings forecasts to accounting variables and stock prices. Li (2011) developed a model for forecasting earnings for loss-making firms, demonstrating its efficacy. Lev and Sougiannis (2010) provided evidence of the usefulness of estimate-based accounting items for predicting next year's earnings, albeit with limited success in subsequent years. Hou et al. (2012) achieved substantial R² coefficients in cross-sectional regression models for forecasting earnings. Li and Mohanram (2014) compared various models, revealing the superiority of some over others. Harris and Wang (2019) found Pope and Wang's (2005) model to be generally less biased and more accurate.

Decision tree-based techniques applied to EPS forecasting using accounting variables have yielded mixed results (Delen et al., 2013; Gerakos and Gramacy, 2013; Elamir, 2020; Chen et al., 2020). Recent advances in machine learning and deep learning have facilitated innovative experiments. Wang (2022) provided an overview of machine learning techniques for financial ratios forecasting, including EPS. Dreher et al. (2024) found that considering tax loss carryforwards did not improve EPS forecasts for German-listed companies and sometimes worsened predictions in out-of-sample tests, when using tax footnotes' information.

The domains of accounting, auditing, and finance frequently produce textual documents intended to communicate a wide array of messages, including but not limited to corporate financial performance, the management's assessment of current and future firm performance, domain standards, and regulations, as well as evidence of compliance with relevant standards and regulations. NLP applications have been used to mine these documents to obtain insights, make inferences, and create additional artifacts to advance knowledge in accounting, auditing, and finance. However, this is a relatively new and unexplored area of research, requiring expertise in both finance and computer engineering.

The paper by Fisher et al. (2016) synthesized the early literature on NLP in accounting, auditing, and finance to establish the state of knowledge at that time and to identify paths for future research. Since then, NLP methods have evolved substantially. Frankel et al. (2017) suggested that NLP techniques for narrative content from management conference calls and earnings prediction have reasonable construct validity, which is likely to be enhanced by further consideration of the unique characteristics of the text. More recently, Kambadura et al. (2023), in their book, presented core NLP techniques, mathematics in NLP, and applications.

A special area of research focuses on a specific class of NLP models. Xu (2019) used financial news from Reuters, Bloomberg, Yahoo Finance, etc., applying NLP to predict stock trends. Different word embedding techniques (i.e., vectorization of text) were compared. Huang et al. (2020) developed a financial text-specific variant of BERT (Bidirectional Encoder Representations from Transformers). In a Master's dissertation, Blomme and Dedeyne (2020) analyzed the impact of companies' financial reports on abnormal stock returns using FinBERT NLP methods. FinBERT, a customized version of BERT (a large language model by Google Inc.) for financial documents, was shown by Huang et al. (2023) to substantially outperform other NLP algorithms in sentiment classification for financial documents. FinBERT also excelled in identifying discussions related to environmental, social, and governance issues, outperforming other models by at least 18% in the textual informativeness of earnings conference calls.

Most existing research has focused on predicting stock prices using different NLP tools. Nabiee (2020) investigated the usefulness of changes in firms' annual and quarterly report language as signals for predicting future stock returns, using various similarity measures. Ishikawa et al. (2020) proposed and analyzed a methodology for forecasting the movements of analysts' net income estimates and stock prices using NLP and neural networks in the context of analyst reports. Differences were found among security firms depending on whether the analysts' net income estimates were forecasted by opinions or facts within reports, with trends in net income estimates proving effective for forecasting.

Wujec (2021) highlighted the increasing importance of text documents as a source of company information, using deep neural networks in NLP to analyze the sentiment of financial texts. The sentiment calculated using this method was directly related to market reactions. Rao et al. (2022) integrated public sentiment data into traditional stock analyses to predict stock price trends, using headlines from news publications and conversations from Yahoo! Finance's forum. The model achieved substantial accuracy. Medya et al. (2022) studied the statistical relationship among earnings calls, company sales, stock performance, and analysts' recommendations, with the semantic features of transcripts proving more predictive of stock price movements than traditional hard accounting data in most cases.

Santana Garcia (2023) analyzed the impact of financial news from two newspapers (the Financial Times and the Wall Street Journal) on the price dynamics of traded oil and gas companies, employing different NLP techniques, including fine-tuned large language models like BERT. Kiran et al. (2023) used sentiment analysis from various social media platforms to predict stock prices, employing different algorithms like K-means, long short-term memory, and convolutional neural networks for prediction, and error metrics like mean absolute error (MAE), mean standard error, and root mean standard error (RMSE).

Bv et al. (2023) analyzed the correlation between earnings call transcripts and subsequent stock price changes, extracting sentiments from quarterly transcripts using NLP techniques and identifying a possible causality between transcript sentiment and stock price movements. Rawte et al. (2021) emphasized the importance of studying similarities between documents over consecutive years in NLP for a comparative analysis of financial documents. Wang et al. (2024) examined the effect of competitive strategies on firm-level stock liquidity, using a machine-learning-based NLP approach, finding that firms with differentiation strategies showed higher stock liquidity due to increased investor attention and trading activities.

Recent studies have demonstrated the increasing application of NLP techniques in various aspects of finance, with a particular focus on the Polish market and Central and Eastern European (CEE) economies. These studies highlight the potential of NLP in enhancing financial forecasting, sentiment analysis, and content-based recommendations. Most of the studies have investigated the impact of sentiment on stock returns and trading volume in the Polish context. Polak (2021) examined the impact of sentiment derived from news headlines on the direction of stock price changes in the Polish stock market, finding no significant explanatory power in a one-day time frame. Wawer and Sobiczewska (2019) compared various NLP methods for predicting the sentiment of Polish-language short texts, finding strong performance from BERT and the multinomial Naive Bayes classifier. Wierzba et al. (2021) presented the Emotion Meanings dataset, a novel dataset of 6000 Polish word meanings derived from the Polish wordnet, annotated for valence, arousal, and basic emotion categories, providing a valuable resource for sentiment analysis in the Polish language. NLP techniques have also been applied to study the interdependencies between stock market indices and uncertainty measures in CEE economies. Kropiński (2023) investigated the relationship between main stock market indices and uncertainty measures quantified by X (formerly Twitter) messages for five CEE economies, including Poland.

The predictive power of narratives and economic research in the Polish market has also been explored using NLP techniques. Rybinski (2023) demonstrated that the content of news articles has a significant impact on their lifespan in five countries, including Poland. Rybinski (2021) presented a ranking of professional forecasters based on the predictive power of their research report narratives, showing that including NLP indexes in forecasting models lowers forecast errors in the Polish context. Rybinski (2020) compared the forecasting power of articles in a major Polish daily newspaper and regular research reports released by professional forecasters, finding that sell-side research may offer forecasting value, depending on the institution.

Dragan and Wróblewska (2019) adapted state-of-the-art NLP methods for determining similarities between text documents and content-based recommendations in the context of a Polish e-commerce platform, comparing various recommendation methods using semantic text analysis techniques to query the search engine index of documents. Klimczak (2020) presented the current state of the art in text analysis and its potential applications in finance, focusing on key innovations in sentiment analysis, topic identification, disambiguation, and multilingualism. Finally, the application of machine learning and NLP methods in literature reviews has been demonstrated by Łaniewski and Ślepaczuk (2024), who utilized these techniques to analyze and organize knowledge in the field of algorithmic trading, highlighting the effectiveness of large language models in refining datasets and addressing complex questions.

This study makes a notable contribution to the existing literature on EPS forecasting by investigating the applicability of NLP techniques, specifically FastText and FinBERT word embeddings, in the context of the Polish stock market. While previous research has explored various methods for EPS forecasting, including time series models, multivariate cross-sectional models, and machine learning techniques, the use of NLP-based approaches for this purpose has been relatively unexplored, particularly in emerging markets like Poland. The findings challenge the notion that sophisticated NLP techniques necessarily lead to improved EPS forecasting accuracy, as the SRW model consistently outperformed the NLP-based models across different periods and error metrics.

3.   Data and methods

3.1.   Data

Following its integration into the European Union after 2004, the Polish stock market demonstrated significant breadth, with market capitalization reaching ＄197 billion and encompassing 774 listed companies by the close of 2021. However, analyst coverage for these stocks remains limited relative to the United States or Western Europe, with approximately 20% of the 711 listed companies receiving attention in 2019. This limited coverage highlights the necessity for statistical or machine learning methods to forecast essential financial metrics. This research concentrates on the EPS series and other financial explanatory variables sourced from EquityRT, a financial analysis platform.

The EPS patterns of companies listed on the WSE are analyzed from the first quarter of 2010 to the fourth quarter of 2019, i.e. between notable structural changes such as the 2008–2009 financial crisis and the onset of the COVID-19 pandemic in 2020. For forecasting purposes, data from the first quarter of 2010 to the fourth quarter of 2018 (36 quarters) are used for model estimation, with data from the first quarter of 2019 to the fourth quarter of 2019 set aside for out-of-sample validation. Forecast horizons range from one to four quarters ahead, supplemented by additional validation samples from the years 2017 and 2018. The dataset, after ensuring comprehensive coverage and excluding any splits or reverse splits, includes 267 companies.

3.2.   The Models

The seasonal random walk (SRW) model

${Q}_{t}$ denotes the realization of EPS in quarter t. The SRW model is described as:

The forecast given by ${\widehat{EPS}}_{t} = {EPS}_{t-4}$ employs the value from four quarters prior as the prediction, negating the necessity for parameter estimation. This model acts as a baseline, with its effectiveness in outperforming many time series models within the Polish context highlighted in the studies by Kuryłek (2023a, 2023b, 2024).

The multivariate models with word embeddings

The Elektroniczny System Przekazywania Informacji (ESPI) system functions as an electronic mechanism for public companies in Poland to disclose significant information. Utilized by entities listed on the WSE's main market, ESPI reports are obligatory to ensure prompt and transparent communication of crucial data such as financial results, major shareholder changes, corporate activities, and other pertinent events influencing stock prices or investor decisions. The oversight and dissemination of ESPI reports fall under the jurisdiction of the Polish Financial Supervision Authority (Komisja Nadzoru Finansowego), which enforces compliance among WSE-listed companies to uphold market transparency and safeguard investor interests.

Meanwhile, the Elektroniczna Baza Informacji (EBI) system serves as another digital reporting platform for Polish public companies. Predominantly used by companies on the NewConnect market (an alternative trading venue managed by the WSE for smaller and developing enterprises, particularly those in nascent stages or the tech sector), EBI reports are intended to ensure timely and transparent dissemination of essential information to investors and the public. While the WSE operates the EBI system, adherence to reporting mandates is ensured and supervised by the Komisja Nadzoru Finansowego, thus preserving market's integrity and protecting investors' interests.

Both ESPI and EBI systems leverage HTML5 (hypertext markup language version 5) to guarantee accessible, readable, and interactive reports across various devices and web browsers. Reports are provided by Notoria Serwis S.A. and categorized into 32 inherent types. For individual companies, text pre-processing involves the removal of HTML tags, URLs, email addresses, and nonalphanumeric characters, and conversion to lowercase, followed by extracting and merging the reports' topics and content. The sequentially ordered extracts for each company and quarter are combined into a string of words and numbers. This string is then transformed into vectors using word embeddings.

Word embeddings in NLP represent words as vectors of real numbers within a continuous vector space, allowing words with similar meanings to have related representations, effectively capturing semantic relationships and contexts. This technique employs dimensionality reduction to map words into a lower-dimensional space, facilitating efficient computations and contextually representing words based on their document usage, thus capturing semantic similarities. Foundational to various NLP tasks, word embeddings enhance models' text comprehension by embedding semantic meanings, significantly improving NLP applications' performance.

This study applies two types of word embeddings: FastText (developed by Meta Inc.) for the Polish language and FinBERT. FastText, an efficient open-source library for text classification and word representation, was introduced by Facebook AI Research (FAIR) with key contributions from Joulin et al. (2016). It achieves results comparable with those of deep learning models at reduced computational costs, making it suitable for large-scale text classification. A specific version trained by web crawling is applied to the Polish language. In contrast, FinBERT (Huang et al., 2020) is a sophisticated NLP model designed specifically for financial text embeddings. Although FinBERT is not language-specific, it applies to various languages, including Polish. As a domain-specific variant of BERT (Bidirectional Encoder Representations from Transformers), developed by Google Inc. (Chang et al., 2018), it utilizes the Transformer architecture, a deep learning paradigm that has revolutionized NLP. Transformers, with their attention mechanism, focus on relevant sentence parts during processing, effectively capturing long-range word dependencies. They consist of an encoder and decoder: the encoder creates contextual word representations, and the decoder generates the desired output by focusing on the pertinent encoded information.

To represent an entire document as a vector, word vectors must be combined into a single vector for the document. Document embeddings, which condense whole documents or text sequences into dense vectors, can be constructed using pooling operations such as a simple mean, as employed in this study. Consequently, the multivariate model utilizing word embeddings can be articulated with FastText and FinBERT embeddings in the following way:

where ${N}_{t, i}^{j}$ is the number of reports in the j-th category for the i-th company in the t-th quarter within n categories, and n equals 32. The vector $\left[{X}_{t, i}^{1}, {X}_{t, i}^{2}, .., {X}_{t, i}^{m}\right]$ represents document embeddings of reports for i-th company in the t-th quarter. For the FastText word embeddings, the vector length (denoted as the parameter m) is 300 dimensions, whereas for FinBERT embeddings, it is 512 dimensions. Thus, incorporating the delayed EPS variable and 32 categorical variables results in independent variable vectors of 333 and 545 dimensions for FastText and FinBERT, respectively. The model was trained on a dataset comprising 7476 observations (calculated from 28 quarters across 267 companies) to forecast EPS for 2019. Implementation of the FastText model was carried out using the "fasttext" library in Python, while the "transformers" library facilitated the FinBERT model.

3.3.   Estimation techniques – a gradient-boosting decision tree (XGBoost)

Initially introduced by Chen and Guestrin in 2016, XGBoost (eXtreme Gradient Boosting) represents a substantial enhancement over the traditional gradient boosting algorithm, known for its notable speed and effectiveness. This advanced machine learning approach, which traces its roots to the decision tree methodologies developed in the 1960s, is extensively utilized for both regression and classification tasks. It develops a predictive model by combining numerous weak learners, usually simple decision trees, in an ensemble manner. Each successive tree aims to correct the mistakes of its predecessors, with the gradient descent algorithm iteratively updating the weights of these weak learners. This process continues until the loss function is minimized or a specific stopping criterion is achieved. XGBoost incorporates several techniques to improve the performance of gradient boosting models, including regularization to reduce overfitting by adding penalties to the loss function, tree pruning to eliminate unnecessary branches and enhance the model's stability, and parallelization to accelerate the training process. Additionally, XGBoost effectively captures nonlinearities in the data. For a comprehensive overview, Simon's 2020 book provides valuable insights. The implementation of this methodology utilizes the "xgb" library in Python, with the hyperparameters optimized (by cross-validation in the training sample) using the "hyperopt" library to enhance forecast accuracy (Banerjee, 2020).

It has been documented by some researchers (Armon et al., 2022; Borisov et al., 2024) that gradient-boosting models such as XGBoost can frequently be effectively used in tabular datasets, particularly with structured and relational data, due to their ability to naturally incorporate features' interactions.

3.4.   Mean arctangent absolute percentage error

The actual EPS for a particular company i across the first to the fourth quarters of 2019 can be denoted by ${A}_{1}^{i}, \dots, {A}_{4}^{i}$. The corresponding forecasts for these quarters, represented as ${F}_{1}^{i}, \dots, {F}_{4}^{i}$ (i.e. ${\widehat{Q}}_{t}$, where t = 37, .., 40 for the i-th company), allow the absolute percentage error (APE) for the forecast in the j-th quarter of 2019 for any i-th firm to be expressed as follows:

The APE, despite its widespread use, has a critical limitation: it tends to produce infinite or undefined values when the actual figures are near zero, a frequent occurrence in earnings predictions. Furthermore, extremely low actual values, typically less than one, can result in disproportionately high percentage errors, causing outliers. This issue is exacerbated when actual values are zero, leading to infinite or undefined APEs. To address this problem, the arctangent APE was introduced by Kim and Kim (2016), offering a more robust alternative for handling such cases in predictive analytics.

The basis for adopting this method stems from the characteristic of the arctangent function, which maps values ranging from negative infinity to positive infinity into the interval [-π⁄2, π⁄2]. As a result, the MAAPE for the j-th quarter across all I companies in the dataset can be expressed as follows:

The preference for MAAPE instead of mean APE was intentional because of the occurrence of companies with actual profits approaching zero in the dataset under study. In cases where only one observation approaches zero while the others are significantly distant, the mean APE for that particular observation can escalate to an extremely high value, approaching infinity. This issue has the potential to dominate the mean calculation, making the remaining observations negligible in their impact.

3.5.   The statistical test

To assess the statistical significance of differences in the MAAPE across multiple models, a nonparametric two-sided Wilcoxon test, originally described by Wilcoxon (1945), was employed. This test serves as a paired difference test for two matched samples. It is important to note that this test does not require specific assumptions about the probability distribution, except for the symmetry of score differences and the independence of these differences. Ruland (1980) extensively discussed the application of the Wilcoxon test in validation, particularly in determining whether errors from various EPS models exhibit statistically significant distinctions. Separate tables containing p-values were generated for each quarter, ranging from one to four, as well as for all quarters combined.

If the p-values from each test fall below the predefined significance level of 0.05, the null hypothesis for each test will be rejected. This widely accepted principle is supported by various sources, including Ruland (1980).

4.   Results

The SRW model, detailed in Table 1, consistently demonstrated superior performance over all multivariate word embedding models across each quarter and throughout 2019, exhibiting the lowest MAAPE overall. In contrast, the model utilizing FinBERT word embeddings showed the poorest performance among all methods, with the FastText approach falling between these two extremes.

To determine whether the errors of the top-performing model significantly differ from those of the other methods, the Wilcoxon nonparametric test was utilized to compare the arctangent APE medians between the SRW model and all alternative approaches. As shown in Table 2, the results indicate that the SRW model consistently exhibits statistically lower errors compared with the other methods across all analyzed periods of 2019.

4.1.   Robustness checks

Robustness assessments were conducted across multiple years using two widely recognized error metrics. It is important to note that across the years examined (2017, 2018, and 2019) the SRW model consistently delivered superior forecasts compared with the other methods, as depicted in Table 3. However, the performance of the FinBERT approach varied; while it underperformed in 2019 and 2017, it marginally outperformed FastText in 2018.

Furthermore, the Wilcoxon test was employed to compare each model pair against the SRW model, with the corresponding p-values for each year detailed in Table 4. Throughout these years, the SRW model consistently exhibited statistically superior results compared with alternative methods. Hence, the sustained dominance of the SRW model over time is evident.

Table 5 provides a performance assessment of the analyzed models using alternative error measures: RMSE and MAE. This evaluation aggregates all quarters for the year 2019. To ensure an equitable comparison, these metrics were adjusted for consumer price index (CPI) inflation, ensuring that future errors' present values in nominal terms are equivalent to the current errors.

Consistent with previous findings in 2019, the SRW model demonstrated the lowest errors across all metrics, encompassing both RMSE and MAE. In contrast, the model utilizing FinBERT word embeddings exhibited the poorest performance in 2019 according to both error criteria, aligning with previous observations regarding MAAPE. Table 6 presents the p-values derived from the Wilcoxon test, highlighting significant differences between the outcomes of the SRW model and other model comparisons.

The research findings suggest that despite incorporating variables from various word embedding algorithms and employing advanced forecasting techniques based on them, there was no observed improvement in performance. Moreover, these sophisticated approaches did not surpass the basic SRW method in terms of predictive accuracy.

5.   Discussion

The comparatively lower performance of word embedding models, utilizing gradient-boosting decision trees, stems from issues related to overfitting. Overfitting causes unstable relationships among variables, which may hold only for specific test datasets, and can occur when the model becomes too complex and starts capturing noise in the training data rather than generalizable patterns. One of the best techniques to reduce the overfitting problem in the XGBoost model is the optimization of hyperparameters with cross-validation in the training sample, and this approach was applied. However, regardless of the use of this method, overfitting can persist. Using such relationships for predictions is reasonable only if their statistical robustness is sufficiently validated, as emphasized by Lev and Souginannis (2010). This observation is consistent with Dreher et al.'s (2024) findings for German listed firms, showing that while complex deep learning techniques enhance explanatory power within the sample, they fail in out-of-sample predictions. Such sophisticated models risk overparameterizing straightforward market behaviors, resulting in larger forecast errors. Additionally, introducing more variables than simply the EPS value delayed by four quarters adds unnecessary noise to the data, exacerbating the challenge of forecasting.

The rationale behind the superior performance of simpler models may resonate within the Polish context. Advanced models often become overly intricate, employing an excess of parameters to describe relatively straightforward economic phenomena. This observation echoes the findings of Kuryłek (2023a, 2023b, 2024), who demonstrated that basic models such as ARIMA and exponential smoothing, and also advanced time series methods, which are effective in the US market, were surpassed by the simple SRW model in Poland. This underscores the hypothesis that the inherent simplicity of the Polish stock market likely underpins the effectiveness of the SRW model, suggesting that additional calibration for out-of-sample predictions may be necessary.

Thus, the straightforward application of sophisticated techniques that are reliant on word embeddings beyond the conventional SRW for EPS forecasting in Polish investment contexts appears to be impractical. Moreover, considering that EPS behavior follows a SRW and acknowledging that stock prices derive from EPS multiplied by the P/E multiple, one might infer that stock prices exhibit at least as much randomness as EPS. Given this EPS behavior characterized by a random walk, accurately predicting stock prices even just one quarter ahead becomes particularly daunting.

In shorter timeframes where the EPS remains constant, stock price forecasting behaves similarly to P/E multiples. Therefore, exploring methods to forecast P/E multiples for periods shorter than one quarter, occurring between the publication of quarterly financial reports, could prove highly relevant from an investment standpoint. The forecast generated by the SRW essentially reflects a value from the corresponding quarter of the previous year. This implies that for predicting future prices, even over extended horizons, the P/E multiple might carry more weight than next year's earnings of companies (EPS). This conclusion aligns with economic theory, positing that the P/E multiple is influenced by anticipated growth in future earnings, future interest rates, and market sentiment or premium reflecting investors' risk appetite, while EPS forecasts relate solely to imminent earnings. In both short-term and long-term investment contexts, the consensus is clear: the P/E multiple surpasses EPS prediction in significance.

6.   Conclusions

The research offers a significant contribution to the literature on EPS forecasting by examining the potential of NLP methods within the Polish stock market context. While past studies have extensively explored EPS forecasting through various approaches such as time series analysis, multivariate cross-sectional models, and machine learning algorithms, the integration of NLP-based techniques remains underexplored, especially in emerging markets like Poland.

The study examined the predictive capabilities of three methodologies, namely the SRW and a gradient boosting tree (XGBoost) utilizing two different word embedding models (FastText and FinBERT), applied to publicly available company reports. These multivariate approaches were trained using 333 and 545 explanatory variables, respectively, encompassing firm-specific textual attributes. EPS forecasting is crucial in emerging markets like Poland, where financial analyst coverage of listed companies is sparse. Analyzing quarterly EPS data from 267 Polish firms spanning 2010 to 2019, the SRW model consistently demonstrated the lowest error rates, providing a more accurate representation of the Polish market compared with alternative models. Additionally, the SRW model consistently outperformed other methods across different periods and error metrics such as RMSE or MAE. This pattern is supported by Wilcoxon tests and can be attributed to the overparameterization of complex models, their susceptibility to overfitting, and the relatively straightforward nature of the Polish stock market.

The research findings indicate that despite incorporating variables derived from word embeddings and employing advanced forecasting algorithms such as XGBoost on tabular data, there was no improvement in performance, challenging the assumption that advanced NLP methods inherently enhance EPS forecasting accuracy. Consequently, the practical implication of this research suggests that employing techniques beyond the standard SRW for EPS forecasting in Poland lacks justification. However, relying on the SRW for EPS modeling implies that forecasted stock prices may exhibit significant randomness, posing challenges for accurate prediction. Therefore, forecasting the P/E multiple might hold greater significance than predicting EPS for future stock price projections, especially in shorter investment horizons where the EPS remains stable.

Future research could explore the relationship between forecasting accuracy and firm size, with industry sector analysis potentially influencing the choice of the most suitable EPS forecasting model. Investigating time series transformations to normalize EPS distributions could offer valuable insights. Furthermore, evaluating the performance and accuracy of various predictive models and financial analysts' forecasts during economic downturns, such as the 2008–2009 financial crisis or the COVID-19 pandemic, could provide valuable perspectives. One could also explore the impact of different text sources on the NLP model's performance. While the current study focuses on public company reports, incorporating additional sources such as news articles, social media posts, or earnings call transcripts may provide a richer set of textual features and potentially improve the model's predictive power. Future researchers can investigate to find out which cases will yield better results than the SRW model when using the NLP-based model. Identifying seasonal patterns through the SRW model may also contribute insights into investment strategies, potentially challenging the "weak form" of the efficient market hypothesis.

Use of AI tools declaration

The author affirms that no artificial intelligence (AI) tools have been used in the creation of this work.

Funding

The authors have received no funding from any source in the preparation of this work.

Conflict of interest

The author declares no conflicts of interest in this paper.