Dagienė, Eleonora

Performance-based funding systems (PBFSs) are widely used to steer national research, but their effects vary significantly, particularly in countries with emerging research ecosystems. Relatively little attention has been paid to PBFSs and their concomitant policy dynamics in these countries, where the pressure to internationalise creates unique challenges. This paper presents a detailed study of the development of the Lithuanian PBFS from 2005 to 2022. Using a multi-level, multi-actor, and multi-issue framework, we combine policy analysis, semi-structured interviews, and bibliometric data to analyse the system’s evolution. Our findings reveal a dynamic of ‘insular policy feedback,’ where a concentrated scientific elite, operating across all levels of governance, shapes policy to its advantage. This results in predictable cycles of strategic gaming, such as the proliferation of domestic journals, by reactive and often inconsistent state countermeasures. The Lithuanian case serves as a model for understanding how concentrated power structures can undermine reform, offering a crucial insight for policymakers: meaningful reform must address the governance structures that empower performance metrics, not just the metrics themselves.

Ar visi mokslai, tiek fundamentalūs, tiek socialiniai gali būti vertinami pagal tą patį kurpalių? Kaip formavosi mokslo vertinimo ištakos laisvoje Lietuvoje? Kiekybiniai ir kokybiniai moksliniai tyrimai – jų specifika ir finansavimas. Šiuos ir kitus klausimus aptarsime laidoje su MRU mokslininke Eleonora Dagiene, kuri pastaruoju metu analizuoja Lietuvos mokslo vertinimo sistemą ir netrukus gins disertaciją Leideno universitete, Nyderlanduose. Jos socialinių mokslų tyrimai atskleidžia Lietuvos mokslo vertinimo ypatumus, mokslo bendruomenės ir mokslo politikų perspektyvas bei tarptautinį Lietuvos mokslininkų publikacijų matomumą. Ši laida bus įdomi kiekvienam, kam rūpi Lietuvos mokslo ateitis.

This mixed-methods study examines the complex interplay of actors and factors shaping research assessment policies in Lithuania in the years 1996–2008, a period of transition towards Western practices. Analysing policy documents, bibliometric data, interviews, and grey literature, we focus on the interactions among policymakers, international experts, providers of publication data, and researchers. Our findings reveal the significant influence of international networks on Lithuanian policymakers, leading to the adoption of quantitative assessment and reliance on Web of Science data. However, this narrow focus, coupled with existing power dynamics and top-down decision-making, led to gamesmanship in publishing practices and conflicts among disciplines, culminating in a Constitutional Court challenge. This research also highlights the substantial role of journal publishers and data providers in influencing policy shifts. We underscore the need for international organisations to adopt open data sources that encompass a wider array of scholarly outputs. Additionally, national policymakers must consider disciplinary differences and research capacity when designing assessment policies. The Lithuanian experience serves as a cautionary tale, emphasising the importance of context-specific, inclusive, and flexible research assessment systems, particularly in light of ongoing reforms such as the Coalition for Advancing Research Assessment (CoARA).

This paper proposes an open-science-aligned approach that uses library metadata to evaluate individual books. I analyse the suitability of this approach for individual book assessment and visibility of national books in the library catalogues, to support responsible research evaluation. WorldCat metadata offers valuable insights for the evaluation of books, but the completeness of this metadata varies. Author, contributor, and publisher data require cleaning, while languages, years, formats, editions, and translations provide rich information. Open access data is currently lacking, and national book visibility in WorldCat depends heavily on contributions from national libraries and metadata suppliers. Encouraging national library engagement could boost the global visibility of domestic research. Further exploration is needed regarding long-term preservation, metadata ownership, and technical integration for effective standardisation and improved book evaluation.

Books are an important output in many fields of research. However, they pose a significant challenge for research assessment systems, partly because of the limited availability of information to support the assessment of books. To inform book assessment practices, I present a systematic examination of the ISBN Manual and the Global Register of Publishers (GRP). I evaluate the extent to which these two sources can be used to determine the genre and publisher of a book as well as the country in which a book was issued. My analysis focuses on books submitted to the research assessment systems in Lithuania and the UK from 2008 to 2020. I show how the ISBN Manual captures the complex interactions between publishers, their imprints, and other organisations active in academic publishing, revealing the pitfalls of measuring books’ quality by their publisher’s prestige. The results also indicate that the ISBN standard provides no basis for the book genres mandated by research assessment systems in some countries. Finally, I demonstrate how the ISBN Manual and metadata accumulated in the GRP are convenient tools for designers of research assessment systems and are suitable for identifying ISBN registrants and performing bibliometric analysis.

Numerous national research assessment policies aim to promote ‘excellence’ and incentivize scholars to publish their research in the most prestigious journals or with the most prestigious book publishers. We investigate the practicalities of assessing book outputs in different countries, concentrating on evaluation based on the prestige of book publishers (e.g. Denmark, Finland, Lithuania, Norway, Poland, and Spain). Additionally, we test whether such judgments are transparent and yield consistent results. We show inconsistencies in the levelling of publishers, such as cases where the same publisher is ranked as prestigious and not-so-prestigious in different states or within the same country in consequent years. Likewise, we find that verification of compliance with the mandatory prerequisites is not always possible because of the lack of transparency. Our findings support doubts about whether the assessment of books based on a judgement about their publishers yields acceptable outcomes. Even more, currently used rankings of publishers focus on evaluating the gatekeeping role of publishers but do not assess other essential stages in scholarly book publishing (e.g. dissemination or preservation). Our suggestion for future research is to develop approaches to evaluate books by accounting for the value added to every book at every publishing stage, which is vital for the quality of book outputs from research assessment and scholarly communication perspectives.

Books and evaluations thereof are essential components of the research system. Despite existing efforts to use citation-based metrics to evaluate books, we argue that a more effective book evaluation system should focus on the entire lifecycle of book publication, not just the final products. Building on this idea, we here present the preliminary results of an ongoing project aimed at establishing such a system. As the first step of this project, we analysed ISBNs used for book outputs submitted to two national research evaluation systems, the UK Research Excellence Framework 2014 and the formal Lithuanian research assessment for 2004–2016. By conducting this analysis, we investigate whether and how ISBNs might be used as a reliable identifier in collecting bibliographic data, which will then serve as the foundation of the new book evaluation system. We describe our approach to ISBN collection, processing and analysis, and discuss the suitability of ISBNs for our purposes. Notably, we found ISBNs a handy indicator, given their relevant ubiquity in the book-centric bibliographic universe. However, processing ISBNs from existing systems is very time-consuming and requires extensive manual effort, which calls for a better book bibliographic data infrastructure.

Numerous national research assessment policies set the goal of promoting "excellence" and incentivise scholars to publish their research in the most prestigious journals or with the most prestigious book publishers. We investigate the practicalities of the assessment of book outputs based on the prestige of book publishers (Denmark, Finland, Flanders, Lithuania, Norway). Additionally, we test whether such assessments are transparent and yield consistent results. We show inconsistencies in the assessment of publishers, such as the same publisher being ranked as prestigious and not so prestigious in different countries or in different years in the same country. Likewise, we find that verification of compliance with the mandatory prerequisites is not always possible because of the lack of transparency. Our findings raise doubts about whether the assessment of books based on a judgement about their publisher yields acceptable outcomes. Currently used rankings of publishers focus on evaluating the gatekeeping role of publishers but do not assess their dissemination role. Our suggestion for future research is to develop approaches for assessing books which consider both quality control and the distribution of books (and their metadata) as measured by the importance of communication between researchers. That means that publishers should be transparent about the services they deliver in both areas, preferably at the level of individual books, so that there is no need to rely on general information about publishers.