Revista IECOS, 25(1), 126-140 | Enero-Junio 2024 | ISSN  2961-2845 |  e-ISSN  2788-7480

 

MANAGEMENT STRATEGY AND RESEARCH PRODUCTIVITY IN THE PERUVIAN INSTITUTE OF NUCLEAR ENERGY FOR THE PERIOD

2000 – 2020

 

 

ESTRATEGIA DE GESTIÓN Y PRODUCTIVIDAD DE LA INVESTIGACIÓN EN EL INSTITUTO PERUANO DE ENERGÍA NUCLEAR PARA EL PERIODO

2000 – 2020

 

Edgardo Martin Figueroa Donayre1, Jose Luis Mantari Laureano2, Modesto Edilberto Montoya Zavaleta3

 

1National University of Juliaca, Puno, Peru

2Department of Sciences, University of Engineering and Technology, Lima, Peru

3Peruvian Institute of Nuclear Energy, Lima, Peru

E-mail: 1em.figueroa@unaj.edu.pe, 2jmantari@utec.edu.pe, 3mmontoya@minam.gob.pe

1https://orcid.org/0000-0001-7891-3334, 2https://orcid.org/0000-0002-3621-3425, 3https://orcid.org/0000-0002-3376-1872

 

https://doi.org/10.21754/iecos.v25i1.1955

 

Recibido (Received): 20/10/2023 Aceptado (Accepted): 27/02/2024 Publicado (Published): 31/03/2024

 

ABSTRACT

 

This paper examines the production indicators of the Peruvian Institute of Nuclear Energy from 2000 to 2020. As an institution focusing on ionizing radiation research and application, it published an average of one article yearly from 1984 to 2000. Notably, between 2001 and 2006, specific policies were implemented: staff salaries were tripled, enabling the recruitment of experienced researchers. Consequently, the number of publications surged from 1 in 2001 to 15 in 2007, reflecting the intensive efforts of 2006.

However, these policies were disregarded in 2007, resulting in a significant drop in research productivity in subsequent years. Overall analysis indicates two key points: (a) management actions vary considerably across different directors and have a substantial impact on research productivity, and (b) the policy of increasing researcher salaries may be of utmost importance compared to augmenting the total annual research budget. This shift can potentially sustain positive trends in research indicators despite budget constraints.

 

 

Keywords: research productivity, management, nuclear plants, policies.

 

RESUMEN

 

Este artículo analiza los indicadores de producción del Instituto Peruano de Energía Nuclear durante el período 2000-2020, una institución dedicada a la investigación y aplicación del conocimiento sobre radiación ionizante. Entre 1984 y 2000, el instituto publicó en promedio un artículo por año. Durante el período 2001-2006, se implementaron ciertas políticas, como triplicar los salarios del personal, lo que permitió la incorporación de investigadores con experiencia en investigación científica. El número de publicaciones aumentó de 1 en 2001 a 15 en 2007 como resultado del intenso trabajo realizado en 2006.

Sin embargo, estas políticas fueron ignoradas en 2007, lo que provocó un cambio significativo en la productividad de la investigación en los años siguientes. Un análisis global determina lo siguiente: (a) las acciones de gestión difieren de un director a otro y estos cambios afectan sustancialmente la productividad de la investigación, y (b) la política de aumentar el salario de los investigadores puede ser absolutamente imperativa en comparación con el aumento del presupuesto anual total para la investigación, ya que aumentar los salarios de los investigadores todavía puede preservar las tendencias de los indicadores de investigación a pesar de las limitaciones presupuestarias.

 

Palabras clave: productividad de la investigación, gestión, plantas nucleares, políticas.

 

1. INTRODUCTION

 

The Peruvian Institute of Nuclear Energy (IPEN in Spanish words), created on February 4, 1975 (DL21094), is a Public Research Institute (PRI) attached to the Ministry of Energy and Mines (MINEM, in Spanish) with the mission to regulate, promote, supervise and develop nuclear energy applications (DS062-2005-EM). Within this framework, IPEN conducts research and applications related to ionizing radiation, while at the same time, through the National Authority Technical Office (OTAN), supervises and oversees the use of ionizing radiation to ensure compliance with the provisions established in Law 28028 and its regulations (DS039-2008). IPEN has an annual budget from the public treasury for current and investment expenses. To carry out nuclear energy application projects, it receives technical cooperation from the International Atomic Energy Agency (IAEA), which allows for the training of scientific and technological personnel, the receipt of equipment and materials, and visits from experts.

 

Montoya in 2006, co-author of the present manuscript, coin the saying “Without Science, there is no Future (in Spanish: “sin Ciancia no hay Futuro”), this term was an earthquake for Peruvian politicians and public administrators that had been conducting Peru without science. Nowadays the situation in this country is not different, science, innovation, and technology are taken as worthless or just ignored. Politicians and public administration are governing this country in many cases with the rule of thumb or based on limited or minimum information.

In this context, Van-Raan (1999) just at the beginning of this new century remark on the importance of science and technology communication by sliding and coining the following statement “Science would not exist, if scientific results are not communicated. Communication is the driving force of science. That is why scientists have to publish their research results …”.

Indicators of the research productivity of a research institute may be found in-country research councils or in some research databases available to the scientific community. However, this information needs to be processed and analyzed. Fortunately, bibliometrics and scientometrics can help understand research productivity trends, authors’ and institutions’ capabilities, citations, research cooperation, etc. With the use of scientometrics, researchers can systematically ensure science communication by motivating and promoting research communication and patents. In addition, recently, the topic is being used to develop an excellent state of arts and find gaps in research activities in a systematic and scientific way.

 

On the other hand, boosting research in a science center requires motivated teams of researchers. Curiosity about universe nature and hops of earth preservation is the main character and motivation for successful researchers. In addition, it is important to remark that, to be productive, researchers must have a calm mind, which is achieved with decent remunerations that allow them a carefree life. Consequently, research management along with serious country research politics may drive and guarantee science and innovation communications through key indicators: (a) scientific publications, and (b) patents, and together produce innovations and businesses for a better quality of life on our planet.

 

With the strong but true sentence described above, it is clear that science generation and communication are crucial to guarantee a better life. It is also clear that the management of research productivity is an important task. Therefore, in this line, it is relevant to study the progress in scientific communication through bibliometric and scientometrics studies around the world. In this sense, there are valuable contributions in regards to the research productivity in public research institutes and with the particular case of the present paper: nuclear aspects.

For example, Bala and Kumari (2013) studied the research publication of a public institution for 10 years. Coccia (2004, 2005) proposed a few scientometric models to determine the research performance of public research institutes. Surendra-Kumar and S. Kumar (2004) studied the scientific publications in agriculture research institutes in India. The fund that multi-authorship is common in the studied articles. Kademani et al. (2005) analyzed the research production of chemical researchers in an atomic research center in India.

Dhawan and Gupta (2007) studied research productivity and the contributions of India in physics as indexed in the INSPEC–Physics database for the year 1998. Jeevan and Sen (2007) investigated the scientific productivity of two accelerator-based research facilities in India in the period 1997 to1999. The study demonstrated that while one of the research facilities follows quantity, the other one persuades quality research. Quality of the manuscript could be measured by the quartile of the journal and the number of article citations. Surwase et al. (2008) evaluated research articles on Neutron Scattering research published in the Scopus database in the period 1991 to 2006.  The authors found that USA, Germany, and France, respectively, lead the research activity in the area.

Upadhye et al. (2010) analyzed the publications of the Nuclear Physics Division at Bhabha Atomic Research Centre (BARC). The authors concluded that for BARC scientists, quality is important. The multi-authorship trend has also been seen in the publication pattern. Jeyshankar (2015) identified research productivity in an Atomic Research facility in India using publication data from Scopus in the period 1989 to 2013. Interestedly, Abramo and D’Angelo (2015) concluded that it is important to comprehend the differences between, the performance of individual scientists and the performance of the scientific fields in a research institution. We believe that in order to plan strategies to improve the research indicators, it is important to understand such facts.

Rijcke et al. (2016) and Waltman (2016) documented a review paper on research indicators and techniques to study research productivity. Mondal and Raychoudhury (2018) revised interesting papers on research productivity in nuclear physics and evaluated the research performance based on the Web of Science dataset of a Nuclear Physics Institute in India.  Pal and Sarkar (2020) conducted an interesting and in-depth review of the literature on institutional research productivity covering several aspects of research productivity in India and worldwide.

This paper analyzes the influence of management on research activities traduced in research productivity between 2000 and 2020. Incentive policies for scientific and technological production of the Peruvian Institute of Nuclear Energy (IPEN) and the national budget for research are also analyzed. The present paper is organized as follows, in Section 2 the methodology is described; in Section 3 Management and policies for research productivity are listed; in Section 4 the discussion on management and research productivity is presented; and finally, conclusions are given in Section 5, followed by acknowledgment and reference list.

 

2. METHODOLOGY

 

Published literature was collected using the IPEN database during the period of 2000-2020, which was corroborated with the Scopus database, and then data preparation was conducted to clean, transform, reformat and enrich the inputs to evaluate the research productivity in IPEN.

It is difficult to analyze research productivity in an interval or period of management. What happens at the beginning and at the end of a management period or in the transition of manager directors (Presidents) is difficult to clarify. Since the research productivity of one year is highly influenced by the research productivity and activities in the previous year(s). It happens because journals have different acceptance time rates. Therefore, it is important to evaluate accumulated research productivity, and this study should be conducted in a reasonable period in order to have a better conclusion on management actions and research productivity.

In this paper, the following equation (Eq. 01) is utilized to count for accumulated research productivity of a research institution in a management period [a,b]. However, conclusions on management actions are made in the interval [a+x,b+y].

 

                                           (01)

 

where: TNR is the total number of researchers (including invited researchers, it could be a decimal number depending on visit time); PNR is the permanent number of researchers (normally a natural and fixed number); a is the beginning year of an interval; b is the end year of an interval; YRP is a year research productivity, and ARP is the accumulated research productivity. The variables x and y should be carefully estimated, it depends on a deeper understanding of the researcher’s preferred journals acceptance rate analysis, which has not been done before and reported in the scientific literature.

 

3. MANAGEMENT AND POLICIES FOR RESEARCH PRODUCTIVITY

 

Between the year of its creation, 1975, and the year 2000, management positions at IPEN were occupied by professionals who did not necessarily have masters or doctoral degrees.

 

Additionally, the budget law did not allow for the promotion of professionals in the staffing table. Consequently, a professional who recently graduated from the university and enters IPEN at a lower salary level may remain at that level until retirement, irrespective of their scientific output. Consequently, scientific production was significantly reduced.

 

With this reality as a basis, in 2001, were established the following incentive instruments:

 

1)      Management positions were occupied by professionals with doctoral or master's academic degrees.

2)      Fellowships financed by the IAEA were awarded to researchers who had demonstrated scientific and technological production.

3)      There were management actions and facilities to support the publication of Scientific and Technological Report, including presentations made at congresses.

4)      Staff salaries were tripled on average, which made it possible to attract professionals with proven high production of scientific publications.

 

In December 2006, authorities at IPEN changed, returning to the previous management model. The first effect was the resignation of the scientists who joined the institute because of the incentive policies.

 

The following sections present the effects of these changes in management models at IPEN.

 

 

 

4. DISCUSSION ON MANAGEMENT AND SCIENTIFIC PRODUCTION

 

4.1 SCIENTIFIC PRODUCTION AT IPEM

 

 

In this paper, it is assumed that x=y=1. In the majority of cases, journals take less than one year to accept a paper for publication. Between 1984 and 2001, IPEN published an average of one article per year. The number of publications in Scopus base journals increased since 2002, reaching 15 in 2007, the biggest year-on-year growth since 1984. However, the number of Scopus publications dropped to 05 in 2008. Between 2008 and 2020, there was an average of 04 articles per year. By the research productivity trends, we may infer that the institution had a stagnation period (see Figure 1).

 

Figure 1

 

The annual number of IPEN publications in Scopus journals. Source: IPEN

 

 

 

Figure 1 shows the research productivity year by year. As it may be inferred, the publication of one year can be influenced by the publication in previous years. For example, it may be considered that the publication in 2007 is higher because the papers submitted in 2006 and not published during this year.

 

4.2 MANAGEMENT ACTIONS AND INDICATORS ON RESEARCH PRODUCTIVITY

 

On the other hand, since data in public entities in Peru is not always easy to persuade, collect, clean, and process information. It is important to mention that until the submission of this paper no detailed information on invited researcher and their invited period in IPEN was obtained, and also knowing that is difficult to hire or invite researchers in public institutions in general, it is assumed that PNR is equal to TNR for the present study.

 

Management actions traduced in policies at IPEN were applied in 2001. The average 01 publication per year during the period 1984 to 2000 (1985 to 2001 in terms of management and research productivity analysis) stated to increase permanently up to 15 in 2007 (due to management actions in 2006).

 

In order to notice the change of management action traduced in research productivity, two management periods were analyzed, see Figures 2 and 3.

 

 

Figure 2 shows the accumulated productivity during the period of 1995 to 2000 (1996-2001) compared with ones obtained during the period of 2001 to 2006 (2002-2007), i.e. after management on research was applied in 2001. As it can be seen, a tremendous change in slope at the inflection point is visible during Montoya management period. This kind of information was not published before for any kind of public institution in Peru, consequently, decision-makers and main governing authorities should take into account this kind of information for better management of public resources.

 

 

 

 

Figure 2

 

Accumulated research publication in two management periods of IPEN (1996-2001 and 2002-2007).

 

Change of Leaders always changes Management actions and strategies, yet these changes don't consistently lead to improved productivity. Figure 3 shows the accumulated productivity during the period of 2002 to 2007, and after different research management during the period of 2008 to 2021. A serious and non-favorable change of slope can be seen at the inflection point. This means the manager directors of the institution could not maintain research productivity projected by Montoya management period.

 

 

 

 

 

 

 

 

 

 

Figure 3

 

Accumulated research publication in two management periods of IPEN (2002-2007 and 2008-2021)

 

From 2008 to 2021 almost no change in management action on research productivity was observed. The trend seems to be the same, consequently, it can be said that the institution has almost move-in autonomous pilotage, since no change from the predecessor, transition, and holder manager director actions on research productivity is noticed. It is important to mention that IPEN also has production and services administrations with indicators that could be interesting to analyze in futures research works. So, a lot of information needs to be properly processed for further conclusion on management’s actions on KPIs of service and production offices in IPEN.

 

 

 

 

 

 

 

 

4.3 THE IPEN EXPERIENCE MAY BE APPLIED TO A NATIONAL LEVEL IN PERU

 

Research productivity of an institution or a country may be also measured through the number of patent applications of residents (NPAR). University of Michigan and China are clear examples of a particular intuition and country leaders on Intellectual Property Rights (IPR) in WIPO 2021.

 

As remarked by Montoya (2011), from 1994 to 2007, the number of patent applications per resident grew roughly linearly at a rate of 32,657 per year. The author emphasizes that although some countries (Argentina, Brazil, Chile, Colombia Spain, Mexico, and Peru) have increased their investment in research and development (R&D), just Argentina and Brazil presented an interesting correlation between the number of patent applications per residents and millions of dollars invested. Interestingly, those countries have both (a) a career law for researchers and (b) a Ministry of Science and Technology.  

 

Recently, Neves et al. (2021) illustrated that patents have an overall positive effect on innovation and growth in developed countries, but a weak positive effect in developing countries. Consequently, the situation for these countries is critical and management strategists need to be designed and implemented immediately in order to overcome tremendous gaps. Policymakers and academics need to conduct further research in this line.

 

Based on the experience of IPEN, the creation of the Occupational Group of Scientific and Technological Researchers was proposed in 2006 in Peru, with a career plan that considers hierarchies in line with the production of articles, patents, and technological services carried out during the professional performance of each researcher (Montoya, 2006).

 

Interestingly, Montoya (2011) outlined that in countries where there was no researcher career, greater investment in science and technology did not have an evident impact on the number of patent applications. So, the creation of the Scientific and Technological Researcher Career (STRC) was proposed, with the following characteristics (Montoya, 2011):

 

i) To enter the STRC, scientific-technological researchers who have at least five articles published in indexed international scientific journals or a patent can apply.

 

ii) Incorporation to the STRC is conducted through a selection process held at the end of each year in areas prioritized in the Strategic Plan for Science, Technology, and Innovation (STI).

 

iii) To move up one level in the STRC, it is required to have published five additional articles in indexed international scientific journals or to have obtained two patents.

 

iv) To remain in the STRC, an article must be published annually in an indexed international scientific journal, or a patent must be obtained.

 

v) The most senior positions in the research institutes will be assigned to scientific and technological researchers who are at the highest career level.

 

In Peru, University Law 30220 enter in force in 2014, in which special bonus of fifty percent of the total salaries of research professors were offered and announced. The results on the scientific production began two year later. Until 2014, Publication In Scopus (PIS) was lower than Investment in Science Technology and Innovation (ISTI) with the exception of 2012, but both values had roughly the same annual growth rate (See Fig. 4). Figure 4 shows that between 2015 and 2016, PIS started to be higher than ISTI in hundreds of thousands of dollars. The only way this change could be explained is the increase of research incomes. What happens in developed countries may not be applied in developing countries so this kind of study is absolutely imperative, similar conclusions are reached by Neves et al. (2021) in regards to intellectual properties.

 

 

 

 

 

 

 

Figure 4

 

The number of Scopus publications (blue), investment in STI (red) (dollars/100k), and the total number of professor with bonus for research called: Bonus for research professors (BRP) at public universities (black) in Peru (public government information).

 

 

 

In 2017, the 50% bonus became effective for the first group of researchers, which involved the transfer of 7.34M soles in the public budget. In 2018 and 2019, the difference between PIS and ISTI became more pronounced. Then, the police of increasing salary may be more significant than increasing the total budget for research. From Figure 4, it can be seen that by increasing salary to researcher (Bonus), it has a positive influence on research productivity. The increment of productivity can be also supported by the increasing of the total budget for research, but it was not the case from 2012 to 2016, so it cannot be generalized. This kind of trends are important for management decision makers.

 

The bonuses corresponding to 2018, 2019, and 2021 were 20, 19, and 34M soles, respectively. The number of professors benefiting from research bonuses in years 2017, 2018, 2019, 2020 y 2021 were 374, 560, 722, 988 y 1295, respectively.

 

These results show that the researcher's bonus has a clear impact on the scientific production of university professors, as it was in the case of the incentives granted to IPEN researchers in the first five years of the 21st century. Consequently, the policy that increase researcher salary may be absolutely imperative than increasing the total annual budget for research, since by increasing the researcher salary is still possible preserve research indicator trends. It is important to understand that a research project budget has several items that consider human resources, logistics, equipment, services, consumables, etc.

 

5. DISCUSSION AND CONCLUSION

 

The present study presents the trend of scientific publications in IPEN during the period of 2000-2020. It also shows the investments in research and development in Peru. Consequently, it was possible to determine the influence of budget on research productivity. The tripling of IPEN's professional salaries, which took place in 2001, allowed the incorporation of researchers with experience in research. In addition, a production incentive policy was implemented. As a result, production indicators began to grow. The management changed in 2007, which caused the progressive resignation of researchers who were incorporated due to their high scientific production. As a result, the annual number of publications decreased.

 

From the above data we can deduce that a scientific institution responds to incentive policies for researchers and technical personnel to increase their production. An interruption of incentives results in a return to production levels similar to those existing before these policies. Although this experience refers to a research institute in the nuclear field, it can apply to institutes dedicated to other fields of science and technology.

 

Limited research related to management action on research productivity exist. So, interesting conclusions are achieved:

 

(a)   Research activities in IPEN steadily increase during the period 2001 to 2006, the situation was less productive in terms of scientific publication during the period of 2007 to 2020. The number of publications increased from 1 in 2001 to 15 in 2007.

(b)   Inflection due to management strategies and policies can be visualized by the method presented in this paper.

(c)   The policy of raise of researcher salary may be absolutely imperative than increasing the total annual budget for research, i.e., by increasing the researcher salary is still possible preserve research indicator trends. It is also supported by the increase of 50% of salary to professors immerse on research productivity in public universities in Peru.

 

Research productivity changes from manager director to manager director in any research centre, and it happens fundamentally due to political changes. Consequently, it is highly recommended:

 

(a)   to design and implement polices to have autonomous research institutions with the highest research standards;

 

(b)   implement policies to warranty meritocracy and a STRC to produce first-hand information and product development for decision makers and politics;

 

(c)   it is also important to have a Ministry that re-orient research institutes and impulses and support STRC. This minister should be uncharged to gradually enrich the academy with professor from STRC.

 

 

 

REFERENCES

 

Abramo, G., & D’Angelo, C. A. (2015). Evaluating university research: Same performance indicator, different rankings. Journal of Informetrics, 9(3), 514-525.

        https://doi.org/10.1016/j.joi.2015.04.002

Bala, A., & Kumari, S. (2013). Research Performance of National Institutes of Technology (NITS) of India During 2001-2010: A Bibliometric Analysis. Journal of Information and Knowledge50(5), 555–572.

        https://www.srels.org/index.php/sjim/article/view/43774

Coccia, M. (2004). New models for measuring the R&D performance and identifying the productivity of public research institutes. R&D Management, 34(3), 267-280.

        https://doi.org/10.1111/j.1467-9310.2004.00338.x

Coccia, M. (2005). A scientometric model for the assessment of scientific research performance within public institutes. Scientometrics, 65(3), 307–321.

        https://doi.org/10.1007/s11192-005-0276-1

Dhawan, S. M., & Gupta, B. M. (2007). Physics research in India: A study of institutional performance based on publications output. DESIDOC Journal of Library & Information Technology, 27(1), 55-67.

        https://publications.drdo.gov.in/ojs/index.php/djlit/article/view/123

Jeevan, V. K. J., & Sen, B. K. (2007). A scientometric analysis of publications on accelerator-based research from nuclear science centre and Tata Institute of Fundamental Research, India. Malaysian Journal of Library & Information Science, 12(2), 89-97. https://mjs.um.edu.my/index.php/MJLIS/article/view/6998

Jeyshankar, R. (2015). Research productivity of the scientists of indira gandhi centre for atomic research (IGCAR) kalpakkam (chennai): A Scientometric Analysis. Library Philosophy and Practice, 0_1,1-16.

        https://www.proquest.com/docview/1781733301?pq-origsite=gscholar&fromopenview=true

Kademani, B. S., Kumar, V., Surwase, G., Sagar, A., Mohan, L., Gaderao, C. R., Kumar, A., Kalyane, V. L., & Prakasan, E. R. (2005). Scientometric dimensions of innovation communication productivity of the Chemistry Division at Bhabha Atomic Research Centre. Malaysian Journal of Library and Information Science, 10(1), 65–89.

        https://vmis.um.edu.my/index.php/MJLIS/article/view/8479

Kumar, S., & Kumar, S. (2004). Productometric study of scientists of ICAR'S National Research Centre for Soybean (NRCS). Annals of library and information studies, 51(1),11-21.

        http://nopr.niscpr.res.in/handle/123456789/7676

Mondal, D., & Raychoudhury, N. (2018). Research productivity of Saha Institute of Nuclear Physics (SINP), India with special reference to international collaborative experimental consortia. Libr. Philos. Pract, 1-15.

        http://digitalcommons.unl.edu/libphilprac/1863

Montoya, M. (2006). Optimización del Sistema Peruano de Ciencia y Tecnología. Fondo Editorial UNMSM.

Montoya, M. (2011). Políticas para impulsar la ciencia, la tecnología y la innovación tecnológica en el Perú. Fondo Editorial UNMSM.

Neves, P. C., Afonso, O., Silva, D., & Sochirca, E. (2021). The link between intellectual property rights, innovation, and growth: A meta-analysis. Economic Modelling97, 196-209. https://doi.org/10.1016/j.econmod.2021.01.019

Pal, J. & Sarkar, S. (2020). Evaluation of Institutional Research Productivity. DESIDOC Journal of Library and Information Technology, 40(1), 58-69.

        https://doi.org/10.14429/djlit.40.01.14804

Rijcke, S. D., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). Evaluation practices and effects of indicator use—a literature review. Research evaluation, 25(2), 161-169. https://doi.org/10.1093/reseval/rvv038

Surwase, G., Kademani, B. S., & Kumar, V. (2008). Scientometric dimensions of neutron scattering research in India. DESIDOC Journal of Library & Information Technology, 28(3), 3-16. https://doi.org/10.14429/djlit.28.3.173

Upadhye, R. P., Kademani, B. S., Surwase, G., & Kumar, V. (2010). Scientometric dimensions of the nuclear physics division at Bhabha Atomic Research Centre. SRELS Journal of Information Management, 47(4), 437-448.

        https://www.srels.org/index.php/sjim/article/view/44969

Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of informetrics, 10(2), 365-391. https://doi.org/10.1016/j.joi.2016.02.007

Van Raan, A. (1999) Advanced bibliometric methods for the evaluation of universities. Scientometrics, 45(3), 417–423. https://doi.org/10.1007/BF02457601