What are the main challenges and opportunities for research evaluation and assessment in the digital age?

The digital age has seen the rise of new research outputs, such as data sets, software, and interactive visualisations. These new outputs can be difficult to evaluate and assess using traditional methods.

Challenges: ✓Managing and interpreting vast amounts of digital data. ✓Ensuring data security and privacy in the digital research landscape. Opportunities ✓Enhanced accessibility and dissemination of research outputs. ✓Integration of advanced technologies for data analytics and visualization.

In the digital age, the main challenges for research evaluation and assessment revolve around the vast volume of data, the rapid pace of technological change, and the need for new methodologies that can accurately capture the multifaceted nature of digital scholarship. Conversely, the opportunities are equally significant; digital transformation offers innovative tools for data analysis, enables broader dissemination and collaboration, and fosters interdisciplinary approaches that can enrich research evaluation. As a researcher, I see the potential to leverage artificial intelligence, and big data analytics to develop more nuanced and dynamic assessment criteria that reflect the evolving landscape of digital scholarship.

Digital transformation refers to the integration of digital technologies into all aspects of an organization, fundamentally changing how it operates and delivers value to its stakeholders. It involves leveraging digital tools and platforms to streamline processes, enhance efficiency, improve decision-making, and ultimately, drive innovation and growth. From adopting cloud computing and data analytics to implementing artificial intelligence and automation, digital transformation enables organizations to adapt to the evolving digital landscape, stay competitive, and meet the changing needs of customers and users.

In my opinion amount of research being produced is growing exponentially. This makes it difficult to keep up with the latest research and to identify the most important findings.

Opportunities: ✓Diversifying assessment criteria beyond traditional citation counts. ✓Incorporating diverse indicators such as social media mentions, downloads, and online engagement. Challenges: ✓Establishing standardized and universally accepted alternative metrics. ✓Addressing potential biases in alternative metrics.

Alternative metrics, or altmetrics, offer a complementary approach to traditional citation-based metrics for evaluating research impact. They measure the broader influence of scholarly outputs by considering factors such as social media mentions, downloads, and online engagement. Altmetrics provide a more nuanced understanding of the societal and cultural impact of research, capturing its reach beyond academic circles and reflecting real-time interactions and discussions.

Consider: 1. Transparency: Share all stages of research, including data, methods, protocols, and code, to enhance visibility and reproducibility. 2. Accessibility: Ensure research outputs are freely accessible to foster inclusivity and collaboration. 3. Quality Assurance: Implement rigorous checks to maintain the integrity of open data and outputs. 4. Confidentiality: Balance openness with the need to protect sensitive information and uphold security. 5. Interoperability: Promote standards that allow data and findings to be effectively used and integrated by others. 6. Collaboration: Foster a culture of shared learning and growth within the research community. With the right incentives, we can make science more accessible.

To add on this, Open science presents challenges in ensuring quality and credibility but offers opportunities for wider dissemination and collaborative advancements. For example, open peer review can be more transparent but may lead to biases. Conversely, open data allows for reproducibility checks and novel insights from shared resources. In my experience, open science accelerates knowledge exchange and fosters innovation, yet it requires robust frameworks to maintain research integrity in the digital landscape.

✓Challenges: Attribution, ethical data sharing, cultural shifts. ✓Opportunities: Transparency, reproducibility, interdisciplinary collaboration.

I perceive the main challenges in research evaluation and assessment to be the adaptation of traditional metrics to the open science framework and the recognition of diverse research outputs beyond publications. The shift towards open science necessitates the development of new assessment criteria that acknowledge data sharing, public engagement, and collaborative projects. On the flip side, the opportunities are immense; open science can democratize knowledge, increase transparency, and accelerate innovation. It allows for a more inclusive evaluation system that values a wider array of scholarly contributions, fostering a culture where knowledge is freely accessible and shared for the greater good of society.

Open science promotes the transparent and collaborative sharing of research outputs, methodologies, and data. It aims to increase accessibility, reproducibility, and accountability in the scientific process by making research findings and resources freely available to the public. Open science practices encompass open access publishing, open data sharing, and open-source software development, fostering innovation, accelerating discovery, and maximizing the societal benefit of scientific research.

Shut down after just three days, Meta's Galactica AI was designed to write scientific research papers and Wikipedia articles. Trained on 48 million scientific articles and textbooks, Galactica outperformed larger models like GPT-3 and BLOOM on measures such as BIG-bench, Beyond the Imitation Game Benchmark. The only problem? Galactica was prone to spewing scientific misinformation and even generated misleading research, including a Wikipedia-style entry on the benefits of eating crushed glass. Of course, humans can also generate fake science, but large language models might significantly lower the barrier to entry and amplify the scale of the issue. Training objectives need to be clearly defined to ensure scientific accuracy.

Responsible metrics advocate for the ethical and equitable use of quantitative indicators in research evaluation. They emphasize contextual understanding, acknowledging diverse research contributions, and avoiding unintended consequences such as gaming or biases. Responsible metrics promote fair and transparent assessment practices that prioritize quality, integrity, and the broader societal impact of research over simplistic numerical measures.

✓Challenges: Balancing quantitative and qualitative, avoiding unintended consequences. ✓Opportunities: Holistic impact, responsible use, ethical frameworks.

The digital age has amplified the call for metrics that not only measure output but also reflect the societal and ethical impact of research. The difficulty lies in creating a framework that encompasses diverse forms of scholarly communication and resists the reductionist tendencies of traditional metrics. However, this era also presents opportunities to redefine excellence in research through responsible metrics that prioritize transparency, inclusivity, and integrity. By embracing a multifaceted approach to assessment, we can foster a research culture that values contributions to knowledge and society equally, ensuring that metrics serve as a guide rather than a definitive measure of scholarly worth.

Stakeholder engagement involves actively involving relevant individuals or groups in decision-making processes to ensure their perspectives, concerns, and needs are considered and addressed. It fosters collaboration, builds trust, and enhances the relevance and impact of initiatives by incorporating diverse insights and expertise from those affected by or involved in a particular project or endeavor.

✓Challenges: Engaging diverse expectations, bridging academia-industry gap, ensuring participation. ✓Opportunities: Collaboration, diverse perspectives, societal impact.

I once heard a researcher talk about using Game theory to facilitate stakeholder engagement within a particular research context. I believe digital transformation if done correctly should help stakeholders understand each other better and catalyse the translation of the output of research into a public good.

Thing though with AI powered peer review is that the confidentiality of raw manuscripts is not guaranteed. Ideally research should be expose new ways of thinking, doing and being which AI may not be able to assess. Further more, AI is limited to information stored digitally whereas there is a wealth of knowledge such as indigenous knowledge that AI doesn’t have access to. If we replace human peer review with AI, we may unfortunately find that only certain ways of thinking become accepted which are not necessarily the best ways to move humanity forward.

✓Challenges: Adapting to emerging tech, navigating ethical shifts, accommodating interdisciplinary research. ✓Opportunities: Innovation, adaptive frameworks, continuous collaboration.

If thAI as Nikita Jain mentions also spewed out scientific misinformation havingbeen trained ofn so many articles then surely that should not just call into question AI for the intentions stated but indicate that is what it learned from tbe source material. We cannot just assume that Scientific articles are immune to containing bad or misinformation. We have all no doubt seen BS in some articles. One can be sure that the results of the AI were looked at very closely. But the source material could not be vetted by humans due to time and volume. It was assumed I presume that peer reviewed papers would be fault free. Its not so much AI which is the problem but quality of source material

In the digital age, research evaluation faces challenges like data overload and rapid tech changes, demanding updated assessment methods and addressing ethical concerns. Conversely, it offers opportunities such as innovative metrics (altmetrics), enhanced collaboration, and accessibility through digital platforms, alongside quicker feedback loops for research iteration. This dynamic poses a dualistic interplay of challenges and opportunities, shaping research evaluation and assessment’s future.

Points not covered above are Digital Transformation: While the digital age’s impact on research evaluation is discussed, the broader concept of digital transformation, which includes the cultural, organizational, and operational changes, is not explicitly addressed. Stakeholder Engagement: The role of various stakeholders (e.g., researchers, funders, publishers, and policymakers) in shaping research evaluation and assessment in the digital age is not specifically mentioned. Future Directions: There is no direct mention of the potential future developments in research evaluation and assessment, such as predictive analytics, AI-driven assessment tools, or the integration of blockchain technology for enhanced transparency and verification.