2

Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?

Avatar
Voices Powered byElevenlabs logo

Avatar
user

Extract facts from this paper.

AI: This paper provides guidance on evaluating scientific software in different contexts such as medical software, and identifies different types of metrics that can be used to evaluate the performance of software, including user metrics, infrastructure metrics, and impact and usage metrics. The paper suggests that guidelines and strategies for various types of software and resources can be proposed, and that web presence, citability, and documentation can enable useful metrics for software evaluation. The authors conducted a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI) to gain a deeper understanding of issues hindering software evaluations, and suggested that various aspects of infrastructure, including social media, extensive documentation, feedback mechanisms, and badges about software health metrics can help increase rates of tool usage. The paper also discusses challenges associated with assessing software usage and impact, such as ethical and security concerns, and provides recommendations for improving code health and security, such as using automated tests and static code analysis tools.
Avatar
user

Tell me more about cancer statistics in this paper

AI: The paper doesn't provide specific cancer statistics, but it mentions that the authors performed a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research program funded by the National Cancer Institute.

Authors

Awan Afiaz, Andrey Ivanov, ..., and Carrie Wright

Abstract

Software is vital for the advancement of biology and medicine. Analysis of usage and impact metrics can help developers determine user and community engagement, justify additional funding, encourage additional use, identify unanticipated use cases, and help define improvement areas. However, there are challenges associated with these analyses including distorted or misleading metrics, as well as ethical and security concerns. More attention to the nuances involved in capturing impact across the spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have compelling typical usage metrics. We propose more general guidelines, as well as strategies for more specific types of software. We highlight outstanding issues regarding how communities measure or evaluate software impact. To get a deeper understanding of current practices for software evaluations, we performed a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that developers recognize the utility of analyzing software usage, but struggle to find the time or funding for such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of evaluations of their software.

Follow Us on

0 comments

Add comment
Recommended SciCasts