Software Engineering (cs.SE)

Abstract: Stack Overflow, the world's largest software Q&A (SQA) website, is facing a significant traffic drop due to the emergence of generative AI techniques. ChatGPT is banned by Stack Overflow after only 6 days from its release. The main reason provided by the official Stack Overflow is that the answers generated by ChatGPT are of low quality. To verify this, we conduct a comparative evaluation of human-written and ChatGPT-generated answers. Our methodology employs both automatic comparison and a manual study. Our results suggest that human-written and ChatGPT-generated answers are semantically similar, however, human-written answers outperform ChatGPT-generated ones consistently across multiple aspects, specifically by 10% on the overall score. We release the data, analysis scripts, and detailed results at https://anonymous.4open.science/r/GAI4SQA-FD5C.

Abstract: Log parsing, the initial and vital stage in automated log analysis, involves extracting log templates from semi-structured logs to generate structured logs. Nonetheless, current log parsers are limited in effectiveness due to two primary reasons. Firstly, traditional data-driven log parsers heavily rely on heuristics or manually crafted features provided by domain experts, which may not consistently yield optimal performance when applied to diverse log systems. Secondly, existing deep learning-based log parsers necessitate model tuning, which is typically confined to training samples and leads to suboptimal performance across the entire log source. To overcome these limitations, we propose a precise log parsing framework named LogDiv, which leverages the in-context inference capability of large language models. Specifically, LogDiv extracts the hidden semantics from multiple log examples through prompt demonstrations. Without the need for model tuning, LogDiv can directly generate a log template for the target log message by leveraging the semantics provided in the prompt context. Additionally, we introduce a simple yet effective prompt format for extracting the output and enhancing the quality of the generated log templates. To validate the performance of LogDiv, we conducted experiments using 16 widely-used public datasets. The results show that LogDiv achieves state-of-the-art performance with an average parsing accuracy of 97.7%, precision template accuracy of 88.1%, and recall template accuracy of 90.8%.

Abstract: Nowadays, AI-based systems have achieved outstanding results and have outperformed humans in different domains. However, the processes of training AI models and inferring from them require high computational resources, which pose a significant challenge in the current energy efficiency societal demand. To cope with this challenge, this research project paper describes the main vision, goals, and expected outcomes of the GAISSA project. The GAISSA project aims at providing data scientists and software engineers tool-supported, architecture-centric methods for the modelling and development of green AI-based systems. Although the project is in an initial stage, we describe the current research results, which illustrate the potential to achieve GAISSA objectives.