TY - BOOK
T1 - The Automated Generation of Source Code Summaries Using Natural Language Processing Techniques
AU - Phillips, Jesse
PY - 2026/9/30
Y1 - 2026/9/30
N2 - This thesis is about the creation of new techniques for training and prompting neural source code summarisation (NSCS) models in order to automatically generate high-quality source code summaries.
Source code summarisation is an important but time-consuming task for many software developers and maintainers. Previous work finds that source code summaries aid developers in comprehending, maintaining, and collaborating on code. However, due to the cost and time constraints faced by developers, summarisation often becomes a low-priority task. NSCS techniques address this by removing the need for the developer to write summaries. There are issues with current NSCS methods to be addressed; from the way in which data is prepared for model training to the way in which models are evaluated, leading to models which are expensive to train and perform poorly.
I consider, in this thesis, three areas of the model building process: data processing, model training, and model evaluation. I have developed an improved method for cleaning and preprocessing both source code and natural language (NL) summaries used for training models, building on work by Mahmud et al. (2021). I have developed a new training method, "Metric-Oriented Training", which uses Natural Language Generation (NLG) metrics to select only the best-performing training epochs to carry forwards and further refine. Throughout these tasks, I have analysed current NLG metrics and their use in NSCS, as well as applied novel Large Language Model (LLM)-based metrics to NSCS tasks, and analysed the correlation between a wide range of metrics and human evaluation of NSCS.
I have found that through implementing a new, thorough method of cleaning and preprocessing the data used to train NSCS models, I have been able to train models that produce summaries which perform better against a large set of NLG metrics - specifically, I observed an improvement of 7.67% Smoothed BLEU-4 and 12.11% ROUGE-L. I have also found that "Metric-Oriented Training" allows models to achieve comparable or better results than using traditional training - In my findings, I observed an improvement of between 7.3% and 18.32% Smoothed BLEU-4. I have shown that traditional and LLM-based NLG metrics can be used to measure the quality of a summary in the NSCS domain, sharing high rank correlation coefficients when compared and contrasted to each other. I have also shown the capability of LLMs to perform NSCS tasks, and the improvements which can be made to the results provided by LLMs using prompt engineering.
The key implications of this work are the contributions made to data processing, model training, prompting, and model evaluation for NSCS models. Much of this work can also be applied in a wider Machine Learning (ML) context to build better models for other purposes as well. Through the dataset cleaning methods, training methods, and LLM prompting methods described in this thesis, it is possible to train and deploy better NSCS models than before, in agile working environments to allow developers to create well-summarised code while reducing the cognitive load of that task on them.
AB - This thesis is about the creation of new techniques for training and prompting neural source code summarisation (NSCS) models in order to automatically generate high-quality source code summaries.
Source code summarisation is an important but time-consuming task for many software developers and maintainers. Previous work finds that source code summaries aid developers in comprehending, maintaining, and collaborating on code. However, due to the cost and time constraints faced by developers, summarisation often becomes a low-priority task. NSCS techniques address this by removing the need for the developer to write summaries. There are issues with current NSCS methods to be addressed; from the way in which data is prepared for model training to the way in which models are evaluated, leading to models which are expensive to train and perform poorly.
I consider, in this thesis, three areas of the model building process: data processing, model training, and model evaluation. I have developed an improved method for cleaning and preprocessing both source code and natural language (NL) summaries used for training models, building on work by Mahmud et al. (2021). I have developed a new training method, "Metric-Oriented Training", which uses Natural Language Generation (NLG) metrics to select only the best-performing training epochs to carry forwards and further refine. Throughout these tasks, I have analysed current NLG metrics and their use in NSCS, as well as applied novel Large Language Model (LLM)-based metrics to NSCS tasks, and analysed the correlation between a wide range of metrics and human evaluation of NSCS.
I have found that through implementing a new, thorough method of cleaning and preprocessing the data used to train NSCS models, I have been able to train models that produce summaries which perform better against a large set of NLG metrics - specifically, I observed an improvement of 7.67% Smoothed BLEU-4 and 12.11% ROUGE-L. I have also found that "Metric-Oriented Training" allows models to achieve comparable or better results than using traditional training - In my findings, I observed an improvement of between 7.3% and 18.32% Smoothed BLEU-4. I have shown that traditional and LLM-based NLG metrics can be used to measure the quality of a summary in the NSCS domain, sharing high rank correlation coefficients when compared and contrasted to each other. I have also shown the capability of LLMs to perform NSCS tasks, and the improvements which can be made to the results provided by LLMs using prompt engineering.
The key implications of this work are the contributions made to data processing, model training, prompting, and model evaluation for NSCS models. Much of this work can also be applied in a wider Machine Learning (ML) context to build better models for other purposes as well. Through the dataset cleaning methods, training methods, and LLM prompting methods described in this thesis, it is possible to train and deploy better NSCS models than before, in agile working environments to allow developers to create well-summarised code while reducing the cognitive load of that task on them.
U2 - 10.17635/lancaster/thesis/3440
DO - 10.17635/lancaster/thesis/3440
M3 - Doctoral Thesis
PB - Lancaster University
ER -