A summary of submission software used in NICE technology appraisals

Objective

This document is intended to be a living document for recording the software used in the evidence submission for NICE technology appraisals (TAs). The motivation is to monitor trends in use of Excel spreadsheet versus code-based models in the development of submission evidence.

Disclaimer: Please be advised that I am an independent entity and am not affiliated with the National Institute for Health and Care Excellence (NICE). This document presents the results of an exploratory analysis conducted through web-scraping. While efforts have been made to ensure accuracy, I cannot guarantee the correctness of individual entries and disclaim any liability thereof.

This document is to be referenced as: Hart R, A summary of submission software used in NICE technology appraisals, 2025, https://doi.org/10.5281/zenodo.15168411

Notice something wrong or missing? Contact us at rhart@darkpeakanalytics.com with more information and we will try to incorporate it

Method

This analysis uses the rvest package and pdftools to go through each of the pages listed as published guidance. No APIs or advanced LLM scraping was used, this was to make the code more accessible and can be forked, run and amended from the repo without need for license or server setup. More information on the specific methods can be found in the code comments in the GitHub Repository

Only Technology appraisals were scraped that were not listed at ‘Terminated appraisals’. The code looks through the ‘Evidence’ tab and identifies the largest and most recent pdf file that contains specific terms (“Committee Papers”, “Document B” and “evidence submission”). It then uses pdftools to read the document.
The software terms that were scraped for were “Microsoft”, “MS Excel”, “in Excel”, “VBA”, ” R ” and “TreeAge”. Lines of the PDF were identified with these terms. These were then cross-referenced with verbs (“built”, “build”, ” constructed”, “developed”, “conducted”, “programmed”, “implemented”, “perform”, “utilises”, “created”) and all lines with software were extracted that were within 2 lines of any selected verb AND either “cost”,“economic” or “model”). Lines were then further cross-references with statistical terminology, removed if they were within 2 lines of any of the selected terms; this was to reduce noise, as often statistical analysis was conducted in R with the economic model being build in Excel.
The extracted lines then were counted for the number and type of software terms present. All TAs that returned software terms that were not only “Microsoft”, “MS Excel” or “in Excel” were checked manually to confirm.

The code for this document is hosted on GitHub at: https://github.com/dark-peak-analytics/NICE-webscraping

Results

Results by year

The graph below shows the number of TAs and the software identified for each by year. ‘No evidence available’ indicates that there was no evidence in the ‘Evidence’ tab to extract. ‘Not stated’ indicates that evidence was identified and scraped but the criteria for extracting the technology were not met.

Results as proportion of total TAs

The pie chart shows each software result as a proportion of all TAs.

Note: Click the legend to remove entries

TAs with evidence submitted using R or VBA

The tables below extract the details of TAs that did not use an Excel spreadsheet to develop the core model engine, instead opting for R or VBA to produce the model engine. The outputs show that code-based engines are being used for a wide variety of conditions and diseases.

TAs that use R for economic evidence submission

TAs that use VBA for economic evidence submission