Robert Gray Atkins

An academic program evaluation system (PES) should enable smart, data-informed decisions about new programs to start, existing programs to grow, and a few programs that may need to be phased out. AI has been the foundation of PES for over a decade. More recently, AI is augmenting the traditional PES, making it more accessible and giving it some superpowers.

Changing the Rules

While it may not be obvious, the traditional labor data often used to evaluate programs combines rule-based systems and AI. The rules that align academic programs with career occupations were developed by hand. In most cases, these rules were never validated: Few checked to see if graduates of a given program went to work in the occupations they were trained to enter. Using web scrapers and AI tools, especially natural language processing, analysts now have data on tens of millions of graduates and what jobs they took. Unsurprisingly, the analysts found that the hand-drawn links from programs to jobs had little relationship to real-world results. The old links found five to 10 occupations for most graduates of a given program; the AI-enabled analysis of real-world data shows that graduates of most programs go into hundreds of different occupations.

Presenting the Data

AI has also revolutionized the presentation of program data. Typically, data on program markets was presented in rows and columns of numbers—lots of them. For many, the data was a little overwhelming and difficult to interpret. People wanted a report they could read in straightforward text. Consultants were happy to do the analysis and write the reports, which cost thousands of dollars and took at least six weeks to prepare. Gray Decision Intelligence’s PES can now feed data on over 50 market metrics into a large language model (LLM) and produce a text report in two minutes.

Taming the AI Beast

Reliably converting a numeric report into text takes a few months of work. Much of this work has little to do with AI itself; as with most AI applications, the challenge is assembling the data. When you set out on the AI journey, be sure to bring a few data heavy-lifters, more formally known as data engineers; they will do the hard work of assembling and cleaning up your data. Then, you have to tame the unruly beasts of AI. Even if you give an LLM a set of numbers, it will often make up a few more, do some strange calculations, like averaging the number of jobs with the number of colleges, and miscalculate other numbers. To make the LLM work, you have to develop a prompt that fences in the LLM and makes it use the exact data and calculations you provide. Then, you need to run hundreds of tests to be sure it behaves itself.

Calling on the Superpowers

Now for the superpowers: If you apply machine learning to about five million data points on higher education institutions and program size, you can accurately estimate the size of potential new programs at an institution. Gray DI’s Predict Program Size tool can perform this minor miracle for any IPEDS program at any institution. Interestingly, AI can even figure out an institution’s strengths (e.g., healthcare or engineering) and adjust its predictions accordingly. For example, Predict’s estimated enrollment for a cybersecurity program at a liberal arts college would be lower than its estimate for cybersecurity at a more technical college. As a side benefit, Gray DI’s Predict Program Size includes pro forma financials so that you can estimate seven-year budgets, cash flow, and breakeven for new programs, all in under an hour.

This combination of a text interface with advanced data, predictions, and cash flow models enables your faculty and administrators to use data to inform their decisions when, before, they might have shied away from all the time and analysis required. They help departments develop sound estimates of program financials so there are no unpleasant surprises when programs roll out. Most importantly, these more advanced program evaluation systems enable you to make better, faster decisions to stay one step ahead of the competition—and well away from the demographic cliff.