A couple months ago, I sat down to think about the best way to streamline the design, development, and test of Artificial Intelligence (AI) systems. It’s part of my job, but also something I spend a lot of time thinking about outside of work, because I think it’s important. It’s also kind of like trying to catch smoke: these systems are ones that are designed to learn and change over time. The adaptability is a feature, and a bug.
To make matters worse, the very definition of what ‘counts’ as AI is evolving before our eyes. Six years ago, I started my PhD working on a NASA project centered on developing ‘smart’ systems for use in future deep space exploration missions. The goal was to leverage cutting-edge technology here on Earth—such as intelligent robotics, digital twins, and artificial intelligence—towards solving some of the toughest challenges in human spaceflight. The grant had been inspired by—you guessed it—the ‘smart home’ craze, which now feels like a quaint piece of Tomorrowland.
At the start of this project, ChatGPT hadn’t launched. No one was talking about Large Language Models, or LLMs, because they weren’t really useful yet; those models were still spitting out garbage. When our team discussed AI, we were mostly referring to abstract neural networks and machine learning techniques—fancy data analytics that could enable autonomous systems to function with reduced oversight. Now, to the public, ‘AI’ almost universally means ‘LLMs,’ and by the end of the 5-year project, ChatGPT was a household name. (At graduation, my advisor joked that I had authored one of the last non-AI theses, as though I had caught the last plane out of Saigon.)

Fast forward to 2026, and I’m thinking about how I can adapt some of the lessons learned from my PhD to the current wave of AI. Technology development in my industry can be notoriously slow. A big part of this is the need to test, test, test. Aerospace systems are tested forward, backwards, sideways, upside down—they’re shaken, baked, throttled, and hit. There are entire teams dedicated to building test environments and executing protocols, all for a good reason—space is expensive. When you’re launching a billion dollar satellite, you want it to work.
So, to get a sense of how the broader tech industry was viewing this issue, I asked some friends. I spoke to people who worked with LLMs and other AI systems in software and biotech. Specifically, I asked them how they evaluate their systems “with a human in the loop.” I’ve found that a lot of STEM people don’t like to think about the human in the loop. Humans are sticky. They’re not really that deterministic; they don’t boil down to an equation. Human factors often gets labeled as a ‘soft’ science—don’t get me started—alongside other underestimated fields like psychology or sociology, as though a science only counts as ‘hard’ if you can derive it from Newton’s second law of motion.
But at the end of the day, humans are the ones interacting with technology. Somewhere, at some point, a human will be there: they’ll be the user or the benefactor, the overseer or the approver. And when I asked people in other industries how they address the sticky human subject, I was shocked to hear that for the most part, they don’t.
When it comes to evaluating AI systems, there’s a lot of different methods. OpenAI has historically used a variety of tests, competitions, and scoring metrics to evaluate the performance of their LLMs. These metrics provide a quantifiable, comparable benchmark over which to evaluate these system, and in that respect, they succeed.

Last. year, OpenAI reported that GPT-5 “sets a new state of the art across math (94.6% on AIME 2025 without tools), real-world coding (74.9% on SWE-bench Verified, 88% on Aider Polyglot), multimodal understanding (84.2% on MMMU), and health (46.2% on HealthBench Hard).” This all sounds great, if your goal is to design an LLM to ace tests. But my question was more specific: how are these systems being evaluated with humans in the loop? In other words, are they trustworthy? Usable? Safe?
In human factors research, we talk a lot about TWSA: trust, workload, and situation awareness. In 2008, Parasuraman et al. proposed TWSA as a primary way of predicting human performance in tasks that include automation, which has now extended to any human-machine (or human-autonomy) interaction. These types of tasks can include anything where a human is working with a machine to accomplish some goal. Think: driving, flying a plane, working with a robotic arm, or operating a toaster. The TWSA metrics are defined as:
Trust: “The attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vulnerability.” (Lee and See, 2004)
Workload: “The degree of activation of a finite pool of resources, limited in capacity, while cognitively processing a primary task over time.” (Longo et al., 2022)
Situation Awareness: “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.” (Endsley, 1988)
In layman’s terms, trust describes your own belief that you can rely on a system to help you achieve your task. It isn’t always something you want to maximize—overtrust can lead to accidents, and can cause your own skills to atrophy. On the other hand, when trust in a system is too low, humans simply won’t use it. Workload captures how much of your brain space the task is taking up, and likewise, it is something you do not want too much or too little of. Low workload leads to boredom, fatigue, and accidents. High workload results in stress, overwhelm, and you guessed it, accidents. Situation awareness describes your awareness of surroundings, and is most relevant in tasks where you’re doing something potentially dangerous with split attention, like driving while navigating with your phone. Generally, you want high situation awareness when working with machines. (Side note: this is also why I am anti-touch-screen in every vehicle.)
As it turns out, a lot of AI companies actually do care about these metrics, or at least approximations of these metrics. Returning to my example of GPT-5: OpenAI has seemingly put in a lot of work to enable “more honest” and “safer, more helpful responses” that they describe in both their GPT-5 system card and a recent paper on output-centric safety training. But once again, these changes hinge on reinforcement or supervised-learning style techniques that are used to assess the AI itself, not the human. Yes, the safety improvements did include a human-evaluation campaign—but its hard not to view these changes as reactive, particularly in the wake of the 2024 death by suicide of 16-year-old Adam Raine, who was allegedly coached and encouraged through the act by ChatGPT.

In most cases, developers are actively trying to cut out the human in the loop wherever possible. Many evaluations rely on a technique called LLM-as-a-Judge: basically, you run the output of one LLM through another LLM that has been trained to act a judge of how ‘good’ the response is. The goal of the LLM-as-a-judge is to reduce cost by cutting out human evaluators, but the result feels a bit like dead internet theory: it’s just robots all the way down.
Other established AI evaluation methods include metrics like FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, which was created to break down text generated by LLMs into itemized, individual facts that could be assess independently of long-form answers. While this framework was initially based on human assessments, it has now been automated. And ultimately, these methods center the AI: they tell us nothing about human behavior, or about trust, workload, or other relevant metrics that explain how users interact with and utilize these systems.
Don’t get me wrong—I think many of these metrics are incredibly useful. They are efficient ways to evaluate products, and for low-stakes environments, they’re probably sufficient. But for high-stakes, risky environments, we need something better.
The US Department of Defense recently adopted a standard called Human Readiness Levels, or HRLs, which are based on the Technology Readiness Levels (TRLs). The TRL scale was created by NASA in the 1970s with the goal of measuring the maturity of a given technology, from basic research to completion. If a technology is TRL 1, it means we only understand its basic principles. When it reaches TRL 4, we’ve demonstrated it in a laboratory environment. At TRL 9, it is now fully operational.
The analogue HRLs were established to evaluate, track, and communicate the readiness of a technology for safe and effective human use throughout design and development. Like the automation of LLM evaluation I described above, the goal here is also to reduce the time and cost of system development—except in this case, we don’t give up the human in the loop.
In many cases, human factors aren’t considered until a system is tested for the first time. This can be in a laboratory environment, but is often in field test—which means the system has reached TRL 5 or 6 at least, at which point 70% of the project’s lifecycle costs have already been committed. After this point, the costs of system modifications can increase up to 100 times the cost of changes earlier in the design phase (Gawron, Dennison, & Biferno, 1996). The goal of HRLs is to be able to assess and alter a system’s design during development, rather than relying on lagging indicators that indicate something is wrong.

I recognize that many of these tools—HRLs and measurements of TWSA—are niche and academic. But they’re widely used in the aerospace industry because our systems operate in high-stakes, high-risk environments, and I think that the AI industry could learn a lot from thinking about the human in the loop. Rather than using human experts as judges or evaluators of LLM outputs, I encourage developers to flip the switch, and study how humans interact with their systems. Do users trust the system? Do they trust it too much? How are they using it? How do they want to be using it?
Put another way: if I were to assign an HRL to GPT-5, I would maybe give it a level 5. I’m not convinced the developers have put any thought towards human-centered requirements, or evaluated their users outside of asking, “which answer did you like better?”
Again, this wouldn’t really be a problem, except that we’re using LLMs for more and more each day. We’re asking them to create our grocery list, design marathon training plans, and give us medical advice. Do I trust OpenAI to have the users’ best interests at heart? No, and you shouldn’t, either. But a sustainable, long-term product should be designed with the user in mind—not just a math score. In the end, it doesn’t matter how accurate your LLM is if the human doesn’t trust it. It doesn’t matter how powerful the system is if it ultimately increases the workload of the human operator because they have to keep correcting it, prompting it, or otherwise overseeing it.
Like anything else, AI is a tool. Let’s learn how we use it.

