Evaluating LLMs for Automated Measurement
Overview
In psychotherapy research, the Experiencing Scale (EXP) is a validated 7-point measure of how deeply a patient explores and engages with their internal emotional state during therapy. Scoring transcripts requires extensive human rater training, making it exceptionally time-consuming and expensive.
Working alongside Brian Yim and Dr. J. Christopher Muran at Adelphi University, our research investigates whether large language models can perform structured human measurement reliably—and how their ratings compare with expert human raters.
Rather than simply running prompt scripts, I engineered the evaluation infrastructure end to end: provisioning cloud GPUs, solving zero-trust networking constraints, building a fault-tolerant Python scoring pipeline resilient to VM preemption, and analyzing the statistical causes of model–human disagreement.
Model Evaluation Matrix
I benchmarked six models in total across two execution tiers to compare frontier proprietary intelligence against self-hosted models capable of running in controlled private-cloud environments:
Proprietary Frontier Tier
Evaluated for peak reasoning and clinical nuance via secure API orchestration:
- Claude(Anthropic) — Known for nuanced reasoning and adherence to complex scoring guidelines.
- GPT-5.4(OpenAI) — Benchmarked for structured extraction and semantic interpretation.
- Gemini(Google) — Long-context synthesis across full-session conversational dynamics.
Self-Hosted Open Weights
Executed locally on isolated cloud GPU instances to minimize external data exposure and support privacy-preserving inference:
- Llama 3.1(Meta) — Open foundational architecture for scalable, reproducible local inference.
- Gemma 3(Google) — Compact, high-efficiency weights benchmarked on specialized NLP rubrics.
- Mistral(Mistral AI) — High-throughput attention mechanism tested across long therapeutic turns.
Cloud GPU Infrastructure
Solving Real Enterprise Cloud Roadblocks
Hosting Llama 3.1, Gemma 3, and Mistral on Google Cloud was not simply spinning up a standard virtual machine. It involved navigating real quota constraints, enterprise security boundaries, and complex network topologies:
GPU Quota Rejection → Spot/Preemptible Pivot
Google Cloud initially rejected our standard on-demand GPU quota request for the project. Rather than stalling progress waiting on administrative approvals, we engineered an operational pivot to Spot / Preemptible GPUs.
This significantly reduced compute costs, but brought a critical trade-off: Google could reclaim the compute instance at any time with minimal warning. Operating under these conditions required designing our software layer to be entirely resilient to unannounced hardware shutdowns.
Enterprise Zero-Trust Networking → IAP Tunneling
The institutional cloud organization enforces strict security policies that block all external/public IP addresses on virtual machines by default. Direct SSH over the public internet was completely locked down.
To establish a secure management pipeline without requesting risky firewall exemptions, we architected an encrypted access pathway using Google Cloud Identity-Aware Proxy (IAP) TCP tunneling. This allowed zero-trust SSH access authenticated through IAM permissions without ever exposing an open port or public IP to the web.
Egress Isolation → Cloud NAT Gateway
Because the VM had no public IP address, it also suffered from zero outbound internet access. This prevented the instance from pulling model weights from Hugging Face, downloading PyTorch CUDA wheels, or installing required Python dependencies.
I diagnosed the gateway routing failure and requested a dedicated Cloud NAT gateway on the VPC network. This granted our isolated VM secure outbound egress to fetch models and updates while preserving strict zero-ingress security for the clinical data.
Custom Scoring Pipeline
825 Model Calls, Overnight Preemption & Zero Data Loss
I authored the complete local scoring and evaluation pipeline in Python from scratch. Because the benchmark required rigorous statistical repetition, building for resilience was paramount:
Catching the Resume Bug Before Production
During a careful code review process across multiple review passes before launching the real run, we caught a critical data-loss bug in the script’s resume logic: when resuming with --yes, the script deleted the existing CSV file outright while the resume logic skipped already-completed pairs—meaning completed work would silently vanish from the final output. We fixed it by having the script rebuild the CSV directly from the completed pairs in the JSON file before continuing, ensuring true fault tolerance before kicking off the actual multi-hour production run.
Surviving Overnight VM Reclaim
The full evaluation encompassed 55 clinical transcripts × 5 repeated passes across all 3 local models (825 total model calls) to measure variance. During the overnight run, Google reclaimed our Spot VM mid-execution. Thanks to the fault-tolerant design, the pipeline resumed seamlessly upon instance restart and completed all 825 calls with zero data loss.
Each of the 55 de-identified transcripts was evaluated 5 independent times by Llama 3.1, Gemma 3, and Mistral (55 × 5 × 3 = 825 calls) under temperature-controlled prompting to quantify intra-model variance, prompt sensitivity, and score convergence.
Statistical Analysis
Diagnosing Inter-Rater Reliability
Beyond the systems and cloud engineering, I spent time digging into the raw cloud-model scores to understand why the initial inter-rater reliability (IRR) numbers between the AI models and human raters were lower than expected:
Models Scored Emotional Peaks Higher
Looking closely at the transcript-by-transcript scores, models tended to score emotional peaks higher than the human raters did. The models reacted more strongly to acute affective language and expressive statements, whereas trained human raters took in the holistic context and evaluated the patient’s overall baseline depth of exploration more conservatively.
Human Ratings Clustered in a Narrow Range
I also observed that the human ratings across the dataset were heavily clustered in a narrow range. In statistical measurement, restricted score variance can substantially depress agreement and correlation statistics, making model–human agreement appear weaker than the underlying qualitative alignment might suggest.
A Useful Diagnostic Observation
Rather than treating the reliability numbers as a mystery, analyzing the data surfaced two grounded explanations: the models’ tendency to over-score emotional spikes, and the statistical compression caused by clustered human ratings. It was a helpful observation that gave us real clarity into how the data and models were actually behaving.
Privacy-Preserving De-Identification
Before any transcript entered cloud APIs or local GPU inference, it passed through a custom Python de-identification pipeline using Microsoft Presidio and spaCy. The pipeline detected and removed the 18 HIPAA Safe Harbor identifier categories across all 55 transcripts before model processing.
Conference
Presentations
Peer-reviewed findings accepted for presentation at international psychotherapy conferences.
SPR North American Conference
Yim, B., Rodriguez, S. & Muran, J. C. (2026, accepted). Automated coding of a psychotherapy measure: Validating the use of large language models in scoring the Experiencing Scale (EXP). Society for Psychotherapy Research North American Conference, New York, NY.
24th World Congress of Psychotherapy
Yim, B., Rodriguez, S. & Muran, J. C. (2026, June). Validating the use of large language models in scoring the Experiencing Scale (EXP). 24th World Congress of Psychotherapy, New York, NY.
Interested in
This Research?
If you're working on something similar or want to collaborate, I'd love to hear from you.