LLM-as-a-Judge: Engineering Trustworthy Automated Evaluation Frameworks
As Large Language Model (LLM) development shifts toward automated evaluation, the "LLM-as-a-Judge" paradigm has emerged to solve the scalability limitations of human-in-the-loop testing. However, treating these models as infallible oracles leads to unreliable metrics due to systematic stochastic biases. To achieve parity with human-human agreement, organizations must transition from raw scoring to a "laboratory instrument" methodology. This involves mitigating specific failure modes—such as verbosity, position, and self-enhancement biases—through rigorous calibration against "Gold Standard" datasets, the implementation of Chain-of-Thought (CoT) reasoning, and the application of Cohen's Kappa to ensure statistical significance in inter-rater agreement.
Bit-Flip Vulnerability in vLLM Prefix Caching via GPU Rowhammer
vLLM's prefix caching mechanism is susceptible to Rowhammer-induced bit flips within shared KV-cache blocks stored in GPU DRAM. By targeting BF16 precision memory locations, attackers can induce "silent divergence," where LLM outputs remain syntactically coherent but become factually manipulated. This vulnerability allows for targeted attacks on users sharing common prompt prefixes and features persistent error accumulation over time. Because KV-cache blocks lack native integrity protection, these faults evade standard detection and scale linearly with request volume. Remediation requires the implementation of low-overhead checksum verification during the request scheduling phase to validate block integrity before batch processing.