Sensory Hallucinations in AI: A Powerful KAIST Fix

A research team at KAIST has published two methods for cutting sensory hallucinations in multimodal AI — the failure mode where a model misreads what a sensor is physically telling it, or invents a perception in one channel because a different channel suggested it. The work was announced on 31 July 2026 and covers thermal cameras, depth sensors, X-ray imagery, and audio-visual video, which is to say the exact conditions where a camera-trained model is least trustworthy and most likely to be deployed anyway.

The framing in the announcement is unusually concrete for an academic release. Sensory hallucinations are not described as a benchmark score, they are described as a robot in smoke, a car at night, a drone with a thermal payload, and a security scanner at an airport. That is the correct framing, because sensory hallucinations only matter when a system has to act on a perception that nobody is going to double-check in time.

What makes the work worth reading rather than filing is the cost structure. Both methods were explicitly designed to avoid retraining. One needs a small amount of sensor-specific data and a fine-tuning pass. The other needs nothing at all — it changes how the model decodes at inference and can be dropped into an existing deployment like a plug-in. In a field where the standard answer to sensory hallucinations is “collect more data and train a bigger model”, two methods that work without doing either are a different kind of contribution.

This article covers what sensory hallucinations actually are and why they differ from the text hallucinations everyone already argues about, the RGB bias that causes them, how KAIST’s DNA optimisation and MAD decoding each attack a different half of the problem, what the reported numbers do and do not establish, where sensory hallucinations do the most damage in production, how to evaluate a system for them, and what this line of work still leaves unsolved.

Sensory Hallucinations in AI: The Quick Answer

Sensory hallucinations are perception errors in multimodal models that come from the sensing side rather than the language side. A text hallucination is a model inventing a fact from language priors. A sensory hallucination is a model misreading a physical signal — calling a hot region in a thermal frame a light reflection — or importing a perception across modalities, such as reporting an engine sound because a car is visible in a silent clip.

KAIST’s team, led by Professor Yong Man Ro in the School of Electrical Engineering, attacked both halves. Diverse Negative Attributes (DNA) optimisation teaches a vision-language model the physical characteristics of non-RGB sensors using the model’s own wrong answers as the training signal. Modality-Adaptive Decoding (MAD) stops one modality bleeding into another at generation time, with no training at all. The two are complementary rather than competing: DNA fixes what the model understands, MAD fixes what the model does with what it understands.

Question	DNA optimisation	MAD decoding
Failure it targets	Misreading sensor physics (thermal, depth, X-ray)	Cross-modal bleed between vision and audio
Type of fix	Fine-tuning with sensor-aware attributes	Inference-time decoding control
Training required	Small amount of sensor-specific data	None at all
Where it acts	During adaptation of the model	At the moment tokens are generated
Deployment shape	A tuned checkpoint	A plug-in over an existing checkpoint
Benchmark introduced	VS-TDX, for diverse vision sensors	Evaluated on AVHBench and CMM
Published	IEEE Transactions on Image Processing	CVPR, June 2026
Reported gain	Improved multi-sensor reasoning across VLMs	Up to 8.7% on CMM, 4.7% on AVHBench
Best for	Fleets running fixed, known sensor types	Systems that cannot be retrained or re-certified
Main limitation	Still needs labelled sensor data	Cannot teach the model anything new

What KAIST Actually Announced About Sensory Hallucinations

The announcement, distributed on 31 July 2026 and carried by EurekAlert, bundles two separate papers under a single practical claim: that sensory hallucinations in multimodal large language models can be materially reduced without the large-scale retraining that normally accompanies a robustness improvement.

The first paper is Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking, published in IEEE Transactions on Image Processing, with Sangyun Chung and Youngjun Yoo as co-first authors and Professor Ro as corresponding author. Its preprint introduces both the DNA optimisation method and the VS-TDX benchmark.

The second is MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models, presented at CVPR in June 2026, with Chung as first author. The camera-ready paper is in the CVPR open-access repository and the implementation is public on GitHub under Apache 2.0.

Professor Ro’s stated summary is that the work “reduces AI’s sensory bias and misperceptions without large-scale retraining”, positioning it as a foundation for trustworthy multimodal AI in industrial settings rather than a leaderboard result. The funding line is worth noting for what it tells you about the intended application: the Institute of Information & Communications Technology Planning & Evaluation’s Human-Centered AI programme, plus Center for Applied Research in Artificial Intelligence grants from Korea’s Defense Acquisition Program Administration and Agency for Defense Development.

Read together, the two papers describe a single position on sensory hallucinations: they are not one problem with one cause, and the cheapest fix for each cause is different.

Why Sensory Hallucinations Are Not the Same as Text Hallucinations

The word “hallucination” has been so thoroughly attached to chatbots inventing citations that it now obscures more than it explains. Sensory hallucinations are a genuinely different failure with a different cause, a different detection method, and a different cost profile.

A text hallucination happens when a language model produces a fluent claim unsupported by any input, driven by the statistical pull of its training distribution. You detect it by checking the claim against a source. The cost is usually embarrassment, occasionally liability.

Sensory hallucinations happen when a model has real sensor input in front of it and interprets that input incorrectly, or lets one input channel overwrite another. There is no source to check against, because the sensor reading is the source. Detection requires ground truth about the physical scene, which in most deployments does not exist at the moment the decision is made.

That asymmetry is the reason sensory hallucinations deserve their own category. A system that hallucinates text can be caught by a reviewer with a search engine. A system that produces sensory hallucinations in a smoke-filled corridor cannot be caught by anyone, because the only entity that can see into the corridor is the system doing the hallucinating.

The cost profile diverges just as sharply. A wrong citation is recoverable. A thermal-camera model that reports a bright region as a reflection when it is a person is not producing a recoverable error, and the same is true of a depth model that misjudges the geometry of an obstacle. This is why the discipline that has grown up around AI oversight and its quieter failure modes applies with more force to perception systems than to text systems, not less.

The RGB Bias That Causes Sensory Hallucinations

The mechanism behind most sensor-side sensory hallucinations is straightforward once stated: vision-language models are trained overwhelmingly on ordinary colour photographs, and they carry the assumptions of that data into images that were produced by entirely different physics.

An RGB photograph encodes reflected visible light. Brightness means more light arrived at the sensor. A thermal image encodes emitted infrared radiation, so brightness means a warmer surface. A depth map encodes distance, so brightness means near or far depending on the convention. An X-ray encodes attenuation, so brightness means the material absorbed less radiation. Four images, four incompatible meanings for the same pixel value.

A model that has internalised only the first meaning will read the others through it. That is exactly the error KAIST calls out: mistaking bright areas in thermal images for simple light reflection. It is not a random mistake. It is a systematic, predictable consequence of the training distribution, and it produces sensory hallucinations that are confidently and consistently wrong rather than noisily wrong.

The KAIST preprint states the finding plainly — current models “process multi-vision sensor images without deep understanding of sensor information, disregarding each sensor’s unique physical properties”, which restricts their ability to answer questions that require reasoning about what the sensor is actually measuring.

This is a more uncomfortable result than it first looks. A model with RGB bias will often still name objects correctly in a thermal frame, because shape survives the modality change. What it loses is the reasoning — the ability to say why a region is bright, what that implies about the object, and whether the reading is consistent with the rest of the scene. Object naming looks like success on a shallow test while sensory hallucinations remain fully intact underneath. Anyone who has tuned a perception stack on top of convolutional architectures for image recognition will recognise the pattern: the accuracy metric moves before the understanding does.

DNA Optimization: Teaching Models Out of Sensory Hallucinations

Diverse Negative Attributes optimisation is the KAIST answer to the RGB bias. Its core idea is to use the model’s characteristic wrong answers as the teaching material rather than treating them as noise to be averaged away.

Conventional fine-tuning shows a model correct examples and pushes probability mass toward them. That works, but it is data-hungry, and for sensor modalities the data is scarce and expensive — thermal, depth, and X-ray corpora with rich language annotation are nothing like as plentiful as captioned photographs. Sensory hallucinations persist because there is not enough positive data to overwrite the RGB prior.

DNA inverts the emphasis. It constructs diverse negative attributes — plausible but physically wrong descriptions of what the sensor is showing — and trains the model to separate them from the correct interpretation. The “bright means reflection” reading of a thermal image becomes an explicit negative rather than an unexamined default. Because negatives can be generated systematically around known failure modes, this converts a data collection problem into a data construction problem, which is enormously cheaper.

The result the team reports is that models trained this way gain the ability to infer object states in darkness or smoke, which is the practical shape of the same claim. The model is not seeing anything new; it is finally reading correctly what the sensor was always reporting.

There is a second-order benefit worth naming. Because DNA targets the physics rather than the objects, a model corrected this way tends to generalise across scenes rather than memorising a particular dataset. Sensory hallucinations that stem from a wrong prior get fixed at the level of the prior.

Inside Sensor-Aware Attributes Fine-Tuning

DNA does not operate alone. The preprint pairs it with Sensor-Aware Attributes Fine-Tuning, a parameter-efficient adaptation step that leaves the model architecture untouched and learns sensor-specific attribute representations instead of retraining the backbone.

The architecture-invariant property is the part that matters commercially. A method that requires you to modify the model structure requires you to rebuild your serving stack, revalidate your latency profile, and in regulated settings redo whatever qualification you did to get the original model approved. A method that produces a tuned checkpoint of the same shape requires none of that.

The data efficiency claim is the other half. The paper describes leveraging “minimal sensor-specific data to enable robust learning of non-RGB characteristics” while overcoming the RGB-centric biases already present. In practice this is the difference between an organisation with a few thousand annotated thermal frames being able to address its sensory hallucinations and being told to go and collect a million more.

It is worth being precise about what this does not claim. SAFT with DNA is still supervised adaptation. It needs labelled sensor data, just far less of it, and the quality of the negatives matters as much as the quantity of the positives. Teams that have run adaptation projects before will recognise the trade: the labelling burden drops, and the design burden — deciding what the model gets wrong, and constructing negatives that isolate it — goes up.

VS-TDX: The Benchmark Built to Measure Sensory Hallucinations

Neither method would mean much without a way to measure sensory hallucinations, and the honest state of the field before this work was that no comprehensive public benchmark existed for sensor-specific understanding.

VS-TDX is KAIST’s answer, described in the announcement as the first comprehensive benchmark for evaluating diverse vision sensors. Its purpose is to test whether a model understands what a sensor is measuring, not merely whether it can name the objects in the frame. The earlier version of the work called this the Multi-vision Sensor Perception and Reasoning benchmark, which describes the intent even more directly.

The distinction between perception and reasoning is the whole point of the benchmark design. A perception question asks what is in the image. A reasoning question asks what the sensor reading implies — why that region is bright, what that means about temperature or distance or density, and whether a proposed interpretation is physically coherent. Sensory hallucinations live almost entirely in the second category, which is why benchmarks built around object naming were never going to surface them.

Benchmarks shape research agendas more than papers do. Object hallucination in ordinary images got serious attention once POPE gave everyone a shared number to move. Cross-modal work accelerated once AVHBench and CMM existed. A public benchmark for sensor understanding is likely to do the same for sensory hallucinations, and its existence may end up mattering more than either method it was built to validate.

MAD: Suppressing Sensory Hallucinations at Decoding Time

The second paper addresses a different class of sensory hallucinations entirely. Here the sensors are understood correctly, but they interfere with each other.

The canonical example is the one KAIST leads with: a model that claims to hear a sound because an object appears in the video. Nothing is wrong with the audio processing and nothing is wrong with the vision processing. The failure is in the fusion — a learned association between “car visible” and “engine audible” that fires regardless of what the audio track actually contains.

This is what the field calls cross-modal hallucination, and it is a distinct problem from unimodal hallucination. As the MAD paper puts it, instead of generating unsupported content purely from language priors, the model produces incorrect predictions due to interference between modalities. The Curse of Multi-Modalities benchmark traced the same phenomenon to two contributors: over-reliance on unimodal priors, and spurious inter-modality correlations learned from statistically frequent but non-causal pairings.

MAD’s contribution is that it fixes this at decoding time. There is no fine-tuning step, no new data, and no checkpoint to redistribute. The method is a modification to how the model turns internal representations into output tokens, which means it can be applied to a model that is already in production without touching anything else in the stack.

For a class of sensory hallucinations that arises from learned correlations baked into the weights, being able to counteract them without editing the weights is a genuinely useful property.

How Modality-Adaptive Decoding Actually Works

The mechanism is worth understanding because it explains both the strength and the ceiling of the method.

MAD runs in two stages. First, the model self-assesses which modalities the current task actually requires — effectively asking itself whether this question is about what is visible, what is audible, or both — and produces relevance scores. Second, it computes output logits under different modality configurations and fuses those branches using the relevance scores as weights.

The effect is that when a question is fundamentally about audio, the audio-conditioned branch dominates and the visual branch loses its ability to inject content that is not there. When the question needs both, both contribute proportionately. The interference channel that produces cross-modal sensory hallucinations is narrowed by construction rather than by hoping the model behaves.

This places MAD in the same family as contrastive decoding approaches, which have become the standard training-free tool for hallucination suppression since Visual Contrastive Decoding showed that contrasting a model’s distribution against a deliberately degraded input suppresses tokens favoured by priors rather than by evidence. MAD’s novelty is that the contrast is drawn across modality configurations and the weighting is derived from the model’s own assessment of what the task needs, rather than fixed in advance.

The authors’ conclusion — that “explicit modality awareness through self-assessment is crucial for robust multimodal reasoning” — is the load-bearing claim. If a model can be induced to state which sensory channel matters, a large share of cross-modal sensory hallucinations can be suppressed without teaching it anything new.

The cost is that decoding branches must be computed for multiple modality configurations, which means more compute per generated token than plain decoding. For an offline analysis pipeline that is irrelevant. For a real-time perception loop on an embedded platform it is a budget question that has to be answered on the target hardware.

The Numbers MAD Reports Against Cross-Modal Sensory Hallucinations

The reported results are specific, which is welcome, and narrow, which is worth stating alongside them.

MAD was evaluated on two audio-visual models. On VideoLLaMA2-AV it improved results by 7.8% on CMM and 2.0% on AVHBench. On Qwen2.5-Omni the corresponding figures were 8.7% and 4.7%. The public repository provides evaluation scripts for both benchmarks and supports Qwen2.5-Omni at 3B and 7B alongside VideoLLaMA2-AV.

Those two benchmarks are well chosen for the claim. AVHBench, itself from KAIST, was built specifically to probe hallucinations arising from cross-interactions between audio and visual signals. CMM covers roughly 1,200 curated samples with about 2,400 probing questions across video-only, audio-only, and paired conditions, and was designed to expose exactly the unimodal-prior and spurious-correlation failures MAD targets.

The honest reading is that these are meaningful single-digit to high-single-digit gains on the specific measurement of cross-modal sensory hallucinations, obtained without training, on two model families. That is a real result. It is not a claim that sensory hallucinations have been solved, and the paper does not make one.

It is also worth separating the two papers when interpreting results. The MAD numbers say nothing about thermal or X-ray sensory hallucinations, which are DNA’s territory, and the DNA results say nothing about audio-visual interference. The press release bundles them because they share an author and a theme, not because they were jointly evaluated.

Why Training-Free Matters More Than the Percentage Points

The most consequential property of this work is not any individual number. It is that one method needs a little data and the other needs none.

Consider what the standard remedy for sensory hallucinations costs. You collect and annotate sensor data in the target modality, you retrain or heavily fine-tune a large multimodal model, you revalidate the entire behaviour surface because a full retrain can move anything, and in a regulated or safety-certified context you repeat whatever qualification process got the previous version approved. That is a project measured in quarters and headcount, and it is why most organisations ship perception systems with known sensory hallucinations rather than fix them.

A decoding-time method changes the shape of that entirely. It is a code change in the inference path. It can be A/B tested, it can be feature-flagged, it can be rolled back in minutes, and it does not invalidate a model qualification because the weights are identical. For teams that have discovered how badly they were underestimating failure rates across multiple AI models, an intervention with that risk profile is worth more than a larger gain that requires a retrain.

The DNA half sits between the two extremes: cheaper than a retrain, more involved than a config change, and appropriate when the sensory hallucinations you are fighting are physical misreadings the model genuinely does not know better than.

The strategic point is that both methods are additive to an existing stack rather than a replacement for it. Neither asks you to abandon the model you already validated.

Sensory Hallucinations at Night: The Thermal Camera Problem

Night operation is the headline application, and it is the cleanest illustration of why sensory hallucinations are a physics problem before they are a machine learning problem.

At night an ordinary camera has almost nothing to work with. A thermal camera has plenty, because it never depended on visible light in the first place — it reads emitted heat, so darkness costs it nothing. This is why every serious night-capable platform, from search-and-rescue drones to driver assistance systems, carries thermal.

The catch is that the model reading the thermal stream was trained on daylight photographs. It sees a bright blob and reaches for the interpretation that bright means illuminated. In a thermal frame, bright means warm — which is to say it means engine, exhaust, animal, or person. The gap between those two readings is the entire safety case, and sensory hallucinations at night live precisely in it.

DNA optimisation attacks this at the level of the prior. By constructing negatives around the reflection reading and forcing separation from the thermal reading, the model stops treating brightness as a lighting cue. What the announcement describes as inferring object states in darkness is that correction stated in application terms.

The reason this matters more than a benchmark delta is that night-time perception has no fallback. In daylight, a hallucinated interpretation of one channel can be cross-checked against a colour image. At night there is no colour image worth checking against, so sensory hallucinations propagate straight into the decision.

Sensory Hallucinations in Smoke: Why Rescue Robots Fail

Smoke is the harder case, and the one the KAIST announcement leads with for good reason.

Smoke scatters and absorbs visible light, which blinds ordinary cameras. It also degrades lidar, because particulates return spurious signals that make depth measurements unreliable and generate false detections. Thermal imaging is the modality that survives, because heat radiation passes through smoke far better than visible light does — which is why thermal drones are standard equipment for fire crews locating hotspots and people through heavy smoke.

But surviving the smoke is only the sensor’s job. The model still has to read the surviving signal correctly, and a model with RGB bias reads a smoke-filled thermal frame badly in a specific way: it treats thermal gradients as lighting gradients, which turns a heat source into a light source and a person into an artefact.

The stakes here are as high as they get in perception. A robot performing a mission in a smoke-filled building is there precisely because no human can be, and there is no operator with a better view available to overrule it. Sensory hallucinations in that setting are not a degraded user experience; they are a missed casualty or a route into a collapsing structure.

There is a further wrinkle that makes smoke harder than darkness. Darkness is a clean absence of one signal. Smoke is an active corruption of several — it attenuates thermal somewhat, it wrecks lidar, it destroys visible light, and it does all three unevenly across the frame. A model that fuses modalities under those conditions is exactly the model most exposed to cross-modal sensory hallucinations, which is the failure MAD addresses. Smoke is the one environment where both KAIST methods have something to contribute at once.

Depth Sensors and the Geometry Class of Sensory Hallucinations

Depth is the least discussed of the three sensor types in the paper and produces the most insidious sensory hallucinations, because depth errors look plausible.

A depth map is a picture of distance. Its failure modes are systematic: reflective surfaces return nothing, transparent surfaces return the wrong thing, edges produce interpolation artefacts, and different sensing technologies encode near and far in opposite directions. A model without an understanding of the depth modality reads a depth map as a strangely coloured photograph and produces a description that is fluent, confident, and geometrically wrong.

That is the dangerous shape. A depth-derived sensory hallucination does not look like an error. It looks like a scene description with a subtly incorrect spatial relationship in it, and a downstream planner will act on the incorrect relationship without ever flagging that anything was uncertain.

For robotics this is the class of failure that matters most, because manipulation and navigation are entirely about geometry. Work on training robots in simulated environments has made progress on the data problem — the approach behind virtual playgrounds that generate training data for robots exists precisely because real depth data at scale is painful to collect. But synthetic depth data teaches a model what depth maps look like, not what they mean. Closing the meaning gap is what DNA-style optimisation is for.

X-Ray Screening and Security-Critical Sensory Hallucinations

Airport X-ray screening is the application in the KAIST list with the strangest risk profile, and it deserves separate treatment.

X-ray imagery encodes attenuation. Dense materials absorb more radiation and appear differently from light ones, objects overlap transparently rather than occluding each other, and the false-colour palettes used in security screening are conventions rather than depictions. Every one of those properties is alien to a model trained on photographs, and every one of them generates sensory hallucinations of a particular kind: confident readings of overlapping objects as single objects, or of material density as colour.

The operational context makes it worse. Screening is a high-volume, high-throughput task with an extreme class imbalance — the overwhelming majority of bags contain nothing of interest — and human operators are being assisted rather than replaced. That combination produces automation bias. An assistant that is right almost all the time trains its operator to stop looking, which means the rare sensory hallucination arrives at exactly the moment nobody is positioned to catch it.

This is why sensor-specific understanding, rather than object detection accuracy, is the right thing to measure in screening. A detector can hit a high score on a curated dataset while remaining unable to reason about what the attenuation values imply. VS-TDX-style evaluation, which asks about the sensor rather than the objects, is a much better proxy for whether sensory hallucinations will appear in the field.

Medical Imaging: The Highest-Stakes Sensory Hallucinations

Medical image analysis is the other application named in the announcement, and it carries the highest consequence per error of anything on the list.

Medical imaging is a collection of non-RGB modalities with strong physical semantics. CT encodes radiodensity on a calibrated scale. MRI intensity depends on the pulse sequence, so the same tissue is bright in one sequence and dark in another. Ultrasound is full of physics-derived artefacts — shadowing, enhancement, reverberation — that are diagnostically meaningful to a trained reader and pure noise to a model that does not know they exist.

A model with RGB priors reading these images produces sensory hallucinations that are indistinguishable from findings. Calling an acoustic shadow a lesion is not a formatting error; it is a fabricated clinical observation delivered in the register of a real one.

Regulatory frameworks for clinical AI already require validation against specific indications and imaging protocols, which is a partial defence. It is only partial, because validation catches the failure modes present in the validation set, and sensory hallucinations arising from misunderstood modality physics can be systematic across an entire imaging protocol rather than distributed randomly. Anyone building in this space — and the constraints in medical imaging software development are unforgiving — should treat sensor-understanding evaluation as a distinct requirement from diagnostic accuracy, because a system can pass the second while failing the first.

The same reasoning drives the movement of clinical inference on-premises. The argument for private AI in healthcare settings is usually made about data residency, but it applies to sensory hallucinations too: a locally hosted model is one you can evaluate, patch at the decoding layer, and roll back, rather than one whose behaviour changes underneath you.

Autonomous Vehicles and the Cost of Sensory Hallucinations

Autonomous driving is where sensory hallucinations have the most public history, and where the multi-sensor argument was settled a decade ago.

Every serious autonomous platform fuses cameras, radar, and usually lidar, because no single modality survives every condition. Cameras fail in darkness and glare, lidar degrades in fog, rain and smoke as particulates scatter returns, and radar has coarse resolution. Fusion is not an optimisation; it is the safety architecture.

Fusion is also the mechanism by which cross-modal sensory hallucinations enter. When channels disagree, something has to arbitrate, and a model that has learned spurious correlations between modalities will resolve disagreement by inventing agreement. That is the failure MAD’s modality weighting is designed to prevent — the branch that should not be driving the answer stops being able to.

The cost asymmetry in this domain is total. A false negative at night is a collision. A false positive is a phantom brake event, which is itself a collision risk on a motorway. Both are sensory hallucinations, they pull in opposite directions, and no single confidence threshold fixes both — which is why fixing the underlying misinterpretation is worth more than tuning the threshold.

There is a structural point here that generalises beyond vehicles. Adding sensors increases the number of conditions the system can handle and simultaneously increases the surface area for cross-modal sensory hallucinations. More modalities is more robustness only if the fusion is sound. Otherwise it is more ways to be confidently wrong.

Drones, Thermal Payloads, and Sensory Hallucinations in the Field

Unmanned aerial vehicles with thermal cameras are the third application named, and they are the one where the technology is already in daily operational use rather than in trials.

Thermal drones are standard equipment for search and rescue and fire response. AI layers that watch the thermal feed in real time, score every heat signature for the probability of being a person, and surface candidates to the pilot have compressed searches that used to take days into hours. The value is not in dispute.

What is in dispute is the failure rate under the conditions that make the deployment necessary in the first place. A thermal signature in the field can be a person, an animal, a warm rock retaining afternoon sun, a vehicle, or a smouldering patch of ground. Distinguishing them requires reasoning about thermal physics — how heat sources differ in shape, gradient, and persistence — which is precisely the reasoning that RGB-biased models do not have and that DNA optimisation is designed to install.

The consequence of sensory hallucinations here is measured in time, and time is the whole resource. Every false candidate the AI layer surfaces is a diverted flight leg and a delayed search of the area where the person actually is. Every missed detection is a search area declared clear when it was not.

Edge constraints sharpen the deployment question. A drone runs its inference on a power and thermal budget, which favours a decoding-time fix over a bigger model, but also means the extra branches MAD computes have to fit in the loop. On this class of hardware the choice between the two KAIST methods is likely to be made by the power envelope rather than by the benchmark.

Where MAD Sits Among Other Anti-Hallucination Decoding Methods

MAD is not the first training-free decoding intervention, and understanding its neighbourhood clarifies what is new.

Visual Contrastive Decoding established the template in 2024: run the model twice, once on the real image and once on a distorted version, and subtract, so tokens that the model would have produced regardless of what it saw get suppressed. The insight was that hallucinated content is content the model produces from priors, and priors survive input distortion while genuine perception does not. CVPR 2026 alone carried several descendants — perturbation-adaptation methods, history-aware residual decoding, and others — all working the same seam.

What distinguishes MAD is the axis of contrast. Prior methods contrast a clean input against a degraded one within a single modality. MAD contrasts across modality configurations, and crucially derives the mixing weights from the model’s own assessment of which modality the task needs. It is the difference between “ignore what you would have said with your eyes closed” and “decide how much this question is really about what you can see”.

That distinction is what makes it applicable to the cross-modal class of sensory hallucinations rather than the unimodal class. A visual contrastive method cannot help with a model that invents an engine sound, because the visual input is not what is wrong — the visual input is correct and is leaking into a channel where it does not belong.

The methods are not mutually exclusive. Nothing prevents applying a unimodal contrastive method within each branch and MAD’s modality weighting across them, and that combination is an obvious direction for follow-up work.

What KAIST’s Approach Does Not Fix About Sensory Hallucinations

Being clear about the boundaries is more useful than restating the claims.

Neither method adds information the sensor did not capture. If a thermal camera cannot resolve a target, no decoding strategy conjures it. Sensory hallucinations caused by inadequate sensing are a hardware problem, and the KAIST work is entirely about interpretation.

MAD cannot teach the model anything. It reweights what the model already computes, so if the underlying model has no concept of thermal physics, MAD’s self-assessment step has nothing correct to weight toward. This is the structural reason the two methods coexist rather than one superseding the other.

DNA still needs data. Less of it, and it uses errors more efficiently, but the sensor-specific supervised examples have to exist. Organisations with an exotic modality and no annotation budget are not rescued by this.

Neither method addresses calibration. A model can be corrected out of a particular misinterpretation and remain badly calibrated about its own uncertainty, which matters enormously in autonomy where the downstream planner consumes confidence scores. Reducing sensory hallucinations and producing trustworthy confidence are separate achievements.

And neither addresses distribution shift over time. A sensor degrades, a lens fogs, a thermal core drifts, and a model tuned against yesterday’s sensor characteristics starts producing sensory hallucinations against today’s. That is a monitoring problem, and it is the one most perception deployments handle worst.

What Sensory Hallucinations Actually Cost in Production

The reason this research reads as engineering rather than academia is that the cost of sensory hallucinations in deployed systems is mostly hidden in operational overhead rather than in dramatic failures.

The visible cost is the incident: the missed detection, the phantom brake, the false alarm at a checkpoint. Those are rare and heavily investigated. The invisible cost is the mitigation architecture built around a perception system nobody fully trusts — the human-in-the-loop review queue, the conservative thresholds that generate false positives to avoid false negatives, the second sensor added not for coverage but for a sanity check, and the operational rule that says a machine finding must be confirmed before anyone acts.

Every one of those is a tax paid continuously because sensory hallucinations cannot be ruled out. That tax is usually larger over a year than the cost of the incidents it prevents, and it is the number that improves when the underlying misinterpretation is fixed rather than compensated for.

This is also why sensory hallucinations tend to be under-reported inside organisations. A caught error is logged as a successful review, not as a model failure. Teams that have gone through a serious AI agent evaluation exercise usually discover the same thing: the failure rate that matters is the one the mitigations are hiding, and nobody was measuring it.

Deployment Roadmap: Reducing Sensory Hallucinations in Your Stack

If you operate a multimodal perception system, this research is directly actionable. The sequence below assumes you have a deployed model and cannot casually retrain it.

Step 1: Establish Whether You Have Sensory Hallucinations at All

Before adopting anything, measure. Assemble a small evaluation set from your own sensor data with ground truth about the physical scene, and ask sensor-reasoning questions rather than object-naming ones. Most teams discover their sensory hallucinations rate is substantially higher than their object accuracy suggests, because the two measure different things.

Step 2: Separate the Two Failure Classes

Sort observed errors into physical misinterpretation of a single sensor and interference between sensors. The remedies differ completely, and mixing them produces a project that addresses neither. If your errors are mostly the first kind, DNA-style adaptation is the relevant path; if mostly the second, the decoding-time route is.

Step 3: Try the Training-Free Fix First

If cross-modal sensory hallucinations are present, MAD is available under Apache 2.0 with evaluation scripts for AVHBench and CMM. It costs an inference-path change and no retraining, which makes it the cheapest experiment you will run. Measure on your own data, not on the published benchmarks.

Step 4: Budget the Decoding Overhead Honestly

Multiple modality branches per token is more compute per token. Measure the latency on your actual target hardware under your actual load, not on a workstation. On edge platforms this is the constraint that decides whether the method ships.

Step 5: Build a Negative Set Before You Build a Positive One

If you are going the adaptation route, catalogue the wrong answers your model actually produces on your sensors. DNA’s efficiency comes from using those errors as signal, and the catalogue is the asset — collecting it well is a bigger determinant of success than the size of the positive set.

Step 6: Evaluate Sensor Understanding Separately From Task Accuracy

Add sensor-reasoning questions to your regression suite as a permanently separate metric. A model can improve on task accuracy while its sensory hallucinations get worse, and a single blended score hides that completely.

Step 7: Instrument for Drift

Sensors change. Log the input distribution per modality and alert on shift, because a model tuned against one sensor profile will start producing sensory hallucinations against a drifted one long before anything else in the system notices.

Step 8: Write Down What the System Is Not Allowed to Decide Alone

Some decisions should not rest on an uncorroborated single-sensor reading regardless of how good the numbers get. Deciding that in advance, in writing, is a governance act rather than an engineering one, and it is the part most teams postpone until after the first incident. The gaps that enterprise AI agent governance has not yet closed are the same gaps here, with physical consequences attached.

Metrics That Matter for Sensory Hallucinations

If you are evaluating a perception stack seriously, these are the numbers worth tracking, along with the thresholds where the character of the system changes.

Metric	What it measures	Good	Poor
Sensor-reasoning accuracy	Understanding of what the sensor measures	Over 85%	Under 60%
Object naming accuracy	Shallow recognition, for contrast	Over 90%	Under 75%
Reasoning-to-naming gap	How much RGB bias remains	Under 10 points	Over 25 points
Cross-modal false positives	Perceptions imported from another channel	Under 3%	Over 10%
Night-versus-day delta	Degradation when visible light is gone	Under 5 points	Over 20 points
Degraded-visibility delta	Degradation in smoke, fog, or rain	Under 10 points	Over 30 points
Confidence calibration error	Whether stated certainty is earned	Under 0.05	Over 0.15
Decoding latency overhead	Cost of the training-free fix	Under 20%	Over 60%
Human override rate	How often reviewers correct the system	Under 5%	Over 20%
Drift alert lead time	Warning before sensor change bites	Over 2 weeks	Under 2 days

The third row is the one most teams have never computed and the one that tells you the most. A large gap between what a model can name and what it can reason about is the signature of RGB bias, and it predicts sensory hallucinations under conditions your test set does not contain.

Common Mistakes Teams Make With Sensory Hallucinations

The mistakes repeat across organisations with enough consistency to list them.

The first is treating sensory hallucinations as a variant of text hallucinations and reaching for the same remedies. Retrieval grounding, citation checking, and prompt discipline do nothing for a model that misreads a thermal gradient. The cause is different, so the fix is different.

The second is benchmarking on RGB data and assuming the result transfers. A model that scores well on photographs tells you nothing about its thermal or depth behaviour, because the failure is specifically about the physics that photographs do not have.

The third is adding sensors as a substitute for understanding them. More modalities without sound fusion increases the cross-modal interference surface, and a system with four sensors it misreads is less predictable than one with two it reads correctly.

The fourth is measuring only object accuracy. It is the easiest metric to collect and the one that most reliably hides sensory hallucinations, because naming survives the modality change and reasoning does not.

The fifth is trusting a vendor’s benchmark numbers on your conditions. Every reported gain in this field, including KAIST’s, is measured on specific models and specific datasets. Your sensors, your environment, and your question distribution are not those, and the only number that matters is the one you measure yourself.

The sixth is assuming that a fix stays fixed. Sensory hallucinations reappear when sensors drift, when firmware changes the image pipeline, or when the operating environment shifts seasonally. Treating the correction as permanent is how a solved problem becomes an incident eighteen months later.

Where This Work on Sensory Hallucinations Still Falls Short

Even taking the results at face value, the limits are worth stating plainly.

The gains are modest in absolute terms. Improvements between 2% and 8.7% on cross-modal benchmarks are meaningful contributions to a hard problem, not a step change. A system that had a sensory hallucinations problem before applying MAD still has a smaller one afterwards.

The evaluation is narrow. Two audio-visual model families, two benchmarks, one paper. Sensor-side results come from a separate paper with a separate evaluation. Nobody has demonstrated that both methods applied together to a fielded multi-sensor platform deliver the sum of their individual gains.

The preprint history of the sensor paper is untidy. The arXiv record shows a withdrawn version and multiple revisions with a title change from the original Are Vision-Language Models Truly Understanding Multi-vision Sensor? framing, and an administrative note about text overlap with an earlier submission. The IEEE Transactions on Image Processing publication is the version to cite; anyone working from the preprint should check which revision they have.

Benchmark composition details for VS-TDX are not fully described in the public announcement, and the strength of a first-of-its-kind benchmark depends entirely on its composition — which sensors, which scenes, which question types, and how the negatives were constructed. That detail determines whether it measures sensory hallucinations broadly or a particular slice of them.

And the real-world validation is absent, as it usually is at this stage. Nothing in either paper reports a fielded deployment on a vehicle, a drone, or a screening line. The applications in the press release are stated as potential, and the gap between benchmark improvement and operational improvement in perception is historically wide.

What to Watch Next

A few concrete developments would move this from promising to established.

The first is independent replication of the MAD results on models outside the two families tested. The code is public under a permissive licence and the benchmarks are public, so this is a low-friction thing for other groups to do, and it is the fastest way to learn whether the gains are general or model-specific.

The second is the public release of VS-TDX with full composition details. A benchmark that others can run is what turns a paper into a research direction, and sensory hallucinations badly need a shared number the way object hallucination got one from POPE.

The third is a combined evaluation. DNA fixing sensor understanding and MAD fixing cross-modal interference should compose, and demonstrating that on a single multi-sensor system would be a stronger result than either paper alone.

The fourth is any deployment report. A published account of these methods reducing sensory hallucinations on a real thermal drone, screening line, or vehicle stack — with before-and-after operational numbers rather than benchmark deltas — would be worth more than another set of percentage points.

The fifth is whether the defence funders behind the work publish anything. Defence and security applications typically get results long before the literature does, and a public statement from that direction would say a great deal about whether these methods hold up outside a benchmark.

Frequently Asked Questions

What are sensory hallucinations in AI?

Sensory hallucinations are perception errors in multimodal models that originate from the sensing side rather than from language priors. They take two forms: misreading the physical meaning of a sensor — treating a bright region in a thermal image as a light reflection rather than a heat source — and importing a perception from one modality into another, such as reporting a sound because a related object is visible on screen. They differ from text hallucinations in that there is no external source against which the error can be checked.

How does KAIST’s method reduce sensory hallucinations at night?

Through the DNA optimisation method. Night-time perception depends on thermal cameras, which read emitted heat instead of reflected visible light, and models trained on ordinary photographs misinterpret thermal brightness. DNA trains the model to separate physically correct interpretations from plausible but wrong ones, using the model’s own characteristic errors as the training signal, so it stops applying visible-light assumptions to a thermal frame.

Why do sensory hallucinations get worse in smoke?

Because smoke corrupts several modalities unevenly at once. It blocks visible light entirely, degrades lidar as particulates scatter returns, and partially attenuates thermal. A model fusing those channels has to arbitrate between disagreeing inputs, and one that has learned spurious cross-modal correlations resolves the disagreement by inventing consistency. That is the cross-modal class of sensory hallucinations, and it is exactly what MAD’s modality weighting is designed to suppress.

Does MAD require retraining the model?

No. MAD is a training-free decoding method. It changes how the model turns its internal representations into output tokens by weighting modality-specific decoding branches according to the model’s own assessment of which modality the task needs. The weights are untouched, so it can be applied to a deployed model as an inference-path change and rolled back the same way.

How much did MAD improve the benchmarks?

On VideoLLaMA2-AV it improved results by 7.8% on the CMM benchmark and 2.0% on AVHBench. On Qwen2.5-Omni the figures were 8.7% and 4.7% respectively. These are gains on specific cross-modal hallucination benchmarks with specific models, and they should be treated as evidence that the approach works rather than as a figure that will reproduce on your own data.

Can I use these methods today?

MAD is available now — the implementation is public on GitHub under Apache 2.0 with evaluation scripts for AVHBench and CMM, supporting Qwen2.5-Omni at 3B and 7B and VideoLLaMA2-AV. The DNA work is published in IEEE Transactions on Image Processing with a preprint on arXiv, and applying it requires your own sensor-specific data and a fine-tuning pass, so it is a project rather than a download.

Do these methods fix sensory hallucinations in medical imaging?

Not directly. Medical modalities have their own physics — CT radiodensity, sequence-dependent MRI intensity, ultrasound artefacts — and neither paper evaluates on clinical data. The approach is applicable in principle, and the announcement names medical image analysis as a target application, but treating a general method as validated for clinical use because the physics rhymes would be a serious error. Clinical deployment needs its own evaluation against its own indications.

What is the difference between sensory hallucinations and sensor failure?

Sensor failure means the hardware did not capture the information. Sensory hallucinations mean the hardware captured it correctly and the model interpreted it wrongly. The distinction matters because the remedies are unrelated: a failed sensor needs replacement or redundancy, while a misinterpreted signal needs a model that understands the modality. Neither KAIST method helps with the first, and no amount of better hardware fixes the second.

How do I know if my system has a sensory hallucinations problem?

Compare object-naming accuracy with sensor-reasoning accuracy on your own data. Ask not just what is in the frame but why a region appears as it does and what that implies physically. A large gap between the two — naming well while reasoning poorly — is the signature of RGB bias, and it predicts sensory hallucinations under conditions your test set does not cover, which are usually the conditions that made you deploy a special sensor in the first place.

Final Verdict

The KAIST work is a well-scoped contribution to a problem the field has been slow to name properly. Sensory hallucinations have been treated as an implementation detail of multimodal deployment rather than a distinct failure class with its own causes, and separating them into physical misinterpretation and cross-modal interference — with a different remedy for each — is the most useful thing these two papers do.

The engineering judgement behind them is the part worth copying. Both methods were designed around the constraint that organisations cannot retrain large multimodal models on demand. One needs a small amount of data, one needs none, and neither requires touching the architecture. That is a deliberate choice to be useful to people with deployed systems rather than impressive to people with training clusters, and it is why this research is likely to be adopted faster than results with bigger numbers attached.

The claims are appropriately bounded. Single-digit improvements on cross-modal benchmarks are honest, and the announcement’s real contribution is the framing — that sensory hallucinations at night and in smoke are interpretation failures rather than sensing failures, and that interpretation can be corrected far more cheaply than the field has assumed.

What the work does not do is finish the job. The gains are modest, the evaluation is narrow, no fielded deployment has been reported, and the benchmark that underpins the sensor half is new enough that the community has not yet stress-tested it. Anyone reading this as a solution to sensory hallucinations in safety-critical perception has read it wrong.

The practical takeaway is smaller and more useful than a headline. If you run a multimodal perception system, measure your sensor-reasoning accuracy separately from your object accuracy, and if the gap is large, you have sensory hallucinations that your current metrics are hiding. There is now a training-free method you can try in an afternoon and a data-efficient one you can plan a quarter around. Both are better options than the answer the field has offered until now, which was to collect more data and hope.