Back to the research
Context Engineeringlog analysisevaluation

Should an AI Read Raw Logs or Pre-Filtered Signals?

A cloud fault-detection paper suggests a layered way to feed operational context to an LLM, and a test you can run yourself.

Should an AI read raw logs or pre-filtered signals?

Probably neither extreme. One 2025 paper pairs conventional detectors with an LLM that interprets logs, so the model sees flagged material rather than everything. Its abstract reports gains, but it does not show that the context design caused them. Treat the layering as a test, not a rule.

At a glance

  • Finding: Ji and Luo (2025) propose a multi-level system for cloud fault detection and automatic recovery.
  • Why it matters: it shows one way to decide what an LLM reads: let cheaper detectors narrow the evidence first.
  • Limit: only the abstract was reviewed, and it gives no numbers or breakdown of which component helped.

What the paper describes

The authors combine existing machine learning fault detection with an LLM. The LLM parses system logs, error reports and real-time data streams, using what the abstract calls semantic context.

The architecture has layers. Supervised learning classifies known fault types. Unsupervised learning flags anomalies. The system can then predict failures and trigger self-healing steps.

The abstract says the method beats traditional fault detection systems on detection accuracy, downtime reduction and recovery speed. It gives no figures, so the size of the gain is unknown.

Why this is a context question

A busy cloud system produces far more log lines than any model can usefully read. Someone must choose which lines, reports and data points reach the LLM for a given decision.

In this design, the detectors appear to act as the selector. That is my reading of the abstract, not a stated finding. If correct, the LLM interprets a narrowed set of evidence instead of searching a flood of it.

Illustrative example: a database error spike is flagged by an anomaly detector. The LLM receives those log lines, the related error report and recent metrics, then explains the likely cause. It does not scan a day of unrelated output.

What the evidence does not show

The abstract compares the full system with traditional detectors. It does not compare an LLM reading raw logs against an LLM reading filtered logs. So it cannot say that filtering is the reason for the gains.

It also does not say how often the filter hides the clue the LLM needed. A detector that misses an unusual failure removes that failure from the model's view.

Another recent paper, on robot evaluation, offers a useful habit rather than a result for logs. Xing et al. (2026) checked whether a simulated success rate matched real-world success, across four manipulation tasks. Their reward supervision raised that agreement from 0.602 to 0.885, measured as a correlation. The setting is robotics and does not transfer to logs. The habit does: check your proxy against real outcomes.

What to test

  1. Collect past incidents with known causes.
  2. Run the same incidents through three setups: raw logs, detector-filtered logs, and filtered logs plus a small random sample of unflagged lines.
  3. Measure correct root cause, missed incidents, false alarms and time to a usable diagnosis.
  4. Read the misses by hand. They show what your filter hides.

If filtering helps without more misses, keep it. If misses rise, widen what the model sees before changing the model.

← Explore more research