Add an LLM fact-check pass over the generated briefing

After summarisation the report is verified against the collected
payload (numbers, directions, unsupported claims) by a second LLM call
with a Verification JSON schema. Major issues trigger one revision and
a re-check; minor-only verdicts are normalized to ok. Failures never
block delivery. The verdict is stored in the result JSON and shown on
the dashboard (badge + 검증 메모). Disable with US2KR_VERIFY_ENABLED.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jsahn
2026-09-01 10:37:39 +09:00
co-authored by Claude Fable 5
parent 449b6d8ca3
commit ed48f4c96b
13 changed files with 499 additions and 47 deletions
+20
View File
@@ -91,6 +91,25 @@ class Report(BaseModel):
data_gaps: list[str] = Field(default_factory=list, description="수집 실패로 반영하지 못한 항목")
class VerificationIssue(BaseModel):
claim: str = Field(description="문제가 된 브리핑 속 서술 (원문 그대로 인용)")
problem: str = Field(description="제공된 데이터와 어떻게 다른지 한 문장")
severity: Literal["minor", "major"] = Field(
default="minor", description="major: 수치·방향이 데이터와 모순 / minor: 근거 불충분·표현 문제"
)
class Verification(BaseModel):
"""LLM fact-check verdict for a generated Report, judged only against the payload."""
ok: bool = Field(description="major 이슈가 하나도 없으면 true")
issues: list[VerificationIssue] = Field(default_factory=list, description="발견된 문제 목록 (없으면 빈 배열)")
@property
def has_major(self) -> bool:
return any(i.severity == "major" for i in self.issues)
class BriefingResult(BaseModel):
payload: MarketPayload
report: Report | None
@@ -98,3 +117,4 @@ class BriefingResult(BaseModel):
model_used: str | None = None
degraded: bool = False # True when LLM failed and only the numeric table was produced
error: str | None = None
verification: Verification | None = None # None: not run (disabled / degraded / verifier failed)