Add an LLM fact-check pass over the generated briefing
After summarisation the report is verified against the collected payload (numbers, directions, unsupported claims) by a second LLM call with a Verification JSON schema. Major issues trigger one revision and a re-check; minor-only verdicts are normalized to ok. Failures never block delivery. The verdict is stored in the result JSON and shown on the dashboard (badge + 검증 메모). Disable with US2KR_VERIFY_ENABLED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -91,6 +91,25 @@ class Report(BaseModel):
|
||||
data_gaps: list[str] = Field(default_factory=list, description="수집 실패로 반영하지 못한 항목")
|
||||
|
||||
|
||||
class VerificationIssue(BaseModel):
|
||||
claim: str = Field(description="문제가 된 브리핑 속 서술 (원문 그대로 인용)")
|
||||
problem: str = Field(description="제공된 데이터와 어떻게 다른지 한 문장")
|
||||
severity: Literal["minor", "major"] = Field(
|
||||
default="minor", description="major: 수치·방향이 데이터와 모순 / minor: 근거 불충분·표현 문제"
|
||||
)
|
||||
|
||||
|
||||
class Verification(BaseModel):
|
||||
"""LLM fact-check verdict for a generated Report, judged only against the payload."""
|
||||
|
||||
ok: bool = Field(description="major 이슈가 하나도 없으면 true")
|
||||
issues: list[VerificationIssue] = Field(default_factory=list, description="발견된 문제 목록 (없으면 빈 배열)")
|
||||
|
||||
@property
|
||||
def has_major(self) -> bool:
|
||||
return any(i.severity == "major" for i in self.issues)
|
||||
|
||||
|
||||
class BriefingResult(BaseModel):
|
||||
payload: MarketPayload
|
||||
report: Report | None
|
||||
@@ -98,3 +117,4 @@ class BriefingResult(BaseModel):
|
||||
model_used: str | None = None
|
||||
degraded: bool = False # True when LLM failed and only the numeric table was produced
|
||||
error: str | None = None
|
||||
verification: Verification | None = None # None: not run (disabled / degraded / verifier failed)
|
||||
|
||||
Reference in New Issue
Block a user