Img ID logo Img IDAI Image Forensics

Transparent methodology

Useful evidence starts with honest limits.

Img ID combines visual model observations, OCR, Content Credentials validation when present, file facts, and metadata. Each signal keeps its own source and limits. This page explains how conclusions are formed and what current visual benchmark can support.

Visual detection status: internal pilot. Current set contains 13 images, too few to estimate real-world accuracy. Deterministic file facts and C2PA validation are live, but Img ID does not market an overall accuracy percentage from visual benchmark.
Run a free scan See example workflows

Evidence boundaries

Observation is not verification.

Model output is a visual hypothesis about pixels. File metadata is parsed from uploaded bytes. Cryptographic provenance and invisible watermarks require dedicated validators. Img ID presents each evidence class separately instead of blending unrelated signals into one probability.

Visual model observations Anatomy, text, lighting, geometry, texture, and directly visible marks. These can support a verdict but cannot prove origin.
Parsed file metadata EXIF fields read from file bytes when present. Metadata can be edited or removed, so it is context rather than proof.
Cryptographic credentials Img ID parses original upload with official C2PA SDK and distinguishes trusted, integrity-valid but untrusted, invalid, and absent credentials. Signed claims do not prove depicted scene.
Invisible watermarks Img ID reports signed C2PA SynthID assertions separately and links to Google's official Gemini verification. Google does not document a supported public verification API, so Img ID does not claim automatic detection. Missing SynthID evidence remains neutral.

Pilot dataset

13 images, documented as a starting point.

Class Count Recorded coverage
AI-generated 9 Four Midjourney, four Grok, one Gemini
Real photographs 4 Portrait, landmark, and landscape photos

One changed result moves raw hit rate by 7.7 percentage points. Dataset also lacks a holdout split, broad camera coverage, editing history, transformation variants, and independent label review. These limits rule out a representative public accuracy claim.

Evaluation

Same prompt, same images, individual errors first.

Each candidate receives same image and detection prompt. Strong and soft AI or real verdicts are compared with recorded binary label. Uncertain answers abstain. Internal ranking rewards a correct confident direction, gives soft verdicts less weight, and penalizes confident errors more heavily.

Weighted score is a product heuristic, not calibrated probability or scientific metric. Model-reported confidence is not assumed to be calibrated. Reviews focus on false positives, false negatives, and uncertain cases rather than one headline number.

Reporting policy

Claims Img ID will and will not make.

  • Call current set a pilot or regression set, not a representative benchmark.
  • Report sample counts, run date, model identifier, prompt revision, and individual errors.
  • Do not turn a 12-of-13 pilot result into a public accuracy claim.
  • Do not claim model attribution unless a separate attribution evaluation supports it.
  • Do not treat missing metadata as evidence that an image is AI-generated.
  • Keep historical provider results labeled with date because hosted models can change.

Expansion plan

Path to a credible public benchmark.

  1. Collect at least 200 rights-cleared files with source, license, ground truth, edit history, and SHA-256 recorded.
  2. Balance real and AI classes, then lock a holdout set before model or prompt tuning.
  3. Cover current generators, varied cameras, people and places, screenshots, CGI, artwork, receipts, memes, and edited photos.
  4. Test cropping, resizing, filtering, recompression, screenshots, upscaling, and social-media re-encoding.
  5. Use independent review for ambiguous labels and exclude unresolved files from headline reporting.
  6. Publish false-positive, false-negative, abstention, calibration, and category-level results with uncertainty intervals.