Img ID logo Img ID

Transparent methodology

Useful evidence starts with honest limits.

Img ID combines visual model observations, OCR, component detection, and file metadata. Each signal has a different meaning. This page explains how detection is tested, what current pilot can support, and what must improve before Img ID publishes a representative accuracy claim.

Current status: internal pilot. Current set contains 13 images. It is useful for regression checks, but too small and narrow to estimate real-world accuracy. Img ID does not market an accuracy percentage from this set.
Run a free scan See example workflows

Evidence boundaries

Observation is not verification.

Model output is a visual hypothesis about pixels. File metadata is parsed from uploaded bytes. Cryptographic provenance and invisible watermarks require dedicated validators. Img ID presents these as separate evidence classes instead of blending them into one claim.

Visual model observations Anatomy, text, lighting, geometry, texture, and directly visible marks. These can support a verdict but cannot prove origin.
Parsed file metadata EXIF fields read from file bytes when present. Metadata can be edited or removed, so it is context rather than proof.
Cryptographic credentials C2PA claims require signature and trust-chain validation. Img ID does not currently claim this verification.
Invisible watermarks Signals such as SynthID require a supported detector. A general vision model cannot verify their presence or absence.

Pilot dataset

13 images, documented as a starting point.

Class Count Recorded coverage
AI-generated 9 Four Midjourney, four Grok, one Gemini
Real photographs 4 Portrait, landmark, and landscape photos

One changed result moves raw hit rate by 7.7 percentage points. Dataset also lacks a holdout split, broad camera coverage, editing history, transformation variants, and independent label review. These limits rule out a representative public accuracy claim.

Evaluation

Same prompt, same images, individual errors first.

Each candidate receives same image and detection prompt. Strong and soft AI or real verdicts are compared with recorded binary label. Uncertain answers abstain. Internal ranking rewards a correct confident direction, gives soft verdicts less weight, and penalizes confident errors more heavily.

Weighted score is a product heuristic, not calibrated probability or scientific metric. Model-reported confidence is not assumed to be calibrated. Reviews focus on false positives, false negatives, and uncertain cases rather than one headline number.

Reporting policy

Claims Img ID will and will not make.

  • Call current set a pilot or regression set, not a representative benchmark.
  • Report sample counts, run date, model identifier, prompt revision, and individual errors.
  • Do not turn a 12-of-13 pilot result into a public accuracy claim.
  • Do not claim model attribution unless a separate attribution evaluation supports it.
  • Do not treat missing metadata as evidence that an image is AI-generated.
  • Keep historical provider results labeled with date because hosted models can change.

Expansion plan

Path to a credible public benchmark.

  1. Collect at least 200 rights-cleared files with source, license, ground truth, edit history, and SHA-256 recorded.
  2. Balance real and AI classes, then lock a holdout set before model or prompt tuning.
  3. Cover current generators, varied cameras, people and places, screenshots, CGI, artwork, receipts, memes, and edited photos.
  4. Test cropping, resizing, filtering, recompression, screenshots, upscaling, and social-media re-encoding.
  5. Use independent review for ambiguous labels and exclude unresolved files from headline reporting.
  6. Publish false-positive, false-negative, abstention, calibration, and category-level results with uncertainty intervals.