How PinDiff works
How PinDiff monitors
and records model changes
The method behind the product: how repositories are observed, what is captured, how observations are compared and classified, how your dependency context is added — and what PinDiff cannot tell you.
The method, step by step
-
Monitor supported repositories
PinDiff tracks public Hugging Face model repositories, each written as organisation/name. One global crawler observes every tracked repository once a day; adding a model dependency to a project does not start a separate crawl.
-
Capture observable state
Each observation records the head revision, the declared licence, the gated and private flags, and the file manifest: every file’s name, size and hash. README.md, config.json, generation_config.json and the licence file are stored as text. Weight files are never downloaded.
-
Resolve revisions where possible
When you record a branch, tag or abbreviated commit, PinDiff asks Hugging Face which commit it points to. If Hugging Face cannot answer, commits PinDiff has already recorded for the repository are used, and the failure is shown.
-
Compare observations
Each observation is compared with the previous one. An identical state extends the previous snapshot. A different state is stored as a new snapshot and compared file by file, value by value and section by section.
-
Classify model-relevant changes
Differences are grouped into change types — weights, licence, remote code, access, model card and repository status — and each change is given a severity.
-
Add dependency and revision context
For each dependency on that model in your projects, the change is checked against the revision recorded for it — pinned, floating or unknown — to decide your exposure.
-
Retain dated evidence
Snapshots and stored text are kept with the time each state was first and last observed, so any two stored states can be compared later.
-
Surface relevant changes
Critical, high and medium changes appear in your workspace, on model pages, in the public change feed and in the GitHub job summary if you use the PinDiff Action.
Technical signals
What PinDiff reads for each monitored signal.
| Signal | How it is observed |
|---|---|
| Model weights | File names, sizes and hashes from the repository file manifest: the Git LFS SHA256 where available, otherwise the Git blob ID. Weight files are never downloaded. |
| Licence | The licence tag declared in the model card metadata, plus the hash and text of a top-level LICENSE or COPYING file. Licence terms are not interpreted. |
| Remote code / configuration | config.json and generation_config.json are stored as text. A change to trust_remote_code or auto_map is classified as a remote-code change. |
| Access / gating | The gated and private flags reported by the Hugging Face API at each observation. |
| Model card | README.md is stored as text and compared section by section. Edits to sections such as intended use, limitations, training data or licence are material; other edits are recorded at low severity. |
| Repository status | Recorded when a tracked repository is no longer returned by the Hugging Face API at its address. |
Change severity
Change severity describes the significance of an observed upstream change.
- Critical e.g. licence change, weight replacement, remote-code switch, repository removed
- High e.g. gating or visibility changes
- Medium e.g. edits to material model-card sections
- Low e.g. cosmetic card edits, other file changes
Severity reflects the significance of the observed repository change, not a safety or compliance verdict.
Limitations
- PinDiff records what was observable at collection time. A change made and reverted between two daily observations may not be seen.
- Historical data starts when PinDiff began monitoring a repository.
- Source metadata can be incomplete. When evidence is missing, PinDiff says so instead of inferring a change.
- PinDiff observes the default branch. Other branches and tags are resolved when you record or refresh them, not observed daily.
- A recorded revision is not proof of what is deployed at runtime.
- Only public Hugging Face model repositories are covered. Private repositories and other registries are not.
- PinDiff does not determine whether a model is safe, and does not provide legal or compliance approval.