Methodology
What we do to the data before showing it, and — more usefully — what we decline to do.
Search
Khmer is written without spaces between words, which is why plain full-text search over it fails. Queries and records are tokenised the same way and indexed with weighted full-text search, with trigram similarity as a fallback.
Sources and how Khmer text is normalised
Each record is stored as its publisher supplied it; the wording of a stored title is never rewritten (the only change is the whitespace clean-up described in the next section). For search only, a separate canonical form is built: Unicode normalisation (NFC), removal of zero-width characters, folding of the two spellings of the subscript consonant (coeng DA is read as coeng TA), and Khmer digits read as ASCII digits. As a result a search for ស្តី finds ស្ដី, and a search for ២០២៣ finds 2023, and the other way round.
Whitespace in titles
Titles and other short text fields from the Open Development Cambodia harvest are cleaned once, on import: Unicode normalisation (NFC), control characters replaced by a space, every run of whitespace (spaces, tabs, line breaks) collapsed to a single space, and leading and trailing whitespace removed. Zero-width characters are not removed from the stored title, so a title is shown as the publisher wrote it apart from its spacing; they are removed only in the separate form built for search. Longer descriptions keep their line breaks, but runs of spaces and blank lines are collapsed.
Text extracted from PDFs (OCR)
When text extracted from a PDF or a scanned page is added, it is used only to help search find a record. It is never shown on a page. Numbers and dates on this site come only from the metadata the publisher supplies, never from extracted text.
Machine-translated titles
When a machine-translated title is added, it is labelled as machine-translated wherever it appears. It is never presented as the publisher's own wording, and a title the publisher wrote is never replaced by a machine translation.
Duplicates
Portals often harvest each other, so the same dataset can appear more than once. We match on the harvest identifier first, then on normalised resource URLs. Title-only matches go to a review queue and are never merged automatically, because annual editions of the same dataset share a title year after year. Headline counts exclude anything marked as a duplicate.
Possible duplicate legal instruments
Records that share the same number and the same date are flagged as a possible duplicate and put on a list for a human reviewer. They are never merged automatically. The reviewer's decision is recorded against the flagged group.
Matching things to provinces
Coordinates are assigned to a province by testing whether the point falls inside the province boundary, not by finding the nearest provincial centroid. Nearest-centroid is wrong along borders, and the failure is silent: the per-province counts look entirely reasonable while being wrong.
Withheld records
A record can be withheld, for example because it names a private individual. Withholding is stored on the record itself. A withheld record disappears from its page, from search results, from the sitemaps and from downloads.
Where the counts come from
Counts shown on this site are read from a snapshot of the database taken by the data harvest, not typed into a page. The snapshot and the time it was computed are shown on the status page. Datasets and library documents are counted separately. "Data files" counts only the files attached to datasets. "Upstream sources" counts the sources listed on the sources page, not the number of scheduled jobs. A withheld record or file is not counted.
OpenStreetMap data
OpenStreetMap is published under ODbL, whose share-alike terms can propagate to a database it is combined with. It is therefore held in separate tables and joined to everything else only at display time, with the result never written back.
What is missing
This section lists the sources that are known to be missing, stale or unreachable. It is filled in as the sources are added.
See also: sources · system status · licensing