T-BDLD Corpus Statistics

Turkish Context-Sensitive Lemmatization Corpus · 21 contextual domains · 3 textual formations · 9,404,213 surface forms · 6,003,511 headwords · 130.29 million sentences

⟳ This page is updated over time; counts and findings are refreshed as the corpus grows.

About This Page — On-Page Guide

Purpose

This page shows how strongly each word in the Turkish Context-Sensitive Lemmatization Corpus (T-BDLD) is associated with a contextual domain such as science, art, technology, law, or history. Novel, short-story, and poetry profiles are inferred separately from sentence content as textual formations.

Domains and Textual Formations

The established 21-domain names remain stable so that results stay comparable over time, but their profiles and counts are relearned from the current corpus. A domain name is used only as an initial retrieval query; the profile itself comes from the full vocabulary and contexts of the corpus documents retrieved by that query. Fixed seed-word lists are not used for classification or counting. Novel, short story, and poetry are estimated on a separate sentence-inferred textual-formation axis.

Method — Overview

  • Balanced deterministic sampling: sentences are selected across the complete corpus sequence. Per-document and per-source-group caps prevent a single long work or prolific source from dominating training.
  • Content representation: cleaned lemma unigrams and bigrams are weighted with sublinear TF-IDF. Title and publication metadata form a separate view with one-quarter of the content weight.
  • Semantic reduction: Truncated SVD projects the sparse content and metadata matrix into a normalized 144-dimensional-or-smaller semantic space.
  • Corpus-derived domain profiles: a domain name starts retrieval, but the centroid and distinguishing vocabulary are learned from the retrieved corpus documents. Precompiled seed-word inventories do not enter the model.
  • Dynamic clustering: MiniBatch K-Means is run for several candidate cluster counts. Selection combines cosine silhouette, Adjusted Rand Index stability across independent initializations, and penalties for oversized or tiny clusters.
  • Assignment and uncertainty: sentence-to-profile proximity, independent cluster evidence, score margin, and confidence gates are combined. Ambiguous sentences remain unassigned rather than being forced into a category.
  • Textual formations: novel, short-story, and poetry evidence is estimated separately from sentence content; it is not copied from source-file labels.
  • Quality gate: canonical lemma corrections are applied during scanning, while confirmed OCR and segmentation noise is excluded from vocabulary rankings.

Metrics

  • Dominance (0–1): the proportion of a word's sentence occurrences assigned to the domain or genre in question.
  • Coverage: the estimated proportion of corpus sentences whose domain evidence passes the assignment confidence gate.
  • Silhouette: cosine-space separation of clusters; higher values indicate clearer boundaries.
  • Stability (ARI): Adjusted Rand Index agreement between cluster assignments produced by two independent initializations. It is evaluated together with silhouette.
  • Domain entropy: the diversity of a word's distribution across contextual domains. A value of 0 indicates concentration in one category; the upper bound changes with the current number of categories.
  • Confidence level: high medium low, determined by score thresholds.
  • Domain status: specific, dominant (one category is clearly ahead), mixed (strong signals occur in several categories), or general (no category is distinctive).
  • Representation score: combines sentence support, semantic confidence, source-group diversity, and distinctive-vocabulary evidence on a 0–100 scale.
  • Balance score: measures how close a category's share is to the ideal share among represented categories. The overall corpus score uses normalized Shannon evenness.
  • Concentration and effective domain count: HHI is calculated from normalized domain shares; its inverse reports how many equally weighted domains would produce the observed concentration.

What Each Tab Does

  • Overview and Guide: live corpus counters, model scores, update time, and this methodology guide.
  • Domain Statistics: alphabetically ordered domain cards and each domain's representation, confidence, balance, and distinguishing vocabulary evidence.
  • Word Search: current domain distribution, dominance, entropy, confidence, and textual-formation evidence for corpus terms.
  • Category Intersections: sentence-level coactivation estimates between domains and textual formations.
  • Findings and Interpretation: a versioned interpretation package generated from coverage, balance, concentration, cluster quality, representation, and lexical evidence.
  • Sentence Similarity: corpus diversity, exact-duplicate signatures, nearest-neighbor TF-IDF cosine similarity, sentence lengths, TTR, and lexical/POS perplexity.

Sentence-Similarity Algorithms

  • Distributed sample: sentences are selected deterministically across the full live corpus sequence, rather than from the beginning of the corpus.
  • TF-IDF cosine: each sample sentence is compared with its nearest neighbor in sparse lexical space. It measures lexical-geometric similarity, not human-like semantic equivalence.
  • Exact duplicates: canonical sentence hashes compare total sentences with unique signatures.
  • Length and TTR: a separate distributed sample measures sentence-length percentiles and lemma type-token ratio.
  • Perplexity: a smoothed lemma bigram model measures lexical predictability; a separately trained POS bigram model measures syntactic-sequence predictability.

Limitations

  • Domain labels are not ontological dictionary classes. They describe contextual usage in the corpus. For example, mektup may occur often in legal contexts; this does not make it inherently a legal term.
  • The proper-noun filter is intentionally strict, so it may exclude some common words.
  • Domain and textual-formation totals are estimates from a deterministic, document-balanced sentence sample.

Abbreviations and Terms Used on This Page

T-BDLD — Turkish Context-Sensitive Lemmatization Corpus, the Turkish corpus of approximately 130.29 million sentences used as the data source for this page.
GTS — Advanced Turkish Dictionary, the site's principal dictionary reference database.
TF-IDF + SVD — Term Frequency–Inverse Document Frequency weights distinguishing words and phrases; Truncated Singular Value Decomposition then projects the sparse matrix into a compact semantic space used for profile and cluster distances.
TTR — Type-Token Ratio, the number of unique word forms divided by the total number of word tokens. Because it falls naturally as sample size grows, it is interpreted only against runs with the same sampling design.
MiniBatch K-Means — a scalable clustering algorithm that updates cluster centers with small deterministic batches; it exposes recurring corpus structures without assigning a prewritten topic list to each sentence.
POS · Perplexity — part of speech is the grammatical word class; perplexity is the model's sequence-surprisal score. It describes predictability under the current model, not grammatical correctness.
Silhouette · ARI · HHI — Silhouette measures cluster separation, Adjusted Rand Index measures agreement across independent cluster runs, and Herfindahl–Hirschman Index measures concentration in normalized domain shares.
Loading...
shared sentence count ≥

Proximity among contextual domains and textual formations is calculated from the first- and second-category distances in sentence-semantic space.

Category Intersection Heat Map

Largest Intersections

Corpus Sentence-Similarity Statistics

Measures sentence diversity and duplicate/near-duplicate rates in T-BDLD using a geometric TF-IDF and cosine-similarity method. It does not use BERT or semantic embeddings.

How is it calculated? Deterministic samples are distributed across the full live corpus sequence rather than being taken from its beginning. The calculation combines nearest-neighbour TF-IDF cosine on 10,000 sentences, length and TTR measurements on 30,000 sentences, lemma/POS bigram models trained on 150,000 sentences and tested on 3,000 independent sentences, and exact hash-signature counts. Runtime depends on corpus and server load. The versioned snapshot and its lag behind the live corpus are shown together; only an administrator can start a new calculation.
Loading...
Interpretation guide:
  • Mean, median, p90, and p99 cosine values describe different parts of the proximity distribution. A low highly-similar rate indicates that near-duplicate neighbourhoods are limited in the sample.
  • A high highly-similar rate (>5%) may indicate templated content, repeated web teasers, or near-duplicate sentences.
  • TTR is sample-size sensitive. It is useful for comparing calculations made with the same sample size and method, but is not an absolute richness score.
  • The exact-duplicate rate compares all sentence rows with unique canonical hash signatures. It captures exact repetition, while cosine captures lexical near-similarity.
  • Lexical perplexity estimates how unexpected the lemma sequence is under the current lemma-bigram model. High outliers may reflect rare language, names, foreign fragments, or OCR noise.
  • Syntactic perplexity estimates how unexpected the POS sequence is under the current POS-bigram model. It measures model surprisal, not grammaticality.
  • Lexical and syntactic perplexities have different vocabularies and scales; compare each metric with its own distribution and earlier runs of the same model, not directly with one another.

What Does This Corpus Tell Us?

This tab provides a detailed reading and interpretation of the measurements on this page. The figures are concrete, but their lexicographic value depends on recognizing both what they reveal and what they do not. The sections below examine coverage, domain balance and concentration, cluster separation and stability, domain representation strength, sentence-inferred textual formations, cross-domain proximity, lexical evidence, and implications for GTS.

⟳ This page is updated over time: both the figures and the interpretation below are refreshed as the corpus grows or the content model is recalibrated. The most recent analysis date appears in the Overview tab.

Preparing findings and interpretation...