Rahvusarhiiv/et_summariser

Estonian summariser

Rahvusarhiiv/et_summariser is an experimental Estonian summarization model fine-tuned from facebook/mbart-large-50-many-to-many-mmt.

This is a research release. It is not claimed to be state of the art; the model should be evaluated for the target domain before production use.

Intended Use

• Summarization of Estonian news, legal/administrative, encyclopedic, or academic-style text.
• Batch/offline summarization, evaluation, and retrieval-assisted workflows where outputs can be checked.
• Not intended as a sole source of factual truth, legal advice, medical advice, or archival description without human review.

How to Use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "Rahvusarhiiv/et_summariser"
text = "Riigikogu arutas täna riigieelarve muudatusi. Valitsuse esindajad rõhutasid vajadust hoida kulud kontrolli all, opositsioon aga kritiseeris kärbete mõju haridusele ja kohalikele omavalitsustele."

tokenizer = AutoTokenizer.from_pretrained(repo_id, src_lang="et_EE")
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=768)
summary_ids = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["et_EE"],
    max_new_tokens=512, num_beams=4, no_repeat_ngram_size=3,
)
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True)[0])

Recommended Generation Settings

• max_new_tokens: 512
• num_beams: 4
• no_repeat_ngram_size: 3

Training Data

Fine-tuned from mBART-50 on cleaned Estonian summarization data, including ERR-style news, LongSum-style long-form examples, and Estonian EurLex/legal material. Grounding filters, metadata dropout, and evidence selection were used during preprocessing.

• Estonian news and long-form summarization corpora prepared locally.
• Estonian EurLex/legal examples after boilerplate filtering.
• Held-out Wiki4 and GDELT-style news evaluations were not used for training.

Known public Hub datasets used or partially used where applicable:

• No public Hub dataset metadata is declared for local-only collected datasets.

Training metadata:

• Base/initial model recorded by trainer: facebook/mbart-large-50-many-to-many-mmt
• Train rows: 23996
• Validation rows: 1000
• Max source tokens: 768
• Max target tokens: 512
• Seed: 20260611

Evaluation

benchmark	n	ROUGE-L	ROUGE-1	ROUGE-2	ref coverage	src precision	judge factuality	judge coverage	judge overall	words	repeat 4g	legalese	max hit
Wiki4 strict v2 LLM judge	100						3.01	3.33	2.73
Wiki4 broad LLM judge	200						3.02	2.42	2.33
Wiki4 final automatic	100	0.294	0.351	0.201	0.387	0.751				62.4	0.0000	0.0000	0.0000
GDELT final automatic	100	0.169	0.212	0.075	0.257	0.805				47.5	0.0000	0.0000	0.0000

The automatic metrics include ROUGE, coverage proxies, repetition and artifact checks. The judge metrics use a local Ollama LLM judge on sampled examples and should be treated as comparative, not absolute.

Limitations and Failure Modes

• The model can hallucinate or over-generalize details, especially when the input is long or under-specified.
• Named entities, numbers, dates, and legal references should be verified against the source.
• Some evaluation datasets reward lead-like summaries; performance may differ on non-news or conversational text.
• The training data mixes sources with different writing styles and upstream terms; users are responsible for checking suitability for their use case.
• Qualitative review still found occasional unsupported details, so factual verification is recommended for publication workflows.

Bias, Risks, and Ethical Considerations

The model reflects the distributions and editorial choices of its source corpora. It may reproduce biases present in news, legal, academic, or synthetic summaries. Human review is recommended for public-facing summaries and high-impact decisions.

CO2 Emissions

Training emissions were not instrumented with CodeCarbon or an equivalent tracker. No numeric CO2 estimate is reported. Training was local fine-tuning of an mBART-50 checkpoint rather than pretraining from scratch.

Model Tree

• Base model: facebook/mbart-large-50-many-to-many-mmt
• Relationship: fine-tuned sequence-to-sequence summarization checkpoint.

License

The model card uses license: other because the training mix contains multiple upstream data sources with different terms. Treat this as a research-use release and verify upstream data/license constraints before redistribution or commercial use.