Comparison of Czech Transformers on Text Classification Tasks

Show simple item record
dc.contributor.authorLehečka, Jan
dc.contributor.authorŠvec, Jan
dc.date.accessioned2022-03-21T11:00:18Z
dc.date.available2022-03-21T11:00:18Z
dc.date.issued2021
dc.description.abstract-translatedIn this paper, we present our progress in pre-training monolingual Transformers for Czech and contribute to the research community by releasing our models for public. The need for such models emerged from our effort to employ Transformers in our language-specific tasks, but we found the performance of the published multilingual models to be very limited. Since the multilingual models are usually pre-trained from 100+ languages, most of low-resourced languages (including Czech) are under-represented in these models. At the same time, there is a huge amount of monolingual training data available in web archives like Common Crawl. We have pre-trained and publicly released two monolingual Czech Transformers and compared them with relevant public models, trained (at least partially) for Czech. The paper presents the Transformers pre-training procedure as well as a comparison of pre-trained models on text classification task from various domains.en
dc.format11 s.cs
dc.format.mimetypeapplication/pdf
dc.identifier.citationLEHEČKA, J. ŠVEC, J. Comparison of Czech Transformers on Text Classification Tasks. In Statistical Language and Speech Processing, SLSP 2021. Cham: Springer, 2021. s. 27-37. ISBN: 978-3-030-89578-5 , ISSN: 0302-9743cs
dc.identifier.doi10.1007/978-3-030-89579-2_3
dc.identifier.isbn978-3-030-89578-5
dc.identifier.issn0302-9743
dc.identifier.obd43933815
dc.identifier.uri2-s2.0-85118152009
dc.identifier.urihttp://hdl.handle.net/11025/47192
dc.language.isoenen
dc.project.IDDG18P02OVV016/Vývoj centralizovaného rozhraní pro vytěžování velkých dat z webových archivůcs
dc.publisherSpringeren
dc.relation.ispartofseriesStatistical Language and Speech Processing, SLSP 2021en
dc.rights© Springeren
dc.rights.accessopenAccessen
dc.subject.translatedtext categorization and summarizationen
dc.subject.translatedmonolingual transformersen
dc.subject.translatedsentiment analysisen
dc.subject.translatedmulti-label topic identificationen
dc.titleComparison of Czech Transformers on Text Classification Tasksen
dc.typekonferenční příspěvekcs
dc.typeConferenceObjecten
dc.type.statusPeer-revieweden
dc.type.versionpublishedVersionen

Files

Original bundle
Showing 1 - 1 out of 1 results
No Thumbnail Available
Name:
Lehečka-Švec2021_Chapter_ComparisonOfCzechTransformersO.pdf
Size:
276.39 KB
Format:
Adobe Portable Document Format
Download

Collections

OBD
 Conference Papers (KKY)
 OPEN License Selector