Daniel Biermann

• Biermann, Daniel; Palumbo, Fabrizio; Goodwin, Morten & Granmo, Ole-Christoffer (2024). Streamlining Attention for Text Classification: Sequence Length Reduction with Pooling Attention. In Bramer, Max & Stahl, Frederic (Ed.), Artificial Intelligence XLI: 44th SGAI International Conference on Artificial Intelligence, AI 2024. Springer. ISSN 9783031779145. p. 61–73. doi: 10.1007/978-3-031-77915-2_5. Full text in Research Archive
• Aaby, Pernille; Biermann, Daniel; Yazidi, Anis; Mello, Gustavo Borges Moreno e & Palumbo, Fabrizio (2023). Exploring Multilingual Word Embedding Alignments in BERT Models: A Case Study of English and Norwegian. In Bramer, Max & Stahl, Frederic (Ed.), Artificial Intelligence XL: 43rd SGAI International Conference on Artificial Intelligence, AI 2023. Springer. ISSN 9783031479939. p. 47–58. doi: 10.1007/978-3-031-47994-6_4. Full text in Research Archive Show summary

  Contextual language models, such as transformers, can solve a wide range of language tasks ranging from text classification to question answering and machine translation. Like many deep learning models, the performance heavily depends on the quality and amount of data available for training. This poses a problem for low-resource languages, such as Norwegian, that can not provide the necessary amount of training data. In this article, we investigate the use of multilingual models as a step toward overcoming the data sparsity problem for minority languages. In detail, we study how words are represented by multilingual BERT models across two languages of our interest: English and Norwegian. Our analysis shows that multilingual models similarly encode English-Norwegian word pairs. The multilingual model automatically aligns semantics across languages without supervision. Additionally, our analysis also shows that embedding a word encodes information about the language to which it belongs. We, therefore, believe that in pre-trained multilingual models' knowledge from one language can be transferred to another without direct supervision and help solve the data sparsity problem for minor languages.

• Biermann, Daniel; Goodwin, Morten & Granmo, Ole-Christoffer (2023). Knowledge Infused Representations Through Combination of Expert Knowledge and Original Input. In Zouganeli, Evi; Yazidi, Anis; Mello, Gustavo Borges Moreno e & Lind, Pedro (Ed.), Nordic Artificial Intelligence Research and Development: 4th Symposium of the Norwegian AI Society, NAIS 2022, Oslo, Norway, May 31-June 1, 2022, Revised Selected Papers. Springer Nature. ISSN 9783031170294. p. 3–15. doi: 10.1007/978-3-031-17030-0_1. Full text in Research Archive Show summary

  Sophisticated applications in natural language processing, such as conversational agents, often need to be able to generalize across a range of different tasks to generate natural-feeling language. In this paper, we introduce a model that aims to improve generalizability with regard to different tasks by combining the original input with the output of a task-specific expert. Through a combination mechanism, we create a new representation that has been enriched with the information given by the expert. These enriched representations then serve as input to a downstream model. We test three different combination mechanisms in two combination paradigms and evaluate the performance of the new enriched representation in a simple encoder-decoder model. We show that even very simple combination mechanisms are able to significantly improve performance of the downstream model. This means that the encoded expert information is transported through the new enriched input representation, leading to a beneficial impact on performance within the task domain. This opens the way for exciting future endeavors such as testing performance on different task domains and the combination of multiple experts.

• Biermann, Daniel; Goodwin, Morten & Granmo, Ole-Christoffer (2023). Transfer Learning Through Knowledge-Infused Representations with Contextual Experts. In Maglogiannis, Ilias; Iliadis, L.; MacIntyre, J. & Dominguez, Manuel (Ed.), Artificial Intelligence Applications and Innovations. Springer. ISSN 9783031341069. doi: 10.1007/978-3-031-34107-6_45. Full text in Research Archive
• Lien, Henrik; Biermann, Daniel; Palumbo, Fabrizio & Goodwin, Morten (2022). An Exploration of Semi-supervised Text Classification. In Iliadis, Lazaros; Jayne, Chrisina; Tefas, Anastasios & Pimenidis, Elias (Ed.), Engineering Applications of Neural Networks - 23rd International Conference, EAAAI/EANN 2022. Springer Nature. ISSN 9783031082221. p. 477–488. doi: 10.1007/978-3-031-08223-8_39. Full text in Research Archive Show summary

  Good performance in supervised text classification is usually obtained with the use of large amounts of labeled training data. However, obtaining labeled data is often expensive and time-consuming. To overcome these limitations, researchers have developed Semi-Supervised learning (SSL) algorithms exploiting the use of unlabeled data, which are generally easy and free to access. With SSL, unlabeled and labeled data are combined to outperform Supervised-Learning algorithms. However, setting up SSL neural networks for text classification is cumbersome and frequently based on a trial and error process. We show that the hyperparameter configuration significantly impacts SSL performance, and the learning rate is the most influential parameter. Additionally, increasing model size also improves SSL performance, particularly when less pre-processing data are available. Interestingly, as opposed to feed-forward models, recurrent models generally reach a performance threshold as pre-processing data size increases. This article expands the knowledge on hyperparameters and model size in relation to SSL application in text classification. This work supports the use of SSL work in future NLP projects by optimizing model design and potentially lowering training time, particularly if time-restricted.

View all works in NVA