UniLM

What is UniLM?

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention heads for different types of language modeling tasks. The UniLM achieves state-of-the-art performance on various natural language understanding and generation benchmarks.