"I love Sebastian Ruder"

2022-05-04 09:09:29

.responsive-header-img { background-image: url(content/images/size/w2000/2017/05/imageedit_8_8459453433.jpg); } @media(max-width: 1000px) { .responsive-header-img { background-image: url(content/images/size/w1000/2017/05/imageedit_8_8459453433.jpg); background-image: -webkit-image-set(url(content/images/size/w1000/2017/05/imageedit_8_8459453433.jpg) 1x, url(content/images/size/w2000/2017/05/imageedit_8_8459453433.jpg) 2x); background-image: image-set(url(content/images/size/w1000/2017/05/imageedit_8_8459453433.jpg) 1x, url(content/images/size/w2000/2017/05/imageedit_8_8459453433.jpg) 2x); } } @media(max-width: 600px) { .responsive-header-img { background-image: url(content/images/size/w600/2017/05/imageedit_8_8459453433.jpg); background-image: -webkit-image-set(url(content/images/size/w600/2017/05/imageedit_8_8459453433.jpg) 1x, url(content/images/size/w1000/2017/05/imageedit_8_8459453433.jpg) 2x); background-image: image-set(url(content/images/size/w600/2017/05/imageedit_8_8459453433.jpg) 1x, url(content/images/size/w1000/2017/05/imageedit_8_8459453433.jpg) 2x); } } Sebastian Ruder About Tags Papers Talks News FAQ Newsletter NLP Progress Media Contact Sebastian Ruder I'm a research scientist at Google. I blog about natural language processing and machine learning. ML and NLP Research Highlights of 2021 This post summarizes progress across multiple impactful areas in ML and NLP in 2021. Sebastian Ruder Sebastian Ruder 24 Jan 2022 • 31 min read Multi-domain Multilingual Question Answering This post expands on the EMNLP 2021 tutorial on Multi-domain Multilingual Question Answering and highlights key insights and takeaways. Sebastian Ruder Sebastian Ruder 6 Dec 2021 • 21 min read natural language processing Challenges and Opportunities in NLP Benchmarking Over the last years, models in NLP have become much more powerful, driven by advances in transfer learning. A consequence of this drastic increase in performance is that existing benchmarks Sebastian Ruder Sebastian Ruder 23 Aug 2021 • 16 min read language models Recent Advances in Language Model Fine-tuning This article provides an overview of recent methods to fine-tune large pre-trained language models. Sebastian Ruder Sebastian Ruder 24 Feb 2021 • 13 min read transfer learning ML and NLP Research Highlights of 2020 This post summarizes progress in 10 exciting and impactful directions in ML and NLP in 2020. Sebastian Ruder Sebastian Ruder 19 Jan 2021 • 15 min read cross-lingual Why You Should Do NLP Beyond English 7000+ languages are spoken around the world but NLP research has mostly focused on English. This post outlines why you should work on languages other than English. Sebastian Ruder Sebastian Ruder 1 Aug 2020 • 7 min read advice 10 Tips for Research and a PhD This post outlines 10 things that I did during my PhD and found particularly helpful in the long run. Sebastian Ruder Sebastian Ruder 22 May 2020 • 10 min read natural language processing 10 ML & NLP Research Highlights of 2019 This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. Sebastian Ruder Sebastian Ruder 6 Jan 2020 • 12 min read cross-lingual Unsupervised Cross-lingual Representation Learning This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. It highlights key insights and takeaways and provides updates based on recent work, particularly unsupervised deep multilingual models. Sebastian Ruder Sebastian Ruder 26 Oct 2019 • 20 min read transfer learning The State of Transfer Learning in NLP This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. It highlights key insights and takeaways and provides updates based on recent work. Sebastian Ruder Sebastian Ruder 18 Aug 2019 • 15 min read events EurNLP The first European NLP Summit (EurNLP) will take place in London on October 11, 2019. It is an opportunity to foster discussion and collaboration between researchers in and around Europe. Sebastian Ruder Sebastian Ruder 4 Jul 2019 • 2 min read events NAACL 2019 Highlights This post discusses highlights of NAACL 2019. It covers transfer learning, common sense reasoning, natural language generation, bias, non-English languages, and diversity and inclusion. Sebastian Ruder Sebastian Ruder 9 Jun 2019 • 8 min read transfer learning Neural Transfer Learning for Natural Language Processing (PhD thesis) This post discusses my PhD thesis Neural Transfer Learning for Natural Language Processing and some new material presented in it. Sebastian Ruder Sebastian Ruder 23 Mar 2019 • 1 min read events AAAI 2019 Highlights: Dialogue, reproducibility, and more This post discusses highlights of AAAI 2019. It covers dialogue, reproducibility, question answering, the Oxford style debate, invited talks, and a diverse set of research papers. Sebastian Ruder Sebastian Ruder 7 Feb 2019 • 11 min read natural language processing The 4 Biggest Open Problems in NLP This is the second post based on the Frontiers of NLP session at the Deep Learning Indaba 2018. It discusses 4 major open problems in NLP. Sebastian Ruder Sebastian Ruder 15 Jan 2019 • 10 min read transfer learning 10 Exciting Ideas of 2018 in NLP This post gathers 10 ideas that I found exciting and impactful this year—and that we'll likely see more of in the future. For each idea, it highlights 1-2 papers that execute them well. Sebastian Ruder Sebastian Ruder 19 Dec 2018 • 8 min read events EMNLP 2018 Highlights: Inductive bias, cross-lingual learning, and more This post discusses highlights of EMNLP 2018. It focuses on talks and papers dealing with inductive bias, cross-lingual learning, word embeddings, latent variable models, language models, and datasets. Sebastian Ruder Sebastian Ruder 6 Nov 2018 • 11 min read natural language processing HackerNoon Interview This post is an interview by fast.ai fellow Sanyam Bhutani with me. It covers my background, advice on getting started with NLP, writing technical articles, and more. Sebastian Ruder Sebastian Ruder 2 Oct 2018 • 7 min read language models A Review of the Neural History of Natural Language Processing This post expands on the Frontiers of Natural Language Processing session organized at the Deep Learning Indaba 2018. It discusses major recent advances in NLP focusing on neural network-based methods. Sebastian Ruder Sebastian Ruder 1 Oct 2018 • 29 min read natural language processing ACL 2018 Highlights: Understanding Representations and Evaluation in More Challenging Settings This post discusses highlights of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). It focuses on understanding representations and evaluating in more challenging scenarios. Sebastian Ruder Sebastian Ruder 26 Jul 2018 • 18 min read natural language processing NLP's ImageNet moment has arrived Big changes are underway in the world of NLP. The long reign of word vectors as NLP's core representation technique has seen an exciting new line of challengers emerge. These approaches demonstrated that pretrained language models can achieve state-of-the-art results and herald a watershed moment. Sebastian Ruder Sebastian Ruder 12 Jul 2018 • 16 min read natural language processing Tracking the Progress in Natural Language Processing Research in ML and NLP is moving at a tremendous pace, which is an obstacle for people wanting to enter the field. To make working with new tasks easier, this post introduces a resource that tracks the progress and state-of-the-art across many tasks in NLP. Sebastian Ruder Sebastian Ruder 22 Jun 2018 • 2 min read natural language processing Highlights of NAACL-HLT 2018: Generalization, Test-of-time, and Dialogue Systems This post discusses highlights of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018). It focuses on Generalization, the Test-of-Time awards, and Dialogue Systems. Sebastian Ruder Sebastian Ruder 12 Jun 2018 • 15 min read semi-supervised learning An overview of proxy-label approaches for semi-supervised learning While unsupervised learning is still elusive, researchers have made a lot of progress in semi-supervised learning. This post focuses on a particular promising category of semi-supervised learning methods that assign proxy labels to unlabelled data, which are used as targets for learning. Sebastian Ruder Sebastian Ruder 26 Apr 2018 • 19 min read tensorflow Text Classification with TensorFlow Estimators This post is a tutorial that shows how to use Tensorflow Estimators for text classification. It covers loading data using Datasets, using pre-canned estimators as baselines, word embeddings, and building custom estimators, among others. Sebastian Ruder Sebastian Ruder 16 Apr 2018 • 13 min read transfer learning Requests for Research It can be hard to find compelling topics to work on and know what questions to ask when you are just starting as a researcher. This post aims to provide inspiration and ideas for research directions to junior researchers and those trying to get into research. Sebastian Ruder Sebastian Ruder 4 Mar 2018 • 13 min read optimization Optimization for Deep Learning Highlights in 2017 Different gradient descent optimization algorithms have been proposed in recent years but Adam is still most commonly used. This post discusses the most exciting highlights and most promising recent approaches that may shape the way we will optimize our models in the future. Sebastian Ruder Sebastian Ruder 3 Dec 2017 • 15 min read word embeddings Word embeddings in 2017: Trends and future directions Word embeddings are an integral part of current NLP models, but approaches that supersede the original word2vec have not been proposed. This post focuses on the deficiencies of word embeddings and how recent approaches have tried to resolve them. Sebastian Ruder Sebastian Ruder 21 Oct 2017 • 17 min read multi-task learning Multi-Task Learning Objectives for Natural Language Processing Multi-task learning is becoming increasingly popular in NLP but it is still not understood very well which tasks are useful. As inspiration, this post gives an overview of the most common auxiliary tasks used for multi-task learning for NLP. Sebastian Ruder Sebastian Ruder 24 Sep 2017 • 16 min read natural language processing Highlights of EMNLP 2017: Exciting datasets, return of the clusters, and more This post discusses highlights of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). These include exciting datasets, new cluster-based methods, distant supervision, data selection, character-level models, and many more. Sebastian Ruder Sebastian Ruder 22 Sep 2017 • 10 min read domain adaptation Learning to select data for transfer learning Domain adaptation methods typically seek to identify features that are shared between the domains or learn representations that are general enough to be useful for both domains. This post discusses a complementary approach to domain adaptation that selects data that is useful for training the model. Sebastian Ruder Sebastian Ruder 18 Aug 2017 • 3 min read natural language processing Deep Learning for NLP Best Practices Neural networks are widely used in NLP, but many details such as task or domain-specific considerations are left to the practitioner. This post collects best practices that are relevant for most tasks in NLP. Sebastian Ruder Sebastian Ruder 25 Jul 2017 • 23 min read multi-task learning An Overview of Multi-Task Learning in Deep Neural Networks Multi-task learning is becoming more and more popular. This post gives a general overview of the current state of multi-task learning. In particular, it provides context for current neural network-based methods by discussing the extensive multi-task learning literature. Sebastian Ruder Sebastian Ruder 29 May 2017 • 29 min read transfer learning Transfer Learning - Machine Learning's Next Frontier Deep learning models excel at learning from a large number of labeled examples, but typically do not generalize to conditions not seen during training. This post gives an overview of transfer learning, motivates why it warrants our application, and discusses practical applications and methods. Sebastian Ruder Sebastian Ruder 21 Mar 2017 • 28 min read meta-learning Highlights of NIPS 2016: Adversarial learning, Meta-learning, and more The Conference on Neural Information Processing Systems (NIPS) is one of the top ML conferences. This post discusses highlights of NIPS 2016 including GANs, the nuts and bolts of ML, RNNs, improvements to classic algorithms, RL, Meta-learning, and Yann LeCun's infamous cake. Sebastian Ruder Sebastian Ruder 21 Dec 2016 • 12 min read cross-lingual A survey of cross-lingual word embedding models Monolingual word embeddings are pervasive in NLP. To represent meaning and transfer knowledge across different languages, cross-lingual word embeddings can be used. Such methods learn representations of words in a joint embedding space. Sebastian Ruder Sebastian Ruder 28 Nov 2016 • 41 min read natural language processing Highlights of EMNLP 2016: Dialogue, deep learning, and more This post discusses highlights of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). These include work on reinforcement learning, dialogue, sequence-to-sequence models, semantic parsing, natural language generation, and many more. Sebastian Ruder Sebastian Ruder 14 Nov 2016 • 4 min read word embeddings On word embeddings - Part 3: The secret ingredients of word2vec Word2vec is a pervasive tool for learning word embeddings. Its success, however, is mostly due to particular architecture choices. Transferring these choices to traditional distributional methods makes them competitive with popular word embedding methods. Sebastian Ruder Sebastian Ruder 24 Sep 2016 • 10 min read events LxMLS 2016 Highlights The Lisbon Machine Learning School (LxMLS) is an annual event that brings together researchers and graduate students in ML, NLP, and Computational Linguistics. This post discusses highlights, key insights, and takeaways from the 6th edition of the summer school. Sebastian Ruder Sebastian Ruder 12 Aug 2016 • 14 min read word embeddings On word embeddings - Part 2: Approximating the Softmax The softmax layer is a core part of many current neural network architectures. When the number of output classes is very large, such as in the case of language modelling, computing the softmax becomes very expensive. This post explores approximations to make the computation more efficient. Sebastian Ruder Sebastian Ruder 13 Jun 2016 • 33 min read word embeddings On word embeddings - Part 1 Word embeddings popularized by word2vec are pervasive in current NLP applications. The history of word embeddings, however, goes back a lot further. This post explores the history of word embeddings in the context of language modelling. Sebastian Ruder Sebastian Ruder 11 Apr 2016 • 15 min read optimization An overview of gradient descent optimization algorithms Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimization algorithms such as Momentum, Adagrad, and Adam actually work. Sebastian Ruder Sebastian Ruder 19 Jan 2016 • 28 min read Sebastian Ruder About Tags Papers Talks News FAQ Newsletter NLP Progress Media Contact Sebastian Ruder © 2022 Latest Posts Twitter Ghost