In a recent digital transformation partnership, Google’s DeepMind, Facebook AI, the University of Washington and New York University have launched a platform for measuring the natural language processing (NLP) capabilities of AI.
SuperGLUE, as the platform is called, is an upgrade of the GLUE platform. It was developed due to conversational AI systems having hit a ceiling with their NLP capabilities and needing a new set of challenges in order to improve.
Facebook AI reports that SuperGLUE contains the following elements of machine learning:
– Self-supervised learning
The benchmark consists of 8 tasks in total, among which are the COPA test, a causal reasoning test, and a textual recognition entailment task. After this is done, the platform summarises the AI’s ability to handle various NLP tasks.
This is relevant because, as Facebook AI says, humans can achieve a 100% accuracy on COPA. Since Google’s BERT managed to achieve merely 74%, this is a clear indicator that there’s still plenty of room for improvement.
In an attempt to move the boundaries, back in July, Facebook AI has come up with a long-form question answering data set and benchmark. This was designed to make the machines answer with long and complex answers – something the algorithms had never been put up against before. An example of this would be to ask an open-ended question of how jellyfish can function without a brain.
Google’s BERT is also getting an upgrade. The upgrade is known as XLNet, a neural network said to be better than BERT, it has been designed to raise the bar in training the computer to process document-language.