Generative AI For The Rest Of US

Foundation Models: The Building Blocks of Generative AI

BERT and MUM: Powering Google Search

Bidirectional Encoder Representations from Transformers (BERT) was developed by Google in 2018. BERT is a transformer-based model that achieved state-of-the-art performance on a wide range of natural language processing tasks. It was trained on a large corpus of text data and fine-tuned on specific tasks using labeled data. BERT has been widely adopted in the NLP community and has become a standard benchmark for many NLP tasks.

"In a little over a year, BERT has become a ubiquitous baseline in Natural Language Processing (NLP) experiments counting over 150 research publications analyzing and improving the model." ~ A Primer in BERTology: What We Know About How BERT Works (Rogers et al., TACL 2020)

Google's foundational models have been used in various applications, such as chatbots, question-answering systems, and sentiment analysis tools. They have also been integrated into various Google products, including Google Search, to improve the search engine's understanding of user queries and provide more relevant search results. A few years after the introduction of BERT, Google released the Multitask Unified Model (MUM) in 2021. MUM is a more advanced version of BERT that can handle more complex queries and tasks. It is designed to understand and generate information across multiple modalities, languages, and tasks, and it can process text, images, and videos. Using this deep learning model, Google was able to introduce new features such as Google Lens, which allows users to search for information using images taken with their smartphone cameras, smarter recommendations in Google Search, and more accurate search results.

To illustrate the impact of BERT and MUM on the user experience, consider the following scenario: Imagine you're exploring a new city and have a craving for Chinese cuisine. You pull out your smartphone and search for "Chinese restaurant" to find a nearby place to satisfy your hunger. In this scenario, Google acts as an advanced concierge and precisely interprets your query. In the past, Google's method of processing search queries was somewhat basic, similar to a simple matching game. It would search for exact matches of the words "Chinese" and "restaurant" across numerous web pages. However, with the integration of more intelligent GenAI models, Google has significantly improved its linguistic comprehension. Generative AI allows Google to understand the intent behind your search, recognizing that "shinese food" is a typo and that you're likely looking for Chinese cuisine, not food for your shinese-crested dog. Now, when you search for "Chinese restaurant," Google is better equipped to interpret your intent accurately, prioritizing relevant local dining options over unrelated results.

Moreover, the nuanced understanding of BERT and MUM benefits website creators as well. In the past, webmasters might have excessively repeated the phrase "Chinese restaurant" to increase their visibility in search results (Keyword stuffing), compromising the quality of their content. With GenAI models like MUM, they can write more naturally, confident that Google will understand the relevance of their content to the search query.

GPT-1 and GPT-2: The Ancestors of ChatGPT

GPT is a family of large language models developed by OpenAI, and it excels in several natural language processing tasks like content generation, translation, summarization, and more. GPT-1, the first version of the GPT model, was released in 2018. The related paper, "Improving Language Understanding by Generative Pre-Training" by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever from OpenAI, demonstrated the effective performance of generative pre-training of a language model on unlabeled text followed by discriminative fine-tuning on specific tasks. It was followed by GPT-2 in 2019, which was significantly larger and more powerful. OpenAI, the organization behind GPT-2, initially decided not to release the full model due to concerns about its potential misuse for generating fake news and other malicious content. "Too dangerous to release" was the rationale behind this decision. However, they later released a smaller version of the model for research purposes.

★ AI/ML experts questioned the "too dangerous to release" qualification of GPT-2 and found it to be more of a marketing strategy than a genuine concern.

You have certainly heard about the fact that GPT-2 has 1.5 billion parameters. This simply means that the model has 1.5 billion weights that are learned during the training process. As a reminder, a weight is a parameter that the model learns from the data. Consider it as the 'knowledge' that the model acquires during training.

GPT-2 has been used in several applications, but perhaps the most famous one is the AI Dungeon game, where it generates text-based adventures based on user input. The game was a hit and showcased the potential of Generative AI for interactive storytelling.

ChatGPT and the Future: The Evolution Continues

OpenAI introduced GPT-3 in 2020, demonstrating the company's advancements. With 175 billion parameters, GPT-3 surpassed its predecessor GPT-2 by almost 116 times in terms of parameters. The model gained recognition for its ability to produce text that closely resembles human writing across various tasks such as creative writing and coding.

Following the success of GPT-3, GPT-4 (and 5) entered the scene, propelling Generative AI to new heights. Comparing GPT-3 to GPT-4+ is like distinguishing between a chef and a novice cook - the output from GPT-4 showcases a quality that truly sets it apart. While GPT-3 excelled in generating human-like text, GPT-4 elevated text generation by offering enhanced quality and precision that users of advanced AI models anticipate. Although not without imperfections, those familiar with both models would easily discern the distinctions between the outputs of both models. There isn't a task that GPT-3 can carry out which cannot be improved upon by GPT-4.

GPT-4 is not just another LLM that is more robust than GPT-3 and GPT-3.5, it is a multimodal model, meaning it accepts image and text inputs and generates text. The future of GPT-4 appears to be part of a broader strategy of iterative development and deployment of AI technologies. OpenAI seems to prioritize releasing impactful and potentially groundbreaking updates or new models that build upon GPT-4's foundation.

Notable platforms emerging from the GPT lineage include ChatGPT, a conversational AI model that uses the power of GPT models to engage in human-like conversations. It's important to note that ChatGPT is not a standalone model; instead, it builds on GPT models and adds chat-specific features like memory, conversation history, and context handling.

“ChatGPT is scary good." ~ Elon Musk

GPT has been used on various platforms, such as:

• Microsoft's Copilot, a chatbot that replaced the discontinued Cortana.
• Copy.ai, a platform that generates marketing copies and product descriptions, has emerged alongside other similar platforms like Writesonic, Jasper, and CopySmith, which also use GPT models.
• Duolingo, a language learning platform, employs AI for personalized learning experiences.
• Replika is an AI chatbot designed to serve as a personal AI friend.
• mycharacter.ai is a dApp built on the AI Protocol. It uses the CharacterGPT V2 Multimodal AI System to create realistic and interactive AI characters, which are collectible on the Polygon blockchain.

Besides GPT models, OpenAI has developed other models like DALL-E, Whisper, TTS, and Sora designed for specific tasks; respectively, image generation, audio synthesis, text-to-speech, and video generation.