ChatGPT

ChatGPT (Chat Generative Pre-trained Transformer) is a large-scale language generation model launched on November 30th, 2022 by OpenAI. The model is trained on a massive dataset of human-generated text and can generate human-like text in a wide range of styles and formats. The OpenAI uses supervised machine learning techniques and reinforcement learning from Human Feedback to train and generate responses. After being released, ChatGPT quickly garnered attention from big tech companies, researchers, artists, and professors, and students. The Chatbot is capable of mimicking human conversation and generating everything from essays to cooking recipes. However, ChatGPT's capabilities raise concerns about misuse, correctness, job market impact, and the impact on education.

OpenAI's ChatGPT Logo

Training

ChatGPT is based on OpenAI's GPT-3.5, an upgraded version of GPT-3, the A.I. text generator . The GPT3.5 language model was initially built from a data set with 175 billion parameters including web content and other publicly available sources . Supervised machine learning techniques were then used to build the model. Supervised learning refers to a class of systems and algorithms that determine a predictive model using data points with known outcomes. The model is learned by training through an appropriate learning algorithm (such as linear regression, random forests, or neural networks) that typically works through some optimization routine to minimize a loss or error function. . It works much like the human brain, using interconnected ‘neurons’ that can learn to identify patterns in data and make predictions about what should come next. By using Reinforcement Learning from Human Feedback, human trainers played the role of both the user and the AI agent—generating a variety of responses to any given input and then evaluating and ranking them from best to worst . This data was used to train a reward model.

An OpenAI reinforcement learning algorithm called Proximal Policy Optimization was used to fine-tune results . Proximal Policy Optimization balances ease of implementation, sample complexity, and ease of tuning to compute an update at each step that minimizes the cost function while ensuring the deviation from the previous policy is relatively small . The Transformer architecture used in ChatGPT is a type of neural network that is particularly well-suited to natural language processing tasks. The architecture is based on the idea of self-attention, which allows the model to weigh the importance of different words in a sentence when generating new text. This allows the model to generate text that is more coherent and relevant to the input.

ChatGPT is similar to other Large Language Models when it comes to alignment problems in training. Typically, these manifest as not following the user's explicit instructions, making up unexisting or wrong facts, difficulty to understand how the model arrived at a particular decision or prediction, and outputs that are biased or toxic because of the biased data it was trained on .

Capabilities

ChatGPT's main function is to mimic a human conversation. However, it can perform other functions as well. It has the ability to generate responses in a chatbot or virtual assistant, brainstorm content ideas on keywords or topics, create personalized communications such as email responses or product recommendations, create marketing content like blog posts or social media updates, translate text from one language to another, generate shorter summaries of long documents, and write and debug computer programs . ChatGPT can understand and generate text in many languages including English, Spanish, French, German, Italian, Chinese, Japanese, Korean, and many more. It can also perform mathematical operations and answer problem sets related to math. It confers the ability to answer questions in science and other general knowledge topics, but it's not an expert in these areas and will tell the user when it is unsure of one of the outputs it generated. The model's vocabulary is quite large, with the ability to understand and generate a wide range of words and phrases. OpenAI attributes ChatGPT's lifelike responses to its vast training sets but warns users that its responses can sometimes be incorrect: "These models were trained on vast amounts of data from the internet written by humans, including conversations, so the responses it provides may sound human-like. It is important to keep in mind that this is a direct result of the system's design (i.e. maximizing the similarity between outputs and the dataset the models were trained on) and that such outputs may be inaccurate, untruthful, and otherwise misleading at times" . Consequently, OpenAI suggests checking whether responses from the model are accurate or not.

ChatGPT also has the potential to make a positive impact on education systems. The AI tool can help with tasks like crafting a syllabus, creating lectures, and grading assignments, which can save educators time and streamline the learning process . ChatGPT can also help students learn by summarizing academic papers and answering questions, acting as a personal tutor. These features can enable students to better understand complex concepts, which could improve their grades and overall academic performance. NPR further explains that the model could create learning plans that are personalized to students. This can help to streamline the efficiency and effectiveness of education and make it more accessible to people all over the world, particularly so in communities that have historically lacked resources for quality accurate education.

Reception and Interest

After launching in November 2022, it took ChatGPT only 5 days to reach 1 million users, a feat that took social media platform Meta (formerly Facebook) 10 months and streaming platform Netflix three years to match . Quickly after ChatGPT gained popularity in late 2022, multiple companies and big tech CEOs showed interest in the chatbot and its abilities. Buzzfeed Inc Chief Executive Jonah Peretti spoke about relying on ChatGPT creator OpenAI to enhance its quizzes and personalize some content for its audiences . In January 2023, Microsoft confirmed it was making a "multibillion dollar" investment in OpenAi and said it plans to expand the partnership as part of a greater effort to add more artificial intelligence to its suite of products. . Shortly after confirming this investment in OpenAI's ChatGPT, Microsoft announced they will be integrating ChatGPT into its Bing search engine which will enable a new kind of search in which people will pose questions to the search engine in natural language and it will generate direct answers and suggestions .

In December 2022, Google internally expressed alarm at the unexpected strength of ChatGPT and the newly discovered potential of large language models to be a serious threat to its main search business. . Sunday Pichai, Google's Chief executive, upended the work of many teams to respond to the threat that ChatGPT poses . In February 2023, Google began introducing an experimental service called "Bard" which is based on its LaMDA AI program. Bard generates text responses to questions asked based on information gathered from the web. Google CEO Sundar Pichai described how this technology would be integrated into existing search capabilities and said some aspects of the technology would be open to outside developers .

Ethical Considerations

Misuse

One of the key ethical considerations with ChatGPT and other similar models is the potential for misuse. For example, the model's ability to generate realistic text could be used to spread misinformation or propaganda. Dr. Marcel Scharth of the University of Sydney demonstrated ChatGPT's ability to generate fake news with a fake news story about former President Barack Obama .

ChatGPT example output shows model's ability to generate harmful fake news stories.

Additionally, the model's ability to impersonate individuals or groups online could be used to harass or defraud others. ChatGPT's ability to write error-free code successfully lowers the bar for code generation and can help less-skilled threat actors effortlessly launch cyber-attacks. In a December 2022 report by Check Point Research, it was shown that ChatGPT was able to successfully conduct a full infection flow, from creating a convincing spear-phishing email to running a reverse shell, capable of accepting commands in English . Notably, the Check Point Research report stated the researchers were able to conduct this full infection flow without writing a single line of code. They were able to complete this cyber-attack solely by putting together the pieces of the ChatGPT outputs. Another Check Point Research report released in January 2023 contained an analysis of major underground hacking communities and found evidence of cybercriminals using OpenAI technologies to develop malicious tools . One such example is the creation of a basic stealer, which searches for 12 common file types (such as MS Office documents, PDFs, and images) across the system, copies them to a temporary folder, zips them, and sends them over the web without encryption or secure transfer methods. Another example is a simple Java snippet that downloads and covertly runs a common SSH and telnet client using Powershell, which can be modified to download and run any program, including malware. Another threat actor used OpenAI's ChatGPT to create a Python script that performs cryptographic operations, including signing files using elliptic curve cryptography, encryption using the Blowfish and Twofish algorithms, RSA keys, certificates stored in PEM format, MAC signing, and hash comparison using the blake2 hash function . The script has the potential to be modified into ransomware if its script and syntax problems are fixed. The threat actor, "UsDoD," is a reputable and active member of the underground community who engages in various illicit activities, such as selling access to compromised companies and stolen databases.

Correctness of Responses

One concern that OpenAI addresses directly is the correctness of ChatGPT's responses. OpenAI notes that "ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers" and fixing this issue is difficult because there’s no source of truth during reinforcement learning, training the model to be more cautious causes it to decline questions that it can answer correctly, and supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows. In early December 2022, Stack Overflow, a question-and-answer site for programmers, temporarily banned users from sharing responses generated by ChatGPT because "ChatGPT simply makes it too easy for users to generate responses and flood the site with answers that seem correct at first glance but are often wrong on close examination" .

Job Market Impact

There is also a concern about the impact of language models like ChatGPT on the job market. With the ability to generate high-quality text, the model could potentially automate the work of writers, editors, and other language-based professions. This could lead to significant job loss and economic disruption. In December 2022, Harvard Business Review warned about the potential for ChatGPT to take over roles traditionally held by humans such as copywriting, answering customer service inquiries, writing news reports, and creating legal documents. . Marketing roles such as market research, Search Engine Optimization, and product descriptions can be automated using ChatGPT which threatens the Marketing Job Market

Impact on Education

ChatGPT has the potential to have negative impacts on education systems as well. One concern is that it could lead to a decrease in critical thinking and creativity among students. When students are relying too heavily on ChatGPT-generated content, quizzes, or homework, they may not be developing their own understanding and problem-solving skills. Additionally, when students are using ChatGPT to generate answers to questions or complete assignments, it could lead to academic dishonesty and cheating. As of January 4th, 2023, New York Public Schools restricted access to the chatbot from its public school internet and devices after citing concerns of "concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of contents” . The decision to ban the chatbot in New York schools came amid widespread fears that it could encourage students to plagiarize. In a statement sent to the Washington Post shortly after these concerns were raised, ChatGPT creator OpenAI said "We don’t want ChatGPT to be used for misleading purposes in schools or anywhere else, so we’re already developing mitigations to help anyone identify text generated by that system" .

Educators aren't alone in worrying about plagiarism and cheating using ChatGPT. Shortly after ChatGPT was released to the public, Edward Tian, an undergraduate senior at Princeton University at the time, saw the need for a system to detect plagiarized material generated by ChatGPT. Tian created a new app called GPTZero to combat the misuse of ChatGPT . Tian's app uses GPT-3 to check for its involvement in creating text, with transparency being his primary motivation for creating it. The app was released on January 2nd and became very popular, with teachers and journalists among those who showed interest in it.

Limitations

There are limitations to what ChatGPT can do, and what it knows. As it is trained on a data set and is not connected to the Internet, there is a cut-off point for its knowledge base, which is currently the end of 2021. So, users may not use ChatGPT to gain information about current events.

A research paper published in January 2023 discovered patterns in ChatGPT content that makes it less suitable for critical applications. The paper found that humans preferred answers from ChatGPT in more than 50% of questions answered related to finance and psychology. But ChatGPT failed at answering medical questions because humans preferred direct answers. Specifically, the researchers cite that "ChatGPT performs poorly in terms of helpfulness for the medical domain in both English and Chinese. The ChatGPT often gives lengthy answers to medical consulting in our collected dataset, while human experts may directly give straightforward answers or suggestions, which may partly explain why volunteers consider human answers to be more helpful in the medical domain" . The paper also mentions that when answering a question that requires professional knowledge from a particular field, ChatGPT may fabricate facts in order to give an answer . For example, in legal questions, ChatGPT may invent some non-existent legal provisions to answer the question. Lastly, ChatGPT may fabricate facts in order to provide a response when a user poses a question that has no existing answer.

With any question, there is always a chance of getting a partially fabricated or entirely fabricated response. ChatGPT is currently still in the process of training and improving. It is OpenAI's suggestion to always include a Human in the loop (HITL). Wherever possible, OpenAI recommends having a person review outputs before they are used in practice. This is especially critical in high-stakes domains and for code generation .

References: