关于 ChatGPT 必看的 10 篇论文

关于 ChatGPT 必看的 10 篇论文 目录1 Transformer2 GPT-33 InstructGPT4 Sparrow5 RLHF6 TAMER7 PPO8 In-Context Learning8.1 Why Can GPT Learn In-Context8.2 What learning algorithm is in-context learning9 Prompt参考

推荐整理分享关于 ChatGPT 必看的 10 篇论文，希望有所帮助，仅作参考，欢迎阅读内容。

文章相关热门搜索词:,内容如对您有帮助，希望把文章链接给更多的朋友！

团队博客: CSDN AI小组

相关阅读

ChatGPT 简介从 ELMo 到 ChatGPT：历数 NLP 近 5 年必看大模型

2022年11月，OpenAI推出人工智能聊天原型ChatGPT，再次赚足眼球，为AI界引发了类似AIGC让艺术家失业的大讨论。

ChatGPT 是一种专注于对话生成的语言模型。它能够根据用户的文本输入，产生相应的智能回答。这个回答可以是简短的词语，也可以是长篇大论。其中 GPT 是 Generative Pre-trained Transformer（生成型预训练变换模型）的缩写。

下面列出了学习 ChatGPT 必看的 10 篇论文。

1 Transformer

ChatGPT 使用的预训练模型 GPT，是在 Transformer 中的 decoder 基础上进行改造的。

Title：Attention Is All You NeedAbstract：The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.摘要：占主导地位的序列转导模型是基于复杂的递归或卷积神经网络，包括一个编码器和一个解码器。性能最好的模型还通过注意机制将编码器和解码器连接起来。我们提出了一个新的简单的网络结构–Transformer，它只基于注意力机制，完全不需要递归和卷积。在两个机器翻译任务上的实验表明，这些模型在质量上更胜一筹，同时也更容易并行化，需要的训练时间也大大减少。我们的模型在WMT 2014英德翻译任务中达到了28.4 BLEU，比现有的最佳结果（包括合集）提高了2 BLEU以上。在WMT 2014英法翻译任务中，我们的模型在8个GPU上训练了3.5天后，建立了新的单模型最先进的BLEU得分，即41.0分，这只是文献中最佳模型的训练成本的一小部分。2 GPT-3

GPT 家族与 BERT 模型都是知名的 NLP 预训练模型，都基于 Transformer 技术。GPT-1 只有12个 Transformer 层，而到了 GPT-3，则增加到 96 层。

Title：Language Models are Few-Shot LearnersAbstract：Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.摘要：最近的工作表明，在许多NLP任务和基准上，通过对大型文本语料库进行预训练，然后对特定的任务进行微调，可以获得巨大的收益。虽然在结构上通常是任务无关的，但这种方法仍然需要特定任务的微调数据集，包括几千或几万个例子。相比之下，人类通常只需通过几个例子或简单的指令就能完成一项新的语言任务–而目前的NLP系统在很大程度上仍难以做到这一点。在这里，我们展示了扩大语言模型的规模，大大改善了与任务无关的、少量的性能，有时甚至达到了与之前最先进的微调方法的竞争力。具体来说，我们训练了GPT-3，一个具有1750亿个参数的自回归语言模型，比以前的任何非稀疏语言模型多10倍，并测试了它在少数情况下的性能。对于所有的任务，GPT-3的应用没有任何梯度更新或微调，纯粹通过与模型的文本互动来指定任务和少量演示。GPT-3在许多NLP数据集上取得了强大的性能，包括翻译、回答问题和cloze任务，以及一些需要即时推理或领域适应的任务，如解读单词、在句子中使用一个新词或进行3位数的算术。同时，我们也发现了一些数据集，在这些数据集中，GPT-3的几率学习仍然很困难，还有一些数据集，GPT-3面临着与大型网络语料库训练有关的方法学问题。最后，我们发现，GPT-3可以生成人类评价者难以区分的新闻文章样本。我们讨论了这一发现和GPT-3总体上的更广泛的社会影响。3 InstructGPT

ChatGPT 的训练流程，主要参考自 instructGPT ，ChatGPT 是改进的 instructGPT。

Title：Training language models to follow instructions with human feedbackAbstract：Making language models bigger does not inherently make them better at following a user’s intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.摘要：让语言模型变得更大并不意味着它们能更好地遵循用户的意图。例如，大型语言模型可以产生不真实的、有毒的或根本对用户没有帮助的输出。换句话说，这些模型没有与用户保持一致。在本文中，我们展示了一个途径，通过人类反馈的微调，在广泛的任务中使语言模型与用户的意图保持一致。从一组标签员写的提示语和通过OpenAI API提交的提示语开始，我们收集了一组标签员演示的所需模型行为的数据集，我们利用监督学习对GPT-3进行微调。然后，我们收集模型输出的排名数据集，我们利用人类反馈的强化学习来进一步微调这个监督模型。我们把产生的模型称为InstructGPT。在人类对我们的提示分布的评估中，尽管参数少了100倍，但1.3B参数的InstructGPT模型的输出比175B的GPT-3的输出更受欢迎。此外，InstructGPT模型显示了真实性的改善和有毒输出生成的减少，同时在公共NLP数据集上的性能回归最小。尽管InstructGPT仍然会犯一些简单的错误，但我们的结果表明，利用人类反馈进行微调是使语言模型与人类意图相一致的一个有希望的方向。4 Sparrow

DeepMind 的 Sparrow，这个工作发表时间稍晚于 instructGPT，其大致的技术思路和框架与 instructGPT 的三阶段基本类似，不过明显Sparrow 在人工标注方面的质量和工作量是不如instructGPT的。反过来，Sparrow 中把奖励模型分为两个不同 RM 的思路，理论上是优于instructGPT 的。

Title：Improving alignment of dialogue agents via targeted human judgementsAbstract：We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.摘要：我们提出了Sparrow，一个寻求信息的对话代理，与提示的语言模型基线相比，它被训练得更有帮助，更正确，更无害。我们使用来自人类反馈的强化学习来训练我们的模型，并增加了两个新的内容来帮助人类评分者判断代理行为。首先，为了使我们的代理更有帮助和无害，我们将良好对话的要求分解为代理应该遵循的自然语言规则，并分别询问评分者每条规则。我们证明，这种分解使我们能够收集更有针对性的人类对代理行为的判断，并允许更有效的规则条件的奖励模型。第二，我们的代理在收集对模型