大语言模型底层原理你都知道吗？大语言模型底层架构之二gpt实现-yb体育官方

3.3. 预处理语料集合

在启动整个模型训练之前，还需要将预训练语料根据训练好的tokenizer 进行处理。如果文档长度超过512 个词元（token），那么就直接进行截断。数据处理代码如下所示：

def encode_with_truncation(examples):
    """mapping function to tokenize the sentences passed with truncation"""
    return tokenizer(examples["text"], truncation=true, padding="max_length",
        max_length=max_length, return_special_tokens_mask=true)
def encode_without_truncation(examples):
    """mapping function to tokenize the sentences passed without truncation"""
    return tokenizer(examples["text"], return_special_tokens_mask=true)
# the encode function will depend on the truncate_longer_samples variable
encode = encode_with_truncation if truncate_longer_samples else encode_without_truncation
# tokenizing the train dataset
train_dataset = d["train"].map(encode, batched=true)
# tokenizing the testing dataset
test_dataset = d["test"].map(encode, batched=true)
if truncate_longer_samples:
    # remove other columns and set input_ids and attention_mask as pytorch tensors
    train_dataset.set_format(type="torch", columns=["input_ids", "attention_mask"])
    test_dataset.set_format(type="torch", columns=["input_ids", "attention_mask"])
else:
    # remove other columns, and remain them as python lists
    test_dataset.set_format(columns=["input_ids", "attention_mask", "special_tokens_mask"])
    train_dataset.set_format(columns=["input_ids", "attention_mask", "special_tokens_mask"])

truncate_longer_samples 布尔变量来控制用于对数据集进行词元处理的encode() 回调函数。如果设置为true，则会截断超过最大序列长度（max_length）的句子。否则，不会截断。如果设为truncate_longer_samples 为false，需要将没有截断的样本连接起来，并组合成固定长度的向量。

3.4. 模型训练

在构建了处理好的预训练语料之后，就可以开始模型训练。代码如下所示：

# initialize the model with the config
model_config = bertconfig(vocab_size=vocab_size, max_position_embeddings=max_length)
model = bertformaskedlm(config=model_config)
# initialize the data collator, randomly masking 20% (default is 15%) of the tokens
# for the masked language modeling (mlm) task
data_collator = datacollatorforlanguagemodeling(
    tokenizer=tokenizer, mlm=true, mlm_probability=0.2
)
training_args = trainingarguments(
    output_dir=model_path, # output directory to where save model checkpoint
    evaluation_strategy="steps", # evaluate each `logging_steps` steps
    overwrite_output_dir=true,
    num_train_epochs=10, # number of training epochs, feel free to tweak
    per_device_train_batch_size=10, # the training batch size, put it as high as your gpu memory fits
    gradient_accumulation_steps=8, # accumulating the gradients before updating the weights
    per_device_eval_batch_size=64, # evaluation batch size
    logging_steps=1000, # evaluate, log and save model checkpoints every 1000 step
    save_steps=1000,
    # load_best_model_at_end=true, # whether to load the best model (in terms of loss)
    # at the end of training
    # save_total_limit=3, # whether you don't have much space so you
    # let only 3 model weights saved in the disk
)
trainer = trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)
# train the model
trainer.train()

开始训练后，可以如下输出结果：

[10135/79670 18:53:08 < 129:35:53, 0.15 it/s, epoch 1.27/10]
step training loss validation loss
1000 6.904000 6.558231
2000 6.498800 6.401168
3000 6.362600 6.277831
4000 6.251000 6.172856
5000 6.155800 6.071129
6000 6.052800 5.942584
7000 5.834900 5.546123
8000 5.537200 5.248503
9000 5.272700 4.934949
10000 4.915900 4.549236

3.5. 模型使用

基于训练好的模型，可以针对不同应用需求进行使用。

# load the model checkpoint
model = bertformaskedlm.from_pretrained(os.path.join(model_path, "checkpoint-10000"))
# load the tokenizer
tokenizer = berttokenizerfast.from_pretrained(model_path)
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
# perform predictions
examples = [
"today's most trending hashtags on [mask] is donald trump",
"the [mask] was cloudy yesterday, but today it's rainy.",
]
for example in examples:
    for prediction in fill_mask(example):
        print(f"{prediction['sequence']}, confidence: {prediction['score']}")
    print("="*50)

可以得到如下输出：

today's most trending hashtags on twitter is donald trump, confidence: 0.1027069091796875
today's most trending hashtags on monday is donald trump, confidence: 0.09271949529647827
today's most trending hashtags on tuesday is donald trump, confidence: 0.08099588006734848
today's most trending hashtags on facebook is donald trump, confidence: 0.04266013577580452
today's most trending hashtags on wednesday is donald trump, confidence: 0.04120611026883125
==================================================
the weather was cloudy yesterday, but today it's rainy., confidence: 0.04445931687951088
the day was cloudy yesterday, but today it's rainy., confidence: 0.037249673157930374
the morning was cloudy yesterday, but today it's rainy., confidence: 0.023775646463036537
the weekend was cloudy yesterday, but today it's rainy., confidence: 0.022554103285074234
the storm was cloudy yesterday, but today it's rainy., confidence: 0.019406016916036606
==================================================

本篇文章详细重点介绍gpt 无监督预训练、有监督下游任务微调以及基于huggingface 的预训练语言模型实践，下一篇文章我将介绍大语言模型网络结构和注意力机制优化以及相关实践。

请记得点赞关注支持一下！！

参考文章：