# Multiple sentences in RoBERTa training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiple-sentences-in-roberta-training%2F9146%3E%5C
**Category:** 🤗Datasets\
**Created:** [August 10, 2021, 11:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiple-sentences-in-roberta-training%2F9146 "2021-08-10T23:32:33Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![HyeyeonKoo](https://avatars.discourse-cdn.com/v4/letter/h/c2a13f/32.png) [@HyeyeonKoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHyeyeonKoo)%5C
**Post date:** [August 10, 2021, 11:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmultiple-sentences-in-roberta-training%2F9146%2F1 "2021-08-10T23:32:33Z")

</div>

Hello.

I have a question which is similar with [/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconcatenate-sentances%2F4233%5D(https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconcatenate-sentances%2F4233)%2C but there was no answer, so I ask a question.

I make the script for training RoBERTa and I make the dataset and collator like below.

```auto
...
        train_dataset = LineByLineTextDataset(
            tokenizer=tokenizer,
            file_path=os.path.join(dir, "train_data.txt"),
            block_size=tokenizer.max_len_single_sentence
        )

        data_collator = DataCollatorForLanguageModeling(
            tokenizer=tokenizer, mlm=True, mlm_probability=0.15
        )

...

        trainer = CustomTrainer(
            model=model,
            args=train_config,
            data_collator=data_collator,
            train_dataset=train_dataset,
            custom_logger=logger
        )

```

I check that the each length of train dataset is less than 512. I know that for RoBERTa training, I need to make multiple sentences to one sequence like [CLS] … [SEP] … [SEP] … [PAD].  
Is trainer process it automatically? or Is there any method that I can handle it?