# Distilbart paper
**URL:**
**Author:**  [@Hildweig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHildweig)%5C
**Post date:** [July 22, 2020, 12:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F1 "2020-07-22T12:43:19Z")
Good evening,
Is there a paper about distilbart? I need documentation for my master thesis and I couldn’t find any. Thanks for your help!
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [July 22, 2020, 2:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F2 "2020-07-22T14:21:41Z")
Hi @Hildweig, There is no paper for distilbart, the idea of distllbart came from @sshleifer’s great mind 😉
You can find the details of the distillation process [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#distilbart)
For the CNN models, the distiiled model is created by copying the alternating layers from `bart-large-cnn` . This is no teacher distillation i.e you just copy layers from teacher model and then fine-tune the student model in stander way.
And for XSUM it uses ombination of Distillbert’s ce\_loss and the hidden states MSE loss used in the tinybert paper.
DistilBERT [paper](https://arxiv.org/abs/1910.01108)
tinybert [paper](https://arxiv.org/abs/1909.10351)
---
**Author:**  [@Hildweig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHildweig)%5C
**Post date:** [July 22, 2020, 2:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F3 "2020-07-22T14:42:47Z")
Thanks a lot! So you’re saying that we still can fine tine distilbart-cnn-12-6 on cnn dm?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [July 22, 2020, 2:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F4 "2020-07-22T14:49:22Z")
distilbart-cnn-12-6 is already fine-tuned on cnn dm. The way it is done is first a student model is created as described above and then its finetuned on cnn dm.
You can create your own student(unfine-tuned) following the readme and then fine-tune it on cnn dm. Here are already available unfine-tuned [students](https://huggingface.co/models?search=sshleifer%2Fstudent)
---
**Author:**  [@Hildweig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHildweig)%5C
**Post date:** [July 22, 2020, 2:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F5 "2020-07-22T14:51:03Z")
Thanks a lot!
---
**Author:**  [@joeddav](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjoeddav)%5C
**Post date:** [July 22, 2020, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F6 "2020-07-22T20:18:21Z")
I’ll add that although there’s nothing out yet, there will likely be a paper at some point.
---
**Author:**  [@Hildweig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHildweig)%5C
**Post date:** [July 30, 2020, 12:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F7 "2020-07-30T12:47:10Z")
For the alternated layers, did he choose for example odd layers/ even layers? Or how does it work?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [July 30, 2020, 3:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F8 "2020-07-30T15:48:31Z")
Pinging @sshleifer
---
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [July 31, 2020, 5:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F9 "2020-07-31T05:04:49Z")
thx for the ping
for 12-6 it’s `[0, 2, 4, 7, 9, 11]` bc I arbitrarily decided I wanted to keep the first and last layer in.
for 12-3 it’s `[0,6,11]` for 12-1 it’s `[0]`. 12-4 `[0, 4, 8, 11]` working way better than 12-3 for mbart-large-enro in early experiments.
Note: A full copy of the teacher would be 12-12.
Code: [https://github.com/huggingface/transformers/blob/09a2f40684f77e62d0fd8485fe9d2d610390453f/examples/seq2seq/distillation.py#L403](https://github.com/huggingface/transformers/blob/09a2f40684f77e62d0fd8485fe9d2d610390453f/examples/seq2seq/distillation.py#L403)
```auto
layers_to_copy = { # maps num layers in student -> which teacher layers to copy
1: [0],
2: [0, 6],
3: [0, 6, 11],
4: [0, 4, 8, 11],
6: [0, 2, 4, 7, 9, 11],
9: [0, 1, 2, 4, 5, 7, 9, 10, 11],
12: all_layers,
}
```
---
**Author:**  [@Hildweig](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHildweig)%5C
**Post date:** [August 8, 2020, 4:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F10 "2020-08-08T16:19:49Z")
Also, what training algorithm was used to train /finetune ? And the learning rate value for which it was fine-tuned on cnn dailymail?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 9, 2020, 8:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F11 "2020-08-09T08:46:17Z")
Hi @Hildweig, it was fine-tuned the way you fine-tune/train any encoder-decoder Transformer model.
On a high level, the encoder takes the input sequence, creates a hidden representation of it.
The deocoder then receives the encoder representation, and is trained to generate the output sequence auto-regressively using teacher forcing.
You may have already read this, but [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/) by Jay Alammar explains it really well.
all the training hparams are in this [file](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/train_distilbart_cnn.sh)
---
**Author:**  [@dhruvrnaik](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhruvrnaik)%5C
**Post date:** [September 22, 2020, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F12 "2020-09-22T07:57:57Z")
@sshleifer is there a reason that the R-L scores in the sheet(~30) differ a lot from R-L scores reported by Facebook in the paper (~40)? (specifically on the CNN/DM dataset)
---
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 22, 2020, 3:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F13 "2020-09-22T15:16:46Z")
rouge package: [https://github.com/huggingface/transformers/issues/6808](https://github.com/huggingface/transformers/issues/6808)
---
**Author:**  [@993](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2F993)%5C
**Post date:** [November 10, 2020, 8:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F14 "2020-11-10T08:49:47Z")
If Anyone is still reading this:
The Paper was released on 28.09.2020:
[Pre-trained Summarization Distillation](https://arxiv.org/abs/2010.13002)
---
**Author:**  [@hiphan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhiphan)%5C
**Post date:** [March 25, 2021, 7:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F15 "2021-03-25T19:50:12Z")
@sshleifer Hi, Could you please share the hyperparameters that you use for Distilbart-xsum-12-3? I am trying to reproduce the result with KD loss but getting a lower result (21.18 Rouge-2 vs. 21.63 reported in the paper). Thank you!
---
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [March 26, 2021, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F16 "2021-03-26T13:36:02Z")
They should be somewhere in the `research_projects/` directory, but things have moved around.
If you show me a command, I can check for glaring errors.
Before you retrain, make sure your `max_length` and `length_penalty` (may have been renamed) match my xsum-12-3 config in the model hub.
---
**Author:**  [@hiphan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhiphan)%5C
**Post date:** [March 26, 2021, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F17 "2021-03-26T15:34:54Z")
This is the command that I am using
```auto
train_distilbart_seq2seq --output_dir /tmp/path/to/dir/ \
--model_name facebook/bart-large-xsum \
--tokenizer_name facebook/bart-large-xsum \
--task summarization \
--dataset_name xsum \
--learning_rate 3e-4 \ (taken from the [example script](https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation))
--use_kd_loss True \
--alpha_data 1.0 \
--alpha_logits 0.8 \
--alpha_hidden 3.0 \
--max_source_length 1024 \
--max_target_length 256 \
--do_train \
--do_eval \
--do_predict \
--num_beams 6 \
--num_train_epochs 5 \
--evaluation_strategy steps \
--save_total_limit 5 \
--load_best_model_at_end \
--predict_with_generate
```
I did not specify `min_length`, `max_length`, and `length_penalty` as I let them take the values from the teacher model (`min_length=11, max_length=62`, which match the config in the model hub, I will need to double-check `length_penalty`). Other than that, please let me know if there’s anything wrong with my command. Thank you!
---
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [March 27, 2021, 6:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistilbart-paper%2F428%2F18 "2021-03-27T18:12:53Z")
Command looks fine, but you may want different `length_penalty` than teacher.