# Summarization on long documents

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsummarization-on-long-documents%2F920%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 29, 2020, 5:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsummarization-on-long-documents%2F920 "2020-08-29T17:57:00Z")\
**Posts on this page:** 1\
**Showing post:** 30

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmarcoabrate)%5C
**Post date:** [December 2, 2020, 9:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsummarization-on-long-documents%2F920%2F30 "2020-12-02T09:49:29Z")

</div>

Hey @MoritzLaurer @echatzikyriakidis

The regex `.(?!\d)|\n` works for Python, it just says to split where there is a full stop (not followed by a number, to avoid splitting floating points) or a new line. Consider changing it to what’s more suitable to you. For example I do not have any URL in my text, otherwise it would be a problem.

`num_tok` is the number of tokens of the entire text `text`.

The `tokenizer` is either Bart or Pegasus, works for both. I use the `tokenize` function so that i do not get BOS and EOS for each sentence.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsummarization-on-long-documents%2F920)._