# Tokenizers v0.8.0 is out! **URL:** **Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C **Post date:** [July 7, 2020, 6:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizers-v0-8-0-is-out%2F51%2F1 "2020-07-07T18:48:00Z") ## Highlights of this release - We can now encode both pre-tokenized inputs, and raw strings. This is especially usefull when processing datasets that are already pre-tokenized like for NER (Name Entity Recognition), and helps while applying labels to each word. - Full tokenizer serialization. It is now easy to save a tokenizer to a single JSON file, to later load it back with just one line of code. That’s what sharing a Tokenizer means now: 1 line of code. - With the serialization comes the compatibility with `Pickle`! The Tokenizer, all of its components, Encodings, everything can be pickled! - Training a tokenizer is now even faster (up to 5-10x) than before! - Compatibility with `multiprocessing`, even when using the `fork` start method. Since this library makes heavy use of the multithreading capacities of our computers to allows a very fast tokenization, this led to problems (deadlocks) when used with `multiprocessing`. This version now allows to disable the parallelism, and will warn you if this is necessary. - And a lot of other improvements, and fixes. ### Fixed - [#286](https://github.com/huggingface/tokenizers/pull/286): Fix various crash when training a BPE model - [#309](https://github.com/huggingface/tokenizers/pull/309): Fixed a few bugs related to additional vocabulary/tokens ### Added - [#272](https://github.com/huggingface/tokenizers/pull/272): Serialization of the `Tokenizer` and all the parts (`PreTokenizer`, `Normalizer`, …). This adds some methods to easily save/load an entire tokenizer (`from_str`, `from_file`). - [#273](https://github.com/huggingface/tokenizers/pull/273): `Tokenizer` and its parts are now pickable - [#289](https://github.com/huggingface/tokenizers/pull/289): Ability to pad to a multiple of a specified value. This is especially useful to ensure activation of the Tensor Cores, while ensuring padding to a multiple of 8. Use with `enable_padding(pad_to_multiple_of=8)` for example. - [#298](https://github.com/huggingface/tokenizers/pull/298): Ability to get the currently set truncation/padding params - [#311](https://github.com/huggingface/tokenizers/pull/311): Ability to enable/disable the parallelism using the `TOKENIZERS_PARALLELISM` environment variable. This is especially usefull when using `multiprocessing` capabilities, with the `fork` start method, which happens to be the default on Linux systems. Without disabling the parallelism, the process dead-locks while encoding. (Cf [#187] for more information) ### Changed - Improved errors generated during truncation: When the provided max length is too low are now handled properly. - [#249](https://github.com/huggingface/tokenizers/pull/249) `encode` and `encode_batch` now accept pre-tokenized inputs. When the input is pre-tokenized, the argument `is_pretokenized=True` must be specified. - [#276](https://github.com/huggingface/tokenizers/pull/276): Improve BPE training speeds, by reading files sequentially, but parallelizing the processing of each file - [#280](https://github.com/huggingface/tokenizers/pull/280): Use `onig` for byte-level pre-tokenization to remove all the differences with the original implementation from GPT-2 - [#309](https://github.com/huggingface/tokenizers/pull/309): Improved the management of the additional vocabulary. This introduces an option `normalized`, controlling whether a token should be extracted from the normalized version of the input text.