# Conversion to CoreML for On-Device Use

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%3E%5C
**Category:** Models\
**Created:** [January 1, 2022, 8:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284 "2022-01-01T08:44:35Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlo)%5C
**Post date:** [January 1, 2022, 8:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F1 "2022-01-01T08:44:35Z")

</div>

Hi Community - I’ve been playing around with converting HF models to CoreML for native, on-device use.

I’ve been able to convert GPT2 and basic BERT models but am having issues with BigBird-Pegasus.

I’m having a host of errors from “Tracer Warnings” to pytorch deprecation warnings. I’ve gone through the original paper, but there is scant information on the implementation for me to get a solid sense of the input shape and conversion requirements. I’m not sure what to even script, because the sparse attention is new to me. I’m hesitant to fork the HF implementation to knock down the obvious errors, because of the sheer volume.

Considering the dynamic input of the model, I am not positive what the input should be, so I played with various random tokens and tokenized inputs. Can’t even get past the trace.

Any help would be great. My current script and the full error are below (yes, I am aware of the Torch version warning).

> ### Conversion Script
> 
> import coremltools  
> import os  
> import torch  
> from torchsummary import summary  
> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
> 
> transformers\_cache\_dir = os.environ.get(‘TRANSFORMERS\_CACHE’) if \>os.environ.get(‘TRANSFORMERS\_CACHE’) else “/Users/m/transformers\_cache/”
> 
> ##### Load pre-trained model.
> 
> torch\_model = AutoModelForSeq2SeqLM.from\_pretrained(“google/bigbird-pegasus-large-pubmed”, torchscript=True, cache\_dir=transformers\_cache\_dir)
> 
> ##### Load tokenizer.
> 
> tokenizer = AutoTokenizer.from\_pretrained(“google/bigbird-pegasus-large-pubmed”, cache\_dir=transformers\_cache\_dir)
> 
> ##### Set model to evaluation mode.
> 
> torch\_model.eval()
> 
> ##### Create dummy input.
> 
> random\_tokens = torch.randint(10000, (1,4096))
> 
> traced\_model = torch.jit.trace(torch\_model, random\_tokens) \<----Errors out here.

> ### Error
> 
> WARNING:root:Torch version 1.10.1 has not been tested with coremltools. You may run into unexpected errors. Torch 1.9.1 is the most recent version that has been tested.  
> {‘input\_ids’: tensor([[8783, 47694, 15934, …, 110, 105, 1]]), ‘attention\_mask’: tensor([[1, 1, 1, …, 1, 1, 1]])}  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1855: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if self.attention\_type == “block\_sparse” and input\_shape[1] \<= max\_tokens\_to\_attend:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:2019: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if padding\_len \> 0:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1979: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> assert (  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:2004: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> blocked\_encoder\_mask = attention\_mask.view(batch\_size, seq\_length // block\_size, block\_size)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:277: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> assert from\_seq\_length % from\_block\_size == 0, “Query sided sequence length must be multiple of block size”  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:278: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> assert to\_seq\_length % to\_block\_size == 0, “Key/Value sided sequence length must be multiple of block size”  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:374: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> if from\_seq\_len // from\_block\_size != to\_seq\_len // to\_block\_size:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:374: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if from\_seq\_len // from\_block\_size != to\_seq\_len // to\_block\_size:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:383: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if from\_seq\_len in [1024, 3072, 4096]: # old plans used in paper  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:857: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> if (2 \* num\_rand\_blocks + 5) \< (from\_seq\_length // from\_block\_size):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:857: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if (2 \* num\_rand\_blocks + 5) \< (from\_seq\_length // from\_block\_size):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:971: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> from\_seq\_length // from\_block\_size == to\_seq\_length // to\_block\_size  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:970: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> assert (  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:974: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> assert from\_seq\_length in plan\_from\_length, “Error from sequence length not in plan!”  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:977: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> num\_blocks = from\_seq\_length // from\_block\_size  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:979: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> plan\_block\_length = np.array(plan\_from\_length) // from\_block\_size  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:981: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> max\_plan\_idx = plan\_from\_length.index(from\_seq\_length)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:407: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.  
> rand\_attn = torch.tensor(rand\_attn, device=query\_layer.device, dtype=torch.long)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:834: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> num\_windows = from\_seq\_length // from\_block\_size - 2  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:835: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won’t change the number of iterations executed (and might lead to errors or silently give incorrect results).  
> rand\_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to\_blocked\_mask, rand\_attn)])  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:415: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> blocked\_query\_matrix = query\_layer.view(bsz, n\_heads, from\_seq\_len // from\_block\_size, from\_block\_size, -1)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:416: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> blocked\_key\_matrix = key\_layer.view(bsz, n\_heads, to\_seq\_len // to\_block\_size, to\_block\_size, -1)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:417: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> blocked\_value\_matrix = value\_layer.view(bsz, n\_heads, to\_seq\_len // to\_block\_size, to\_block\_size, -1)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:781: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if params.shape[:2] != indices.shape[:2]:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:790: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> torch.arange(indices.shape[0] \* indices.shape[1] \* num\_indices\_to\_gather, device=indices.device)  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:422: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> bsz, n\_heads, to\_seq\_len // to\_block\_size - 2, n\_rand\_blocks \* to\_block\_size, -1  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:426: UserWarning: **floordiv** is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the ‘trunc’ function NOT ‘floor’). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding\_mode=‘trunc’), or for actual floor division, use torch.div(a, b, rounding\_mode=‘floor’).  
> bsz, n\_heads, to\_seq\_len // to\_block\_size - 2, n\_rand\_blocks \* to\_block\_size, -1  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1950: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if padding\_len \> 0:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:2082: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if input\_shape[-1] \> 1:  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1290: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if attn\_weights.size() != (bsz \* self.num\_heads, tgt\_len, src\_len):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1296: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if attention\_mask.size() != (bsz, 1, tgt\_len, src\_len):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py:1327: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  
> if attn\_output.size() != (bsz \* self.num\_heads, tgt\_len, self.head\_dim):  
> Traceback (most recent call last):  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/\_trace.py”, line 443, in run\_mod\_and\_filter\_tensor\_outputs  
> outs = wrap\_retval(mod(\*\_clone\_inputs(inputs)))  
> RuntimeError: The following operation failed in the TorchScript interpreter.  
> Traceback of TorchScript (most recent call last):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(408): bigbird\_block\_sparse\_attention  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(284): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1190): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1382): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1928): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(2391): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(2520): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/_trace.py(958): trace\_module  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/trace.py(741): trace  
> /Users/m/Open Source/bigbird-pegasus-large-pubmed/torch\_to\_coreml.py(122):   
> RuntimeError: set\_storage\_offset is not allowed on a Tensor created from .data or .detach().  
> If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage\_offset)  
> without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.  
> For example, change:  
> x.data.set(y)  
> to:  
> with torch.no\_grad():  
> x.set_(y)
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):  
> File “/Users/m/Open Source/bigbird-pegasus-large-pubmed/torch\_to\_coreml.py”, line 122, in   
> traced\_model = torch.jit.trace(torch\_model, inputs[‘input\_ids’])  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/\_trace.py”, line 741, in trace  
> return trace\_module(  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/\_trace.py”, line 983, in trace\_module  
> \_check\_trace(  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/autograd/grad\_mode.py”, line 28, in decorate\_context  
> return func(\*args, \*\*kwargs)  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/\_trace.py”, line 516, in \_check\_trace  
> traced\_outs = run\_mod\_and\_filter\_tensor\_outputs(traced\_func, inputs, “trace”)  
> File “/Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/\_trace.py”, line 449, in run\_mod\_and\_filter\_tensor\_outputs  
> raise TracingCheckError(  
> torch.jit.\_trace.TracingCheckError: Tracing failed sanity checks!  
> encountered an exception while running the trace with test inputs.  
> Exception:  
> The following operation failed in the TorchScript interpreter.  
> Traceback of TorchScript (most recent call last):  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(408): bigbird\_block\_sparse\_attention  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(284): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1190): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1382): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(1928): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(2391): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/transformers/models/bigbird\_pegasus/modeling\_bigbird\_pegasus.py(2520): forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1090): \_slow\_forward  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py(1102): \_call\_impl  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/_trace.py(958): trace\_module  
> /Users/m/venvs/pytorch-coremltools-transformers/lib/python3.9/site-packages/torch/jit/trace.py(741): trace  
> /Users/m/Open Source/bigbird-pegasus-large-pubmed/torch\_to\_coreml.py(122):   
> RuntimeError: set\_storage\_offset is not allowed on a Tensor created from .data or .detach().  
> If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage\_offset)  
> without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.  
> For example, change:  
> x.data.set(y)  
> to:  
> with torch.no\_grad():  
> x.set_(y)

---

<div class="post-metadata">

**Author:** ![atomicai](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@atomicai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fatomicai)%5C
**Post date:** [March 27, 2022, 2:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F2 "2022-03-27T02:03:36Z")

</div>

Wait, what: you converteg GPT-2 to “.mlmodel” using coremltools without any errors and successfully tested it on ios device ?

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjulien-c)%5C
**Post date:** [April 26, 2022, 2:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F3 "2022-04-26T14:23:17Z")

</div>

tagging @Matthijs just in case!

@mlo @atomicai are you still exploring CoreML conversion of HF-hosted models? we are exploring this subject a bit too right now

---

<div class="post-metadata">

**Author:** ![iamcos](https://avatars.discourse-cdn.com/v4/letter/i/f475e1/32.png) [@iamcos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiamcos)%5C
**Post date:** [August 24, 2022, 7:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F4 "2022-08-24T19:56:06Z")

</div>

Wow, that means that these models are getting NE and GPU support out of the script.

> [@mlo](#):
>
> My current script

Coremltools i tried today and with simple models it seems to work and be pretty self explanatory, yet would love to learn about your expirience too.  
Could you share details of your approach and scripts to converting these networks you mentioned here also?

---

<div class="post-metadata">

**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlo)%5C
**Post date:** [August 24, 2022, 8:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F5 "2022-08-24T20:11:09Z")

</div>

I could convert basic models but could never get other transformers with more cutting-edge attention to work (e.g. BB-P).

That said, it’s been a long time since I played around with this, and Apple has been actively working on improving the conversation significantly.

---

<div class="post-metadata">

**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlo)%5C
**Post date:** [August 24, 2022, 8:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F6 "2022-08-24T20:21:54Z")

</div>

Yes, if setup properly with CoreML, you can technically get NE.

It’s been a long time since I’ve played with this but basic transformers always converted for me. If I remember correctly, not too many people were doing these conversions, but Apple has been pushing to improve broader conversion of various models.

If I have time, I’ll write up some of my notes, but I’m rusty and like you said, the standard conversation is relatively straightforward. If you are converting larger, more complex models, my advice would be to look at coreml tool’s new weight compression utilities. They didn’t have that when I was playing around.

Like most things, it’s the optimization and tweaking that can be difficult and opinionated.

---

<div class="post-metadata">

**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlo)%5C
**Post date:** [August 24, 2022, 8:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F7 "2022-08-24T20:24:31Z")

</div>

In short, yes.

---

<div class="post-metadata">

**Author:** ![julien-c](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/julien-c/32/41937_2.png) [@julien-c](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjulien-c)%5C
**Post date:** [August 25, 2022, 10:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F8 "2022-08-25T10:06:10Z")

</div>

Tagging once more @Matthijs and @osanseviero to this interesting thread!

---

<div class="post-metadata">

**Author:** ![Matthijs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthijs/32/4462_2.png) [@Matthijs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatthijs)%5C
**Post date:** [August 25, 2022, 11:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F9 "2022-08-25T11:30:49Z")

</div>

Hi all, I’m currently adding an automated way to do Core ML export from HF Transformers, similar to how the ONNX exports work. I didn’t attempt to convert the BigBird-Pegasus model yet, but I’ll add it to the list of models to try.

Core ML conversion problems typically happen because the original model does something the converter (coremltools or the PyTorch JIT trace) doesn’t understand, or because there are bugs or limitations in coremltools. It might require modifying the original HF implementation, which is not trivial. Unfortunately, conversion can be a (lengthy) process of trial-and-error.

If anyone has any models they want to see converted to Core ML and are running into problems with it, let me know and I’ll put them on my list. 🙂

---

<div class="post-metadata">

**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlo)%5C
**Post date:** [October 15, 2022, 8:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F10 "2022-10-15T20:47:23Z")

</div>

@Matthijs and everyone playing around with this, I think in addition to limitations in coreml tools, I’ve encountered real-world issues. When running converted models on iPhone 12 Pro (or likely older), there are weird, unpredictable runtime issues (the app crashes).

For example, for real-world devices with variable storage limits, the app running the model can suddenly crash (on some runs). On initial investigation, it seems to be memory-related (when the device is running low on storage or many apps running). I think this could be easily tested by running it on a “fresh” device with plenty of storage. Just something to keep in mind.

I’ve only been spending a few minutes here and there during breaks, so this isn’t as rigorous as I’d like to be.

---

<div class="post-metadata">

**Author:** ![Matthijs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthijs/32/4462_2.png) [@Matthijs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatthijs)%5C
**Post date:** [October 17, 2022, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F11 "2022-10-17T08:01:38Z")

</div>

> [@mlo](#):
>
> When running converted models on iPhone 12 Pro (or likely older), there are weird, unpredictable runtime issues (the app crashes).

It is very likely this is due to running out of memory indeed. Most of the models in Transformers are rather large and so they need a lot of RAM. When the app tries to use more RAM than is available, iOS will terminate it. I’m also not entirely sure that Core ML itself has no memory issues, but there’s nothing we can do about that, unfortunately.

---

<div class="post-metadata">

**Author:** ![Horou](https://avatars.discourse-cdn.com/v4/letter/h/e19adc/32.png) [@Horou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHorou)%5C
**Post date:** [November 9, 2022, 9:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F12 "2022-11-09T09:33:34Z")

</div>

Hello @Matthijs @mlo, I’ve been playing around Torchscript and coreML translation for large models too, do you have any further progress on topic? Didn’t meed much success within models like pegasus yet

---

<div class="post-metadata">

**Author:** ![Matthijs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matthijs/32/4462_2.png) [@Matthijs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatthijs)%5C
**Post date:** [November 9, 2022, 9:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F13 "2022-11-09T09:51:54Z")

</div>

I’ve been able to make some progress but it’s still not as good as I want it to be. It will take some time to work out all the kinks.

Using [GitHub - huggingface/exporters: Export Hugging Face models to Core ML and TensorFlow Lite](https://github.com/huggingface/exporters) it should be possible to convert the `PegasusForConditionalGeneration` model (haven’t tried `PegasusForCausalLM` yet) but you’ll need to build `coremltools` yourself from the main branch to get all the latest patches.

---

<div class="post-metadata">

**Author:** ![Lecra9](https://avatars.discourse-cdn.com/v4/letter/l/e480ec/32.png) [@Lecra9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLecra9)%5C
**Post date:** [December 11, 2022, 2:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F14 "2022-12-11T14:55:23Z")

</div>

I’m trying to convert BloomZ right now, but I get errors the whole time…  
But maybe there is some progress with it.  
I saw that apple builded an own chain for stable diffusion,  
maybe that is helpful for anyone:

> **[GitHub - apple/ml-stable-diffusion: Stable Diffusion with Core ML on Apple...](https://github.com/apple/ml-stable-diffusion)**
>
> Stable Diffusion with Core ML on Apple Silicon. Contribute to apple/ml-stable-diffusion development by creating an account on GitHub.

I already tried it and it runs very smooth with the generating of picture.  
Maybe anyone of you can find something helpful in their even if it’s an diffusion model

---

<div class="post-metadata">

**Author:** ![Leszekasdfff](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Leszekasdfff](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLeszekasdfff)%5C
**Post date:** [July 15, 2023, 9:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconversion-to-coreml-for-on-device-use%2F13284%2F15 "2023-07-15T09:17:32Z")

</div>

is there a way to convert gpt4all / gpt-j (fine tuned) models for ios use in the \*.mlmodel format?
