# Converting CLIP to CoreML

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 8, 2023, 7:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345 "2023-02-08T19:45:43Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![alkibijad](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@alkibijad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falkibijad)%5C
**Post date:** [February 8, 2023, 7:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F1 "2023-02-08T19:45:43Z")

</div>

Hi, I’d like to convert CLIP model to CoreML but got some errors. Has anyone managed to do this? If someone can help me go in the right direction it would be great.

I thought of first converting the vision model. Here’s my code so far:

```auto
import coremltools as ct
import torch
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained(model_version)
visual_model = model.vision_model
visual_model.eval()

# Trace the model with random data.
example_input_image = torch.rand(1, 3, 224, 224)

traced_model = torch.jit.trace(visual_model, example_input_image)
out = traced_model(example_input_image)

```

There’s a couple of issues:

1. RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module’s inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.

2. Warning: [site-packages/transformers/models/clip/modeling\_clip.py:222](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/clip_env/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:222): TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if attn\_weights.size() != (bsz \* self.num\_heads, tgt\_len, src\_len):

3. Warning: [/site-packages/transformers/models/clip/modeling\_clip.py:262](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/clip_env/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py:262): TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if attn\_output.size() != (bsz \* self.num\_heads, tgt\_len, self.head\_dim):

---

<div class="post-metadata">

**Author:** ![radames](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/radames/32/28246_2.png) [@radames](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fradames)%5C
**Post date:** [February 8, 2023, 9:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F2 "2023-02-08T21:04:32Z")

</div>

I’ll tag @pcuenq here

---

<div class="post-metadata">

**Author:** ![pcuenq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pcuenq/32/32135_2.png) [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [February 8, 2023, 11:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F3 "2023-02-08T23:57:47Z")

</div>

Thanks @radames! And hello, @alkibijad 🙂

Regarding the first issue, you might avoid getting a `dict` in the output by passing `return_dict = False` to the invocation. I’m not sure this will be enough, as I think the default for the model is to return a `tuple`, see [the documentation here](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPVisionModel.forward.return_dict).

The other warnings are frequent when tracing or compiling models. They are usually fine, as long as you use the model (in production) with inputs of the same shape as the one you used for tracing. The warning means that if you use a different shape, then the code path might be different and the traced code may no longer be correct.

I hope these are enough to get you started, please let us know how it goes!

---

<div class="post-metadata">

**Author:** ![alkibijad](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@alkibijad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falkibijad)%5C
**Post date:** [February 15, 2023, 1:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F4 "2023-02-15T13:20:17Z")

</div>

Thanks for the initial guidance @pcuenq & @radames ! I definitely needed to read the docs a bit more, so I went through both [CLIP’s docs](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPVisionModel.forward.return_dict) and the [CoreML docs](https://coremltools.readme.io/docs/pytorch-conversion).

I’ve managed to:

- trace the model. The two warnings are still present, but shouldn’t have any impact.
- Convert the model to CoreML
- Run inference on the CoreML model

The standard and traced model produce exactly the same output, as they should. But the converted **CoreML model produces slightly different** output.

The conversion to CoreML yields no errors and warnings:

```auto
Converting PyTorch Frontend ==> MIL Ops: 100%|█████████▉| 696/697 [00:00<00:00, 4630.25 ops/s]
Running MIL Common passes: 100%|██████████| 40/40 [00:00<00:00, 96.36 passes/s]
Running MIL Clean up passes: 100%|██████████| 11/11 [00:00<00:00, 56.34 passes/s]
Translating MIL ==> NeuralNetwork Ops: 100%|██████████| 838/838 [00:54<00:00, 15.37 ops/s] 

```

When I pass the exact same image through the models, the CoreML has different results. Here’s how my code looks (simplified a bit):

```auto
# Load CLIP
processor = CLIPProcessor.from_pretrained(model_version)
model_pt = CLIPModel.from_pretrained(model_version)
model_pt.eval()

# Trace
# wrapped_model -> wrapped CLIPModel so that forward() function returns get_image_features()
example_input = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(model_pt, example_input)

# Convert traced model to CoreML
model_coreml = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input.shape)]
)

# Inference through all 3 models. Convert to numpy for easier comparison
image = ... # Load real image from path
processed_image = processor(text= None, images=[image], return_tensors="pt", padding=True)

res_pt = model_pt.get_image_features(processed_image).numpy()
res_traced = model_traced(processed_image).numpy()
res_coreml = model_coreml.predict({'input_image': processed_image.numpy()})['output_name']

# Compare outputs
print(np.array_equal(res_pt, res_traced)) # True -> standard and traced model produce the same results
print(np.array_equal(res_pt, res_coreml)) # False -> different output

# How close are the outputs?
print(np.allclose(res_pt, res_coreml, atol=1e-5)) # False
print(np.allclose(res_pt, res_coreml, atol=1e-2)) # False
print(np.allclose(res_pt, res_coreml, atol=1e-1)) # True -> Results differ ~0.1

```

Does anyone know what could cause this difference in output? The weights are stored as float32, so there’s no 32 → 16 bit conversion.

---

<div class="post-metadata">

**Author:** ![pcuenq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pcuenq/32/32135_2.png) [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [February 16, 2023, 4:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F5 "2023-02-16T16:58:21Z")

</div>

Hi @alkibijad! Glad to know that you are making progress 🙂

Regarding your observation about the outputs, you are right: it is normal that the results are not numerically identical to the ones from PyTorch. Even if the weights are stored in 32-bit format, it doesn’t mean that the model has been converted using 32-bit precision, or runs with it.

The default conversion procedure is not guaranteed to happen in float-32, as described [in this documentation](https://coremltools.readme.io/docs/typed-execution#choosing-the-appropriate-precision). Furthermore, execution precision depends on the hardware you **run** your model on. Another factor that also adds some confusion is the conversion _format_ (the legacy “Neural Network” format, vs the newer “ML Program”). If you convert to Neural Network, then inference precision will be 32-bit when the model (or portions of it) run on CPU, but 16-bit on GPU and Neural Engine. Using ML Program, you can also run 32-bit precision on GPU (but not on Neural Engine).

You can force conversion to happen in 32-bit mode. In ML-Program mode, you can even convert most of your operations using 16-bit precision but preserve some of them in 32-bit mode.

To verify the correctness of the conversion, you can first run the result on CPU, selecting the appropriate compute units, and measure the error with respect to the outputs from the original model, but without expecting numerical equivalence. [This article](https://coremltools.readme.io/docs/typed-execution-example) suggests a signal-to-noise metric to measure the difference. Then you can run predictions incorporating GPU and/or NE, and measure the quality again. It’s usually ok to use conversion defaults and run in 16-bit mode on all eligible devices, but depending on your project (and the model) you might need to override some settings or resort to more exotic things such as per-op precision specification using typed tensors.

Let us know how it goes 🙂

---

<div class="post-metadata">

**Author:** ![alkibijad](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@alkibijad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falkibijad)%5C
**Post date:** [February 28, 2023, 8:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F6 "2023-02-28T20:41:42Z")

</div>

Thank you @pcuenq it’s nice of you to detail the answer that much! Much useful information in it.

The difference in the output has no impact in my use case, so I can say that the CLIP model can be successfully converted to CoreML 🙌

---

<div class="post-metadata">

**Author:** ![wvandergrift](https://avatars.discourse-cdn.com/v4/letter/w/47e85d/32.png) [@wvandergrift](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwvandergrift)%5C
**Post date:** [April 7, 2023, 11:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F7 "2023-04-07T23:17:13Z")

</div>

@alkibijad Thanks for posting your findings while you went through the conversion process!

Would you be able to post a snippet of the finished code that you were able to use to do the conversion to coreml? I’m attempting to do the same and have ran into a few issues.

For example, if I try converting the model using the code below, I get the following error:

```auto
# Load CLIP
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
model_pt = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
model_pt.eval()

# Trace
example_input = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(model_pt, example_input)

# Convert traced model to CoreML
model_coreml = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input.shape)]
)

```

Error:

> ValueError: You have to specify pixel\_values

---

<div class="post-metadata">

**Author:** ![wvandergrift](https://avatars.discourse-cdn.com/v4/letter/w/47e85d/32.png) [@wvandergrift](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwvandergrift)%5C
**Post date:** [April 8, 2023, 3:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F8 "2023-04-08T03:09:46Z")

</div>

Made some progress. If I run the below code I can get through the first step.

```auto
Converting PyTorch Frontend ==> MIL Ops: 100%|█████████▉| 1379/1380 [00:01<00:00, 1155.22 ops/s]

```

```auto
# Load CLIP
model_pt = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
visual_model = model_pt.vision_model
visual_model.eval()

example_input_image = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(visual_model, example_input_image, strict=False)

# Convert the traced model to CoreML
coreml_model = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input_image.shape)]
)

```

Unfortunately, I get the following error:

> RuntimeError: PyTorch convert function for op ‘dictconstruct’ not implemented.

---

<div class="post-metadata">

**Author:** ![adilahmed3](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@adilahmed3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadilahmed3)%5C
**Post date:** [April 20, 2023, 6:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F9 "2023-04-20T06:05:11Z")

</div>

I have this same error. Did you figure out how to fix it?

---

<div class="post-metadata">

**Author:** ![alkibijad](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@alkibijad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falkibijad)%5C
**Post date:** [April 28, 2023, 12:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F12 "2023-04-28T12:38:17Z")

</div>

Sorry for late reply. I had issues with converting the vision model only. The hack/solution was to trace the entire CLIP model with only image as input.

Also, if you don’t need the output of full CLIP model (which is a dictionary/tuple I think), you can wrap CLIP into your own model, whose `forward pass` calls `clip_model.get_image_features(x)`. That way you’re sure to not have any dictionaries that could cause conversion issues.

---

<div class="post-metadata">

**Author:** ![dmayboroda](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dmayboroda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdmayboroda)%5C
**Post date:** [June 6, 2023, 7:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F13 "2023-06-06T19:14:50Z")

</div>

Hey  
Did you fix the issue?  
Cause i faced the same problem  
Thank you

---

<div class="post-metadata">

**Author:** ![dmayboroda](https://avatars.discourse-cdn.com/v4/letter/d/c89c15/32.png) [@dmayboroda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdmayboroda)%5C
**Post date:** [July 12, 2023, 8:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F15 "2023-07-12T08:15:38Z")

</div>

Thank you so much

---

<div class="post-metadata">

**Author:** ![jeremyface](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jeremyface/32/17709_2.png) [@jeremyface](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjeremyface)%5C
**Post date:** [July 18, 2023, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F17 "2023-07-18T09:30:53Z")

</div>

I found a solution. I entered the source code file of CLIPVisionTransformer and located the “forward” function. I forcefully set the “return\_dict” variable to False.

---

<div class="post-metadata">

**Author:** ![nathancooperjones](https://avatars.discourse-cdn.com/v4/letter/n/f17d59/32.png) [@nathancooperjones](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnathancooperjones)%5C
**Post date:** [December 12, 2023, 3:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-clip-to-coreml%2F31345%2F18 "2023-12-12T15:08:48Z")

</div>

I know this thread is a bit old at this point, but in case you are like me and trying to do this today and struggling to run these code snippets without error, this solution code snippet is _so close_. A super easy way to accomplish the

`# wrapped_model -> wrapped CLIPModel so that forward() function returns get_image_features()`

line in this code snippet is to hack the `forward` method so that it always calls the `get_image_features` method instead, like so:

```python
model_pt.forward = lambda *args, **kwargs: model_pt.get_image_features(*args,**kwargs)

```

If you add that, this entire code snippet posted in the solution should work properly.

In addition, if you also want to see how the text encoder could be converted to CoreML, I found this repo to be super helpful: [GitHub - mazzzystar/Queryable: Run OpenAI's CLIP model on iOS to search photos.](https://github.com/mazzzystar/Queryable/tree/main)
