# Issue with KOSMOS-2 encoding and decoding
**URL:**
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [January 22, 2024, 1:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F1 "2024-01-22T01:42:07Z")
Hi @ydshieh,
First of all I would like to say thank you for your work on Kosmos-2. I want to using Kosmos to for document AI. I am using trying to convert doclaynet dataset to embeddings. I read your code and research paper and found that first I need to convert bounding box to patch\_XXXX thing. I tried that using `processor` but when I decode input\_ids then I found miss match on the bounding box on the image.
```auto
test = processor(images = [example['image']], text = [text], bboxes = [float_val])
test_decode = processor.decode(test['input_ids'][0])
```
As you can see in the image that bounding boxes were overlapped and all.
Can you help me to figure this out?
Here is the code: [Google Colab](https://colab.research.google.com/drive/1J6xOsx3Kbs5LLKW1gj8D9L3yMLOHv2fc?usp=sharing)
Thank you so much.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 1:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F2 "2024-01-24T13:59:47Z")
Hi @Mit1208
If I understand correctly, the attached contains the bounding boxes obtained via the processing(s) of `Kosmos2Processor`
> processed\_text, entities = processor.post\_process\_generation(test\_decode)
And you want to demonstrate via this image that the bounding boxes are not matching the original input of bounding boxes (where you specified via `set_box` then go through `normalized_box(convert_box`). Is this correct?
I can see your image width/height is set to `1025` (I didn’t verify however). `Kosmos2Processor` splits the image into a 32 x 32 grid. With the original input size `224`, each cell is of siez `7x7`. However, with your image size `1025`, each cell will have size `32x32` which is quite large in document AI I believe.
In order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64). If the differences are all inside this range, I wouldn’t say there is something wrong in the code of `Kosmos2Processor`. In this case, it’s just the limitation of such processing.
If the differences are large than 32 or 64, something is likely to be wrong and I can take a more close look.
If you don’t mind to train a model from scratch, there are some arguments could be changed to modify the default properties of `Kosmos2Processor`. I can share more information if you are donw to this.
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [January 24, 2024, 2:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F3 "2024-01-24T14:25:16Z")
Thank you so much @ydshieh for lookin into this issue. Yes your understanding is correct.
You are right that the image size is 1025. About bounding boxes - labeled bounding box is ranging from 0-1025. I normalized it to 0-1 from `Kosmos2Processor`. I tried resizing image to 224 but found same results.
You mentioned below, how can I do that?
> In order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64).
I think training it from scratch is difficult because I don’t have GPUs. Do you think I can train the model on free Colab?
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 2:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F4 "2024-01-24T14:48:32Z")
It would be helpuful if you can:
- check first what I suggest above
> In order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64). …
For
> I tried resizing image to 224 but found same results.
Could you add the extra cells (from the end of the original notebook) showing this
**However** , I think with 224 document image, the word/line gaps are very smaller too. The ideal situation is that word/line gaps are large enough (compared to the word font size), so a bit off of the bounding bboxes won’t have undesired visual effect.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 2:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F5 "2024-01-24T14:52:27Z")
> I think training it from scratch is difficult because I don’t have GPUs. Do you think I can train the model on free Colab?
First, this model is not trained on documents, I would doubt if it will work well (even with finetuning).
train the model from scratch on free Colab → it probably could run but will be quite slow (data processing, training speed etc.)
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [January 24, 2024, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F6 "2024-01-24T15:04:39Z")
I have added 224\*224 resizing code and output, it’s the same as the normal image. Can you see if my bbox conversion from (x, y, w, h) to [x, y, x+w, y+h] is correct and do I need to repeat that step after decoding from `Kosmos2Processor`?
It seems I am very close but missing something small.
I just wanted to see how this model reacts to the document compared to LayoutLMv3.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 5:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F7 "2024-01-24T17:04:55Z")
Hi , thanks. I see you post the desired outputs. Looking both visually, I think it’s really just the limitation of `Kosmos2Processor` (which using cells in a 32x32 grid), which works well in a common images with smaller size. For documents, the cell approaches isn’t good as we need higher precision of the locations.
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [January 24, 2024, 5:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F8 "2024-01-24T17:32:22Z")
Hi, is there a workaround for this like, creating more patches to handle this? I don’t have a lot of experience other than fine-tuning models but I can give it a shot. Can you refer to some documentation or tutorial which I can follow to learn and overcome this issue?
Thanks.
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 6:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F9 "2024-01-24T18:09:21Z")
Yes, you can specify `num_patch_index_tokens` when creating `Kosmos2Processor`. (I think it works when we use `from_pretrained` and passing `num_patch_index_tokens`)
And while doing process(es), we need to pass `num_patches_per_side`.
(see `clean_text_and_extract_entities_with_bboxes`)
(`post_process_generation` doesn’t have this argument, which I should probably update this method)
---
**Author:**  [@ydshieh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fydshieh)%5C
**Post date:** [January 24, 2024, 6:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F10 "2024-01-24T18:09:33Z")
Let me know how it goes!
---
**Author:**  [@Mit1208](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMit1208)%5C
**Post date:** [January 26, 2024, 12:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F11 "2024-01-26T12:21:32Z")
Thanks, sure I will keep you posted.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [January 28, 2024, 7:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-with-kosmos-2-encoding-and-decoding%2F70019%2F12 "2024-01-28T19:01:00Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.