# Poster2Plot: Generate Movie/T.V show plot from poster
**URL:**
**Author:**  [@dsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdsr)%5C
**Post date:** [November 21, 2021, 9:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F1 "2021-11-21T21:44:20Z")
### Description
Our team is working on building an image captioning model which can generate a movie/t.v show plot from it’s poster.
The goal of this project is to create an image captioning model using a transformer encoder model like Vision Transformer (ViT) and a transformer decoder language model like GPT-2
### Model(s)
Any vision based encoder and language model decoder would be a good candidate to train the VisionEncoderDecoderModel for image captioning. We are trying the following models first:
- Encoder: [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k)
- Decoder: [gpt2](https://huggingface.co/gpt2)
### Datasets
We are using publicly available IMDb datasets to train the model.
Some examples:
- [https://www.kaggle.com/stefanoleone992/imdb-extensive-dataset](https://www.kaggle.com/stefanoleone992/imdb-extensive-dataset)
- [48K IMDB Movies With Posters | Kaggle](https://www.kaggle.com/rezaunderfit/48k-imdb-movies-with-posters)
### Challenges
The main challenge is to create a good dataset of poster and movie plots. Also it will be interesting to see if the model gives good predictions for non-english movies/tv shows.
### Desired project outcomes
We will create a Streamlit or Gradio app on [🤗 Spaces](https://huggingface.co/spaces) that can predict a movie/t.v show plot from it’s poster.
---
**Author:**  [@dk-crazydiv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdk-crazydiv)%5C
**Post date:** [November 22, 2021, 1:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F2 "2021-11-22T01:21:02Z")
Let’s give it a try.
---
**Author:**  [@dsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdsr)%5C
**Post date:** [November 22, 2021, 7:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F3 "2021-11-22T19:10:00Z")
@dk-crazydiv and I were able to train a `VisionEncoderDecoderModel` to generate movie/t.v show plot from poster. We used `google/vit-base-patch16-224-in21k` encoder and `gpt2` decoder.
We have uploaded the model to 🤗 model hub [poster2plot](https://huggingface.co/deepklarity/poster2plot)
@lewtun Link to the Gradio app on 🤗 Spaces [poster2plot](https://huggingface.co/spaces/deepklarity/poster2plot)
We are still working on improving the model.
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [November 23, 2021, 2:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F4 "2021-11-23T14:50:45Z")
Wow, this is an incredibly cool project and Space that you’ve created - great job! Thank you for taking part in the course event 🙂
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [November 23, 2021, 2:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F5 "2021-11-23T14:56:54Z")
Awesome work @dsr and team!
---
**Author:**  [@dk-crazydiv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdk-crazydiv)%5C
**Post date:** [November 24, 2021, 3:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fposter2plot-generate-movie-t-v-show-plot-from-poster%2F12132%2F6 "2021-11-24T03:38:50Z")
Thank you. It was indeed very fun for us to build and super fun to manually test as well. The course content along with some existing code snippets helped a lot. The ease with which one could build and demo projects like these in HF ecosystem cannot be praised enough. The entire ecosystem cuts down the entire idea to deployment time by at least 10x. With the hub, datasets, abstractions in transformer lib and spaces, we were able to do in a day, what would take probably weeks.