# Build a Twitter topic extractor
**URL:**
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [November 10, 2021, 4:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F1 "2021-11-10T16:45:52Z")
> š Please read the topic category [description](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fabout-the-course-projects-category%2F11510) to understand what this is all about
### Description
Twitter classifies trending tweets according to predefined set of topics like āData scienceā, āHip hopā, āSportā etc. However, their algorithm appears to often get confused by certain keywords in the tweet or the content of the image (see [here](https://twitter.com/Michael_D_Moor/status/1457425576583647233?s=20) for some funny examples).
The goal of this project is to explore whether itās possible to create a better topic extractor, or at least one that is more targeted for a smaller set of domains. Thereās several ways to approach to task:
- Frame it as a multiclass classification problem
- Frame it as an unsupervised clustering problem, and combine this with techniques like [UMAP](https://umap-learn.readthedocs.io/en/latest/clustering.html) and/or HDBSCAN
### Model(s)
Since Tweets are short in length, picking one of the `sentence-transformers` [models on the Hub](https://huggingface.co/sentence-transformers) is likely a good place to start.
### Datasets
There are various Twitter datasets on the Hub and hereās a few examples to start with:
- [tweet\_eval](https://huggingface.co/datasets/tweet_eval)
- [sentiment140](https://huggingface.co/datasets/sentiment140)
- [emotion](https://huggingface.co/datasets/emotion)
### Challenges
If you take the unsupervised learning approach, be warned that thereās no ācorrectā answer and you will have to experiment with the dimensionality reduction / cluster algorithms to get meaningful clusters.
### Desired project outcomes
- Create a Streamlit or Gradio app on [š¤ Spaces](https://huggingface.co/spaces) that either can automatically classify a Tweet according to a topic, or visualises the 2D projection of the embeddings and colours them by topic.
- Donāt forget to push all your models and datasets to the Hub so others can build on them!
### Additional resources
- [https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668](https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668)
- [https://huggingface.co/spaces/edugp/embedding-lenses](https://huggingface.co/spaces/edugp/embedding-lenses) (useful for UMAP inspiration)
### Discord channel
To chat and organise with other people interested in this project, head over to our [Discord](http://xwo7x.mjt.lu/lnk/CAAAAkdkZwAAAAAAAAAAALG-MTEAAYCr4H8AAAAAAAyFyQBhjpnimM4rcE5DT5ibCUtRT96tWwAMh8Y/15/U6BYNMNDksyyGG4jgtzPKQ/aHR0cHM6Ly9odWdnaW5nZmFjZS5jby9qb2luL2Rpc2NvcmQ) and:
- Follow the instructions on the `#join-course` channel
- Join the `#twitter-topic-extractor` channel
Just make sure you comment here to indicate that youāll be contributing to this project š
---
**Author:**  [@wilmerags](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwilmerags)%5C
**Post date:** [November 15, 2021, 8:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F2 "2021-11-15T20:11:26Z")
Iād like to work on this project! š
---
**Author:**  [@rg089](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frg089)%5C
**Post date:** [November 16, 2021, 9:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F3 "2021-11-16T21:53:40Z")
Iād like to join this project!
---
**Author:**  [@alghar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falghar)%5C
**Post date:** [November 17, 2021, 12:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F4 "2021-11-17T00:27:02Z")
I am interested as well. What is your timeframe?
---
**Author:**  [@wilmerags](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwilmerags)%5C
**Post date:** [November 17, 2021, 1:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F5 "2021-11-17T01:27:37Z")
I think that the project needs to be available for reviewal in November 24th.
---
**Author:**  [@alghar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falghar)%5C
**Post date:** [November 17, 2021, 5:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F6 "2021-11-17T05:29:52Z")
Hello Wilmer,
Would you like to have a zoom chat tomorrow (after the class)?
I am in Calif (-3 hours from you)
Al
---
**Author:**  [@wilmerags](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwilmerags)%5C
**Post date:** [November 19, 2021, 2:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F7 "2021-11-19T02:53:00Z")
Oh sorry didnāt see this, Yes, letās catch up in discord š
---
**Author:**  [@anupama4821](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanupama4821)%5C
**Post date:** [March 7, 2023, 1:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbuild-a-twitter-topic-extractor%2F11571%2F8 "2023-03-07T13:43:00Z")
Interested