# Filter Large Dataset Entry by Entry
**URL:**
**Author:**  [@QiyaoWei](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FQiyaoWei)%5C
**Post date:** [August 27, 2024, 8:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F1 "2024-08-27T20:31:16Z")
I am not sure if this is a new feature, but I wanted to post this problem here, and hear if others have ways of optimizing and speeding up this process.
Letâs say I have a really large dataset that I cannot load into memory. At this point, I am only aware of `streaming=True` to load the dataset. Now, the dataset consists of many tables. Ideally, I would want to have some simple filtering criterion, such that I only see the âgoodâ tables. Here is an example of what the code might look like:
```auto
dataset = load_dataset(
"really-large-dataset",
streaming=True
)
# And let's say we process the dataset bit by bit because we want intermediate results
dataset = islice(dataset, 10000)
# Define a function to filter the data
def filter_function(table):
if some_condition:
return True
else:
return False
# Use the filter function on your dataset
filtered_dataset = (ex for ex in dataset if filter_function(ex))
```
And then I work on the processed dataset, which would be magnitudes faster than working on the original. I would love to hear if the problem setup + solution makes sense to people, and if anyone has suggestions!
---
**Author:**  [@mikehemberger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikehemberger)%5C
**Post date:** [August 28, 2024, 1:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F2 "2024-08-28T13:53:28Z")
Hi @QiyaoWei ,
Donât know about streaming datasets but filter() looks like what your looking for?!
> **[Process](https://huggingface.co/docs/datasets/process)**
>
> Weâre on a journey to advance and democratize artificial intelligence through open source and open science.
Best,
M
---
**Author:**  [@QiyaoWei](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FQiyaoWei)%5C
**Post date:** [August 28, 2024, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F3 "2024-08-28T15:30:40Z")
Hey @mikehemberger, I did look at filter(), but from what I understand, filter() applies specifically to a large table and filters the table row-by-row, not to a set of tables and filter table-by-table. Is there a functionality here that I am missing out on?
---
**Author:**  [@mikehemberger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikehemberger)%5C
**Post date:** [August 28, 2024, 5:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F4 "2024-08-28T17:50:23Z")
Hey again, seems like I misinterpreted your question / code example. Could you elaborate on what the entries of your sliced dataset consist of?
Best,
M
---
**Author:**  [@QiyaoWei](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FQiyaoWei)%5C
**Post date:** [August 28, 2024, 6:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F5 "2024-08-28T18:06:28Z")
Hey @mikehemberger, sure thing! When streaming=True, the dataset object will become an IterableDataset object that can be used by next(iter(dataset)). In my case, next(iter(dataset)) = pd.Dataframe(), so thatâs why I was not sure whether the filter() function could be applied to an iterator of dataframes, if that makes sense
---
**Author:**  [@mikehemberger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikehemberger)%5C
**Post date:** [August 28, 2024, 6:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F6 "2024-08-28T18:17:47Z")
From what I gather a single entry of your dataset is a dataframe, correct?
Would it be feasible then to check each entry of your data in respect to your condition and then filter on the resulting Boolean array?
---
**Author:**  [@QiyaoWei](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FQiyaoWei)%5C
**Post date:** [August 28, 2024, 7:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F7 "2024-08-28T19:00:21Z")
Hey @mikehemberger, yes and yes. That is exactly what I did in the last line of the code I provided with the iterator using the filter\_function. I was just wondering if I did it in the best way, but looks like that is the best solution
---
**Author:**  [@mikehemberger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmikehemberger)%5C
**Post date:** [August 28, 2024, 7:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffilter-large-dataset-entry-by-entry%2F104270%2F8 "2024-08-28T19:30:10Z")
Hey @QiyaoWei ,
Maybe not the best but if it works in a reasonable amount of time still great! đ
Best,
M