# Add new column to a dataset
**URL:**
**Author:**  [@luka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluka)%5C
**Post date:** [November 22, 2021, 10:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F1 "2021-11-22T10:54:10Z")
In the dataset I have 5000000 rows, I would like to add a column called āembeddingsā to my dataset.
`dataset = dataset.add_column('embeddings', embeddings)`
The variable _embeddings_ is a numpy memmap array of size (5000000, 512).
But I get this error:
> ArrowInvalidTraceback (most recent call last)
> in
> ----\> 1 dataset = dataset.add\_column(āembeddingsā, embeddings)
>
> /opt/conda/lib/python3.8/site-packages/datasets/arrow\_dataset.py in wrapper(\*args, \*\*kwargs)
> 486 }
> 487 # apply actual function
> ā 488 out: Union[āDatasetā, āDatasetDictā] = func(self, \*args, \*\*kwargs)
> 489 datasets: List[āDatasetā] = list(out.values()) if isinstance(out, dict) else [out]
> 490 # re-apply format to the output
>
> /opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py in wrapper(\*args, \*\*kwargs)
> 404 # Call actual function
> 405
> ā 406 out = func(self, \*args, \*\*kwargs)
> 407
> 408 # Update fingerprint of in-place transforms + update in-place history of transforms
>
> /opt/conda/lib/python3.8/site-packages/datasets/arrow\_dataset.py in add\_column(self, name, column, new\_fingerprint)
> 3346 :class:`Dataset`
> 3347 āā"
> ā 3348 column\_table = InMemoryTable.from\_pydict({name: column})
> 3349 # Concatenate tables horizontally
> 3350 table = ConcatenationTable.from\_tables([self.\_data, column\_table], axis=1)
>
> /opt/conda/lib/python3.8/site-packages/datasets/table.py in from\_pydict(cls, \*args, \*\*kwargs)
> 367 @classmethod
> 368 def from\_pydict(cls, \*args, \*\*kwargs):
> ā 369 return cls(pa.Table.from\_pydict(\*args, \*\*kwargs))
> 370
> 371 @inject\_arrow\_table\_documentation(pa.Table.from\_batches)
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from\_pydict()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.\_from\_pydict()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.asarray()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.\_ndarray\_to\_array()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check\_status()
>
> ArrowInvalid: only handle 1-dimensional arrays
How can I solve?
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [November 22, 2021, 11:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F2 "2021-11-22T11:31:30Z")
Hi,
it should work if you use `concatenate_datasets` instead:
```python
import datasets
dset_embed = datasets.Dataset.from_dict({"embeddings": embeddings})
dset_concat = datasets.concatenate_datasets([dset, dset_embed], axis=1)
```
---
**Author:**  [@luka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluka)%5C
**Post date:** [November 22, 2021, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F3 "2021-11-22T12:00:37Z")
I have also the problem that the array āembeddingsā does not fit the RAM, so I suspect that the method you are proposing is not actually feaseble.
Iāve tried anyway and I got this error:
> ArrowInvalidTraceback (most recent call last)
> in
> 1 import datasets
> ----\> 2 dataset\_embed = datasets.Dataset.from\_dict({āembeddingsā: embeddings})
>
> /opt/conda/lib/python3.8/site-packages/datasets/arrow\_dataset.py in from\_dict(cls, mapping, features, info, split)
> 783 for col, data in mapping.items()
> 784 }
> ā 785 pa\_table = InMemoryTable.from\_pydict(mapping=mapping)
> 786 return cls(pa\_table, info=info, split=split)
> 787
>
> /opt/conda/lib/python3.8/site-packages/datasets/table.py in from\_pydict(cls, \*args, \*\*kwargs)
> 367 @classmethod
> 368 def from\_pydict(cls, \*args, \*\*kwargs):
> ā 369 return cls(pa.Table.from\_pydict(\*args, \*\*kwargs))
> 370
> 371 @inject\_arrow\_table\_documentation(pa.Table.from\_batches)
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from\_pydict()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.\_from\_pydict()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.asarray()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.\_handle\_arrow\_array\_protocol()
>
> /opt/conda/lib/python3.8/site-packages/datasets/arrow\_writer.py in **arrow\_array** (self, type)
> 111 out = pa.ExtensionArray.from\_storage(type, storage)
> 112 elif isinstance(self.data, np.ndarray):
> ā 113 out = numpy\_to\_pyarrow\_listarray(self.data)
> 114 if type is not None:
> 115 out = out.cast(type)
>
> /opt/conda/lib/python3.8/site-packages/datasets/features/features.py in numpy\_to\_pyarrow\_listarray(arr, type)
> 921 n\_offsets = reduce(mul, arr.shape[: arr.ndim - i - 1], 1)
> 922 step\_offsets = arr.shape[arr.ndim - i - 1]
> ā 923 offsets = pa.array(np.arange(n\_offsets + 1) \* step\_offsets, type=pa.int32())
> 924 values = pa.ListArray.from\_arrays(offsets, values)
> 925 return values
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.\_ndarray\_to\_array()
>
> /opt/conda/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check\_status()
>
> ArrowInvalid: Integer value 2147483648 not in range: -2147483648 to 2147483647
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [November 25, 2021, 4:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F4 "2021-11-25T16:31:42Z")
Hi ! Not sure if `datasets` is able to convert a memory-mapped NumPy array to an Arrow array without bringing the NumPy array in RAM (I havenāt tested).
However the error youāre getting looks more like an integer precision issue because the array you are passing has more than 2147483648 elements, maybe you could try chunking your list of embeddings, then get a dataset object per chunk, and concatenate them to get the dataset with all the embeddings.
---
**Author:**  [@nadahlberg](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnadahlberg)%5C
**Post date:** [September 5, 2022, 2:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F5 "2022-09-05T14:51:23Z")
I ran into this issue and was able to address it using the chunking suggestion above. The following worked for me. Where I was previously just using convert\_to\_dataset I could now pass the same arguments to ChunkedDataset:
```auto
def convert_to_dataset(texts, labels, tokenizer, max_sequence_length):
inputs = tokenizer(texts, padding="max_length", max_length=max_sequence_length, truncation=True)
inputs['label'] = labels
return Dataset.from_dict(inputs)
class ChunkedDataset(torch.utils.data.Dataset):
def __init__ (self, texts, labels, tokenizer, max_sequence_length=512, chunk_size=20000):
self.chunk_size = chunk_size
self.datasets = []
for i in tqdm(range(len(texts) // chunk_size + 1)):
batch_slice = slice(i * chunk_size, (i + 1) * chunk_size)
if len(texts[batch_slice]):
self.datasets.append(convert_to_dataset(texts[batch_slice], labels[batch_slice], tokenizer, max_sequence_length))
def __len__ (self):
return np.sum([len(x) for x in self.datasets])
def __getitem__ (self, idx):
dataset_idx = idx // self.chunk_size
idx = idx % self.chunk_size
return self.datasets[dataset_idx][idx]
```
---
**Author:**  [@jxm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjxm)%5C
**Post date:** [September 3, 2023, 10:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F6 "2023-09-03T22:50:42Z")
> [@lhoestq](#):
>
> 7483
Ran into this problem too trying to do something like this:
```python
datasets.Dataset.from_dict({"embeds": embeddings })
```
Really annoying limitation. I shouldnāt have to implement chunking myself.
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [September 4, 2023, 8:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F7 "2023-09-04T08:55:58Z")
Feel free to open an issue on github so we can investigate that š
---
**Author:**  [@nishaanthkanna](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnishaanthkanna)%5C
**Post date:** [January 15, 2024, 3:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F8 "2024-01-15T15:46:57Z")
But isnāt this trying to be solved by this old PR? [https://github.com/huggingface/datasets/pull/4800](https://github.com/huggingface/datasets/pull/4800)
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 18, 2024, 12:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-new-column-to-a-dataset%2F12149%2F9 "2024-01-18T12:17:18Z")
Ah yes itās closely related indeed ! Thanks for bringing this PR up.
Iāll try to find some time to dive again into this. IIRC it was mostly a matter of finding the appropriate API to switch from regular to large lists in Arrow