# LLM ingores max\_memory in inference

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-ingores-max-memory-in-inference%2F93190%3E%5C
**Category:** Models\
**Created:** [June 20, 2024, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-ingores-max-memory-in-inference%2F93190 "2024-06-20T12:44:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas245](https://avatars.discourse-cdn.com/v4/letter/t/c37758/32.png) [@Tomas245](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTomas245)%5C
**Post date:** [June 20, 2024, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-ingores-max-memory-in-inference%2F93190%2F1 "2024-06-20T12:44:01Z")

</div>

It seems, like my model ingores max\_memory parametre. It crashes with `torch.cuda.OutOfMemoryError: CUDA out of memory.`. And the problem is that first 3 generations goes well. Just model allocates 25GB, GPU Memory is 80GB, but than it’s just grow. I’ve tried to delete all objects, but it doesn’t work for me.  
UPD: Forgot to mention. In error said, that 79 GB allocated and used, by I’ve specified only 70 in the model.  
Model init code:

```auto
base_model = 'google/flan-t5-xxl'
ckpt = './results/checkpoints_t5_1/checkpoint-4600'
device = 'cuda:3'

tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)

model = AutoModelForSeq2SeqLM.from_pretrained(
    base_model,
    device_map=device,
    max_memory={3:'70GB'},
    trust_remote_code=True,
    torch_dtype=torch.float16,
)

model = PeftModel.from_pretrained(
    model,
    ckpt
)

model.config.use_cache = False
model.eval()
model = torch.compile(model)

```

Generation config and code:

```auto
generation_config = GenerationConfig(
    do_sample=True,
    temperature=0.8,
    top_p=0.75,
    top_k=40,
    num_beams=4,
    max_new_tokens=256,
    stream_output=False,
    model=model,
)
inputs = create_prompt(example)
inputs = inputs.to(device)
top1 = example['input'][0]
gt = example['output']

with torch.inference_mode():
    generation_output = model.generate(
    input_ids=inputs["input_ids"],
    generation_config=generation_config,
    return_dict_in_generate=False,
    output_scores=False,
)

```