ayeshaishaq/DriveLMMo1

DriveLMM-o1: A Large Multimodal Model for Autonomous Driving Reasoning

Paper

DriveLMM-o1 is a fine-tuned large multimodal model designed for autonomous driving. Built on InternVL2.5-8B with LoRA-based adaptation, it leverages stitched multiview images to produce step-by-step reasoning. This structured approach enhances both final decision accuracy and interpretability in complex driving tasks like perception, prediction, and planning.

Key Features:

• Multimodal Integration: Combines multiview images for comprehensive scene understanding.
• Step-by-Step Reasoning: Produces detailed intermediate reasoning steps to explain decisions.
• Efficient Adaptation: Utilizes dynamic image patching and LoRA finetuning for high-resolution inputs with minimal extra parameters.
• Performance Gains: Achieves significant improvements in both final answer accuracy and overall reasoning scores compared to previous open-source models.

Performance Comparison:

Model	Risk Assessment Accuracy	Traffic Rule Adherence	Scene Awareness & Object Understanding	Relevance	Missing Details	Overall Reasoning Score	Final Answer Accuracy
GPT-4o (Closed)	71.32	80.72	72.96	76.65	71.43	72.52	57.84
Qwen-2.5-VL-7B	46.44	60.45	51.02	50.15	52.19	51.77	37.81
Ovis1.5-Gemma2-9B	51.34	66.36	54.74	55.72	55.74	55.62	48.85
Mulberry-7B	51.89	63.66	56.68	57.27	57.45	57.65	52.86
LLaVA-CoT	57.62	69.01	60.84	62.72	60.67	61.41	49.27
LlamaV-o1	60.20	73.52	62.67	64.66	63.41	63.13	50.02
InternVL2.5-8B	69.02	78.43	71.52	75.80	70.54	71.62	54.87
DriveLMM-o1 (Ours)	73.01	81.56	75.39	79.42	74.49	75.24	62.36

Usage:

Load the model using the following code snippet:

from transformers import AutoModel, AutoTokenizer
import torch

path = 'ayeshaishaq/DriveLMMo1'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    path,
    trust_remote_code=True,
    use_fast=False
)

For detailed usage instructions and additional configurations, please refer to the OpenGVLab/InternVL2_5-8B repository.

Code: https://github.com/ayesha-ishaq/DriveLMM-o1

Limitations: While DriveLMM-o1 demonstrates strong performance in autonomous driving tasks, it is fine-tuned for domain-specific reasoning. Users may need to further fine-tune or adapt the model for different driving environments.