likhithv/meta-sdk-baseline

Meta SDK Baseline — LoRA Adapter

LoRA adapter for Qwen/Qwen3.5-4B fine-tuned on 1,209 chunk-based training samples generated by Meta's Synthetic Data Kit from the same financial and medical source documents.

This is the Meta SDK baseline model from the paper "Knowledge Graph-Guided Fine-Tuning Data Generation: A Rigorous Benchmark" — the industry-standard chunk-based approach used as the comparison point against KnowledgeMesh.

Benchmark Results

Evaluated by Gemini 2.5 Flash pointwise judge (1–5 scale, 4 dimensions):

Eval Set	Base	This Model	KM Full	Delta (KM − this)
Primary (n=473, KM-generated)	1.79	1.93	2.47	+0.54
Independent (n=955, Gemini-generated)	1.96	2.17	2.90	+0.72

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3.5-4B"
adapter_id = "likhithv/meta-sdk-baseline"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [{"role": "user", "content": "What were Apple's total net sales in 2023?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs.to(model.device), max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Training Details

Parameter	Value
Base model	Qwen/Qwen3.5-4B (4-bit quantized via bitsandbytes)
Fine-tuning method	LoRA (rank=16, alpha=16)
Training samples	1,209 (chunk-based QA, Meta Synthetic Data Kit)
Epochs	3
Learning rate	2e-4
Effective batch size	8
Hardware	Kaggle T4 GPU (16 GB)
Domains	Financial (Apple 10-K 2023), Medical (PubMed abstracts)

Eval Datasets

• likhithv/knowledgemesh-benchmark-eval

Compared Models

• likhithv/km-full-model — KnowledgeMesh, 4,361 KG-guided samples (+0.72 on independent eval)

Citation

@misc{knowledgemesh2026,
  title={Knowledge Graph-Guided Fine-Tuning Data Generation: A Rigorous Benchmark},
  author={Likhith V},
  year={2026},
  howpublished={https://huggingface.co/likhithv/meta-sdk-baseline}
}