Project Roadmap

Phase 0: Project Foundation (DONE)

• Clone reference repos into Examples/ (git-ignored)
• Create project skeleton under Src/finetune_lab/
• Create README, .gitignore, AGENTS.md
• Create docs/research_notes.md with Bonsai + GGUF LoRA research
• Config dataclasses (LoRAConfig, TrainingConfig, ModelConfig, DataConfig)
• Dataset loader (CSV, JSONL, Parquet, DataFrame, HF Datasets)
• Chat template formatter
• Abstract ModelLoaderBase
• SafetensorsModelLoader (Unsloth wrapper)
• GGUFModelLoader (on-demand dequant + QuantizedLinear)
• LoRAMerger (merge + export)
• LoRATrainer (safetensors + GGUF training paths)
• GGUFExporter (adapter + merged model export)
• CLI entry point (train, export, info subcommands)
• 26 unit tests passing
• Forgejo repo created and pushed

Phase 1: GGUF Loading (IN PROGRESS)

• Implement Q4_0 dequantization kernel (CPU fallback)
• Implement Q8_0 dequantization kernel
• Test GGUF header parsing with a real .gguf file
• Verify tensor name mapping (HF -> GGUF) is correct

Phase 2: Integration

• Verify SafetensorsModelLoader works with Unsloth on actual models
• Verify GGUFModelLoader loads a small GGUF model
• Train a tiny LoRA on both paths (integration tests)

Phase 3: Custom Format

• Define custom model format spec (compatible with Q1_0_g128)
• Implement safetensors -> custom format converter
• Implement custom format -> two loading paths (inference + training)
• Train LoRA from custom format

Phase 4: Advanced Features

• SVDQuant correction LoRA (compensate quantization error before training)
• Support Bonsai 1-bit models for inference
• Support Bonsai ternary models
• Multi-GPU support via Accelerate
• wandb/tensorboard integration

Hardware Notes

Target: RTX 4070 Super (16 GB VRAM), Intel i5-12400F

• GGUF Q3_K_M: ~8-10 GB VRAM for 8B models (with LoRA r=16)
• GGUF Q4_K_M: ~10-12 GB VRAM for 8B models
• Safetensors 4-bit QLoRA: ~12-14 GB VRAM for 8B models