Sponge/finetune-lab
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Integrate gguf library for GGUF dequantization #1

Open
Sponge wants to merge 3 commits from feature/gguf-dequant-improvement into main
pull from: feature/gguf-dequant-improvement
merge into: Sponge:main
Conversation 0 Commits 3 Files changed 5 +796 -220
Sponge commented 2026-09-15 05:40:35 +00:00
Owner
Copy link

Uses gguf library's dequantize() for full quantization type support. Adds CPU fallback and Q1_0 (Bonsai 1-bit) support.

Uses gguf library's dequantize() for full quantization type support. Adds CPU fallback and Q1_0 (Bonsai 1-bit) support.
- Rewrite GGUFDequantizer.dequant_tensor to use gguf library's
  dequantize() when available, providing correct support for all
  quantization types (Q4_0, Q5_0, Q8_0, Q2_K, Q3_K, Q6_K, IQ*, Q1_0/Bonsai)
- Add CPU fallback for F32, F16, Q4_0, Q4_1, Q5_0, Q8_0 via numpy
- Fix BLOCK_SIZE format from int to (weights_per_block, bytes_per_block)
  tuple matching gguf's GGML_QUANT_SIZES
- Add _DTYPE_TO_INT mapping including Bonsai Q1_0 (type 41)
- Add Q1_0 to block size table for Bonsai 1-bit format support
- Fix QuantizedLinear.nn.Module device attribute conflict (_device)
- Fix _load_tensor_data to use correct (block_size, block_bytes) lookup
- Fix _parse_blocks to use centralized BLOCK_SIZE table
- Add 7 tests for dequantization paths (gguf lib + CPU fallback + F16)
- All 33 tests pass
- Add rank_pattern field to LoRAConfig (dict[str, int] | None = None)
- Replace broken type-check logic (isinstance(target_modules[0], dict))
  with proper None check on lora_config.rank_pattern
- target_modules is always list[str], so the old code's dict check
  on list elements was dead code that could never be True
- Add tests for rank_pattern config defaults and custom values
- Restore accidentally-removed use_rslora block
- Fix GGUF header parser: correct GGUFValueType IDs (STRING=8 not 6,
  UINT32=4 not 1, INT32=5 not 2, FLOAT32=6 not 3, BOOL=7 not 4)
  and value format (scalars have no length prefix; only STRING/ARRAY do)
- Add ARRAY metadata value support with _read_array method
- Add _parse_with_reader method using gguf library's GGUFReader for
  more robust parsing (preferred path when gguf is available)
- Fix QuantizedLinear.forward: cast input to weight dtype, cast
  LoRA output back to result dtype for mixed-precision support
- Fix _freeze_base_model call in GGUFModelLoader.get_peft_model so
  only LoRA params are trainable (base model params were all trainable
  because AutoModelForCausalLM.from_config() initializes everything)
- Fix add_lora: remove redundant zeros_ before kaiming_uniform_ init
- Add 10 integration tests (header parsing, dequantization round-trip,
  QuantizedLinear forward/backward with LoRA)
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feature/gguf-dequant-improvement:feature/gguf-dequant-improvement
git switch feature/gguf-dequant-improvement

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff feature/gguf-dequant-improvement
git switch feature/gguf-dequant-improvement
git rebase main
git switch main
git merge --ff-only feature/gguf-dequant-improvement
git switch feature/gguf-dequant-improvement
git rebase main
git switch main
git merge --no-ff feature/gguf-dequant-improvement
git switch main
git merge --squash feature/gguf-dequant-improvement
git switch main
git merge --ff-only feature/gguf-dequant-improvement
git switch main
git merge feature/gguf-dequant-improvement
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Sponge/finetune-lab!1
No description provided.