B5: pak data-area walker — 608 DataTables / 5,681 rows (40x B4 haul, item names+descs+stats) #8
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feature/dune-extract-b5-data-area-walker"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
User directive: "I need item names, descriptions, and stats as well, for everything that has it, if we dont have the ability to extract that data, make it so we can."
Done. 608 DataTables / 5,681 rows recovered end-to-end (vs B4's 14 / 159 — a 40× haul). Real per-item tables now decode: emotes with names + descriptions + durability + stack sizes, melee weapons, augments, weapon mods, buildables, edibles, perishables, fillables, customizations.
How
The B1 EncodedPakEntries decoder caps at ~70 % in the average pak and 39 % in Systems.pak (where the bulk of item DataTables live) — the B1-FU virtual-vs-physical gap. Fix: stop walking the index entirely and walk the pak data area directly. Every file is
(FPakEntry_header, data)contiguously; the header reports its own size + method so we stride file→file.pak_extract.walk_data_area— lock-on then exact strides. Resync ladder for Funcom's MiB-aligned chunks (one 34 MiB gap observed in Systems.pak): 2 MiB aligned (1 M / 64 K / 4 K) → 32 KiB bounded linear → unbounded 1-MiB-aligned scan to EOD withOffset==0pre-filter (~1 ms per gap regardless of pak size). 8 KiB header window covers ~500-block entries.pak_extract.read_data_area_entry— mirror ofread_uassetforDataAreaEntry; same AES + Oodle pipeline.enrichment.enrich_with_row_datarewritten to drive offwalk_data_area. Size pre-filter (200..50K) skips the bulk of.uexp/.ubulksiblings without decompression.Coverage (data-area walk vs FDI declared)
Full extract
Sample row (real item data)
Names + descriptions arrive as localization keys (
UI/Emote_Title_WormSign) referencing rows inST_Localization_UI.uasset. Resolving them to actual English text is Phase C enrichment work; the row shape itself is now complete and structurally correct.Perf
~10 min for the full 37-pak extract. DuneSandbox + Systems each take a few minutes (many aligned-boundary resyncs). Acceptable for one-time-per-update extraction; parallelism is Phase E hygiene.
No regression
B1, B2, B3 probes still pass identically. The B1 EncodedPakEntries decoder isn't removed — it's just not the primary source for row enrichment anymore.
Test plan
User directive: "I need item names, descriptions, and stats as well, for everything that has it, if we dont have the ability to extract that data, make it so we can." The B1 EncodedPakEntries decoder caps at ~70% of files in the average pak and 39% in Systems.pak (where the bulk of item DataTables live) due to the virtual-vs-physical mapping gap. Fix: stop walking the index entirely and walk the pak DATA AREA directly. Every file is laid out as (FPakEntry_header, data) contiguously, and the header reports its own size + compression method, so we stride from one file to the next. Implementation: * pak_extract.walk_data_area(pak_path, ioff_raw) yields a DataAreaEntry per file in pak-order. Lock-on via forward scan, then exact strides. Funcom aligns chunks at 1MiB/64KiB/4KiB boundaries with sometimes-tens-of-MiB gaps (Systems.pak has a 34 MiB gap between chunks). Resync via: - 2 MiB aligned-boundary search (1MiB -> 64KiB -> 4KiB) - 32 KiB bounded linear scan - Unbounded 1-MiB-aligned scan to end-of-data with Offset==0 pre-filter (~1ms per gap, regardless of pak size) Multi-block header read window is 8 KiB (covers ~500 blocks). * pak_extract.read_data_area_entry — mirror of read_uasset but takes a DataAreaEntry. Same AES + Oodle pipeline as B2. * enrichment.enrich_with_row_data rewritten to drive off walk_data_area. Size pre-filter (200..50_000) skips the bulk of .uexp/.ubulk siblings without decompression. Same physically- adjacent .uasset/.uexp pairing. Coverage results (data-area walk vs FDI declared count): Abilities 812 / 812 100% Consumables 96 / 96 100% Weapons 3,580 / 3,580 100% Vehicles 11,242 / 11,242 100% Systems 19,274 / 19,293 99.9% DuneSandbox 16,406 / 16,463 99.7% Full extract: 608 DataTables / 5,681 rows recovered (vs B4's 14 / 159). Real per-item tables now flow: DT_ItemTableBuildables 571 rows DT_ItemTableCustomizations 368 DT_ItemTable_Augments 123 DT_ItemTableWeaponMods 101 DT_MeleeWeaponItemTable 83 <-- melee weapons DT_ItemTablePerishables 72 DT_ItemTableEdibles 62 DT_ItemTableFillables 58 DT_BaseItems_Emotes 33 + ~600 more Sample row (DT_BaseItems_Emotes / Emote_WormSign): Icon = /Game/Dune/.../T_UI_IconEmoteWormsign_D StaticData = { Name=`UI/Emote_Title_WormSign`, ShortDesc=`UI/Emote_Description_WormSign`, LongDesc=`UI/Emote_Description_WormSign`, ... } StackAndDurability = { MaxStackSize=1, MaxDurability=0.0, ... } bIsDeprecated = ✗ Names + descriptions arrive as localization keys (UI/Emote_Title_WormSign) that reference rows in ST_Localization_UI.uasset; resolving them to actual English text is Phase C enrichment work. The row shape itself is complete. Perf: ~10min for the full 37-pak extract. DuneSandbox + Systems each take a few minutes (many aligned-boundary resyncs). Acceptable for one-time-per-update extraction; parallelism is Phase E hygiene. No regression — B1/B2/B3 probes still pass identically. The B1 decoder isn't removed; it's just not the primary source for row enrichment anymore.