Phase C: localization + C1 sample re-publish + C2 crafting + C3 loot tables — 27,816 key→English pairs #9
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feature/dune-extract-phase-c"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Goal: complete task C. Closes C1 (re-publish sample catalog), C2 (crafting-tree), C3 (loot-tables) — plus the implicit "names + descriptions in plain English" dependency that followed B5 (user: "I need item names, descriptions, and stats as well, for everything that has it, if we dont have the ability to extract that data, make it so we can").
Prerequisite: multi-block Oodle decompression
ST_Localization_UI.uexpis 8.3 MB / 128 blocks. The earlier single-call Oodle decompress returned-1on multi-block streams. New_decompress_blockswalks the in-pak header'sblocks[]and feeds each to Oodle with UE's default 64 KiB per-block unc size (last block takes the remainder). Catches every multi-block entry the walker yields and lifts the row haul from B5's 608 / 5,681 to 675 / 15,010.Localization resolver (new
localization.py)Parses
ST_Localization_*.uassetUStringTable exports (UObject-None + int32 preamble + TableId FString + NumEntries + (Key, Value)×N).resolve_in_rowsubstitutes English into every row's FText fields (top-level + one level of nested structs — coversStaticData/Descriptionshape).__main__.pyapplies it post-enrich_with_row_data.C1 — re-published sample catalog
DataExtract/ITEMS/regenerated from the shipped tool. Per-category Markdown grew 5–20×.C2 — crafting-tree (
crafting_tree.py)Identifies recipe tables (
DT_Recipe_/DT_Schematic_/DT_Crafting_/DT_Production/DT_FabricatorSettings/DT_NPCRecipesby prefix + field-shape fallback). EmitsITEMS-Crafting-Trees.mdwith normalized{inputs, outputs, station, tier}per recipe. 103 rows / 6 tables. Some cells show—because the underlying ingredient list is nested in opaque structs emitted as_raw_— Phase E hygiene work.C3 — loot-tables (
loot_tables.py)Identifies loot tables (
DT_LootTable_*/DT_Loot_*/DT_ContainerLoot*/DT_NPCLoot*/DT_InventoryItemSets*). EmitsITEMS-Loot-Tables.mdwith normalized{item_refs, weights, tier_gates, tag_gates}per pool entry. 1,782 loot-pool rows / 429 loot-table assets.NumRolls/PercentageChance/Weight/RequiredLootDifficultyLevelranges all surface clean;LootWeightedTablerefs resolve to/Game/...paths.End-to-end run
~11 min wall-time.
DataExtract/ITEMS/grew from ~1.5 MB to ~5.8 MB committed in-repo.Honest scope note
Some struct fields still emit
_raw_(FGameplayTagContainer, FPrimaryAssetId, some recipe ingredient arrays) — Phase E hygiene task widens theuasset_parser._read_valueswitch. C-phase deliverables themselves are all shipped.Test plan
Closes C1/C2/C3 + the implicit "names+descriptions in English" dep that followed B5. Prerequisite fix — multi-block Oodle decompression pak_extract._decompress_blocks walks the in-pak header's blocks[] and feeds each block to Oodle independently (UE default 64 KiB per-block unc, last block takes remainder). ST_Localization_UI.uexp (8.3 MB / 128 blocks) and other large multi-block entries now decode cleanly. Lifts row haul from B5's 608 / 5,681 to 675 / 15,010. Localization resolver (new dune_extract/localization.py) Parses ST_Localization_*.uasset UStringTable exports — wire format is UObject-None + int32 preamble + TableId FString + int32 NumEntries + NumEntries × (Key, Value) FString pairs. build_localization_index scans every ST_* asset and returns {table_id: {key: english}}. Catalog haul: 10 string tables, 27,816 key→English pairs. Tables (with key counts): ST_Localization_Items 9,887 ST_Localization_UI 5,833 ST_Localization_Buildings 3,995 ST_Localization_Lore_Pickups_Contracts 3,776 ST_Localization_Progression 1,934 ST_Localization_NPCs_World 986 ST_Localization_Abilities 653 ST_Localization_Dunipedia 514 ST_Localization_Communinet 232 ST_Localization_General 6 resolve_in_row substitutes English into every row dict's FText fields (top-level + one level of nested structs — covers StaticData / Description shape). __main__.py runs it after enrich_with_row_data. C1 — re-publish sample catalog (DataExtract/ITEMS/) Regenerated from the shipped tool path with row data + resolved English. Per-category Markdown files grew 5-20x. Sample (Emote_WormSign in DT_BaseItems_Emotes): Name = "Worm Sign" ShortDesc = "Warn of an approaching sandworm" LongDesc = "Warn of an approaching sandworm" StackAndDurability = { MaxStackSize=1, ... } bIsDeprecated = ✗ C2 — crafting-tree (new dune_extract/crafting_tree.py) Identifies recipe tables (DT_Recipe / DT_Schematic / DT_Crafting / DT_Production / DT_FabricatorSettings / DT_NPCRecipes by prefix + field-shape fallback). Normalizes per recipe into {inputs, outputs, station, tier, raw}. catalog_writer.write_crafting_trees -> ITEMS-Crafting-Trees.md. First pass: 103 recipe rows / 6 tables. Ingredient/output cells sometimes read "—" because the underlying ingredient list is nested in an opaque struct emitted as _raw_ by uasset_parser; deeper struct decoding is a Phase E hygiene task. C3 — loot-tables (new dune_extract/loot_tables.py) Identifies loot tables (DT_LootTable_* / DT_Loot_* / DT_ContainerLoot / DT_NPCLoot / DT_InventoryItemSets by prefix + field-shape fallback). Normalizes per pool entry into {item_refs, weights, tier_gates, tag_gates, raw}. catalog_writer.write_loot_tables -> ITEMS-Loot-Tables.md. Result: 1,782 loot-pool rows across 429 loot-table assets. NumRolls, PercentageChance, Weight, RequiredLootDifficultyLevel ranges, ItemTagsFilter / RequiredTags shapes all surface. ItemTemplateId refs show as _raw_ (same struct-decoding limitation as C2). End-to-end run (~11 min wall-time): FDI parse: 131,654 entries / 37 paks (3.3 s) Categorized 25,618 stems into 8 categories Row data: 675 DataTables / 15,010 rows Localization: 10 tables / 27,816 pairs applied to 15,009 rows Wrote 11 .md (9 per-category + Crafting-Trees + Loot-Tables) DataExtract/ITEMS/ grew from ~1.5 MB to ~5.8 MB committed in-repo as the new sample catalog. The C1 user directive ("Ship our pre-baked catalog (generated from the same tool path) as a quick-start sample") is now satisfied with resolved English item names + descriptions + mechanical stats per the user's "I need item names, descriptions, and stats" ask. Honest scope note: some struct fields still emit _raw_ (FGameplayTag Container, FPrimaryAssetId, some recipe ingredient arrays). Phase E hygiene work expands the uasset_parser._read_value type switch.