Shaders/SkinnedMesh/TFT_UV_Shift

Authoritative notes on the TFT companion flipbook shader path Shaders/SkinnedMesh/TFT_UV_Shift: how the binary data is shaped, how the extractor resolves it, how GLTF extras are written, and how the web engine consumes it (including the loop grace fix for visibility).

Binary Structure (Skin BIN)

Source: skin.skins[*].material.dynamicMaterial.parameters[] in files like petchibileesin_skin1.bin.json.

• SwitchMaterialDriverElement / mElements
  • mCondition.mAnimationNames: list of animation identifiers
    • May be BinTreeString (plain names) or BinTreeHash (FNV1a of name)
  • mValue.graph:

    {
      "times": [0, 0.25, 0.5, 0.75, 1.0],   // normalized 0–1 timeline
      "values": [0, 1, 2, 3, 4],            // flipbook frame indices
      "mDefaultValue": 7                    // idle/default frame
    }

  • times are normalized; must be multiplied by animation duration seconds.

Extractor Pipeline (C#)

Files: Skin.cs, StaticMaterial.cs, SkinExtensions.cs, SimpleGltf/Json/GltfAsset.cs.

1. Parse elements/drivers

   • ExtractKeyframeFlipbookMappingsFromElements / ExtractKeyframeFlipbookMappings read mElements/mDrivers.
   • Store keyframes into StaticMaterial.FlipbookKeyframes (hash-keyed) and FlipbookKeyframesByName (name-keyed).
   • Set HasFlipbookEvents = true when data exists.

2. Resolve animation identifiers

   • _clipHashToAnimationName: clip hash → exported animation name (from mClipDataMap).
   • _nameHashToAnimationName: FNV1a32(name) → exported animation name.
   • During CollectFlipbookEvents(), hashes are resolved to names; unresolved entries are skipped (no anim_xxxx fallbacks). Result is FlipbookEvents: { animationName -> { frameTime -> frameIndex } }.

3. Export to GLTF extras

   • SkinExtensions writes flipbookEvents into skeletonRoot.Extras (not userData).
   • SimpleGltf.Json.GltfAsset serializes flipbookEvents into GLTF extras for Three.js.
   • Example:

     "flipbookEvents": {
       "Joke": { "0": 7, "0.25": 5, "0.5": 2 },
       "Dance": { "0": 7 }
     }

   • Uses animation names only (matches submeshVisibilityEvents). No hashes reach the frontend.

Web Engine Consumption (packages/web/src/engine/index.ts)

1. Load

   • Reads skeletonRoot.userData.flipbookEvents into this.flipbookEvents.
   • Materials with hasFlipbookEvents are registered in flipbookEventMaterials.

2. Per-frame update

   • Guarded: updateFlipbookEvents() runs only if flipbookEventMaterials.length > 0.
   • currentTime = AnimationAction.time (respects mixer.timeScale).
   • Event times are normalized; multiply by animationDuration.
   • Sorted event times are cached per animation to avoid allocations each frame.
   • Frame selection: last event where currentTime >= normalizedTime * duration; defaults to first event, else material default frame.

3. Loop grace (flicker fix)

   • Detect loop: currentFrame < lastVisibilityFrame - 1.
   • Skip visibility updates for ~2.5 frames after a loop (loopGraceFrames) to avoid flicker at animation restart.
   • Reset lastVisibilityFrame/loopGraceFrames on model load, animation start, and stop.

4. Reset behavior

   • stopAnimation() and T-pose resets call resetFlipbooksToDefault() and clear loop tracking.

Related Systems

• submeshVisibilityEvents: Same per-animation name keying; flipbook events mirror this (names only).
• Standard flipbooks: Exported as flipbookKeyframes keyed by accessory hash; handled separately.
• Auto flipbooks: Use flipbookSpeed for time-based cycling; unaffected by TFT_UV_Shift events.

Testing Checklist

• Use a TFT companion (e.g., PetChibiLeeSin) and verify:
  • extras.flipbookEvents present with animation names.
  • Mouth/face frames change over full animation duration (normalized times honored).
  • No flicker at loop boundaries (grace period working).
• Regression: ensure standard flipbooks (e.g., Milio) and auto flipbooks (e.g., Annie Skin31) still behave; guards prevent overhead when no hasFlipbookEvents materials.

Design Principles Recap

• Name-first export; skip unresolved hashes.
• Write to GLTF extras, not userData.
• No frontend hashing; all resolution in extractor.
• Performance: guarded calls, cached event times; zero overhead for models without events.
• Default frame: prefer mDefaultValue (FlipbookDefaultFrame) as idle frame.