Documents | I tried to build a LitRPG sandbox with on the fly generated sprites

I tried to build a LitRPG sandbox with on the fly generated sprites

by

Kadhir Mani

(5.8 minutes)

<audio url="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=fc8b8c62-71bb-401f-a9fa-0a49d335bf5d"></audio> I've always loved LitRPG novels for one feeling: the world is capable of so much, and it's yours to discover — not in some guide, but through exploration and experimentation. You're limited only by your creativity and understanding. It felt like LLMs should be able to do something here. I want to experience that! My first thought was if I could get sprite sheets generating on the fly, I bet I could slowly figure the rest out. So that's where I started. The hard part wasn't generating a plausible still — it was keeping a character coherent across animation frames and generating it on the fly based on player actions. Here's where I ended up with this experiment: <animation data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=4c79b13b-a3ca-4b4d-8f2f-ffd1f369efc1" data-media-type="image"></animation> <animation data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=86e32c68-c840-4080-b7b4-83e3a394d44c" data-media-type="image"></animation> <h2 id='fb694b63-42f7-45c3-8c33-ef5d5847c183'>What I built and how it works</h2>First problem was figuring out how to request and deliver assets on the fly. I ended up going with a pretty simple web-based setup. The server runs in NestJS, the front end loads Phaser inside Next.js. The structure kept things clean: one HTTP call to request asset generation, another to load the validated result into the game. NestJS owns managing generation and validation, Phaser owns the canvas, input, animation, and game-state. Great easy enough. Next I had to figure out how to generate the assets in a reliable way. I've tried single-shotting these things before, but they don't come out neatly. So instead, I tried something different. In the same way I would iterate to clean up a single sheet, I baked that process (to some approximation) into a Mastra agent loop. I added two things to the engineering harness:<ol><li value="1">Validating the correctness of the sprite sheet -> which should provide specific feedback when it fails</li><li value="2">Cleaning up the sprite sheet -> removing the background, converting formats, etc</li></ol> If the agent produced a poor sprite sheet, it would receive feedback from the harness and try again with the same sheet, asking the model to fix it rather than create a new one. Here's the full flow:<mermaid-diagram-slides data-speed="3000" data-height="320"><mermaid-slide data-title="Step 1: Request" data-description="The front end fires a single HTTP call to trigger generation. NestJS immediately returns 202 In Progress and kicks off the agent loop asynchronously."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST /generate-asset"| API["NestJS Server"] API -->|"202 In Progress"| FE </mermaid-slide><mermaid-slide data-title="Step 2: Generate" data-description="NestJS triggers the Mastra Agent (powered by Claude Sonnet), which calls Gemini 2.0 Flash as a tool to produce a candidate sprite sheet."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST"| API["NestJS Server"] API -->|"202 In Progress"| FE API -->|"Trigger async"| MA["Mastra Agent\n(Claude Sonnet)"] MA -->|"Prompt + ref image + pose sequence"| GEM["Gemini 2.0 Flash"] GEM -->|"Candidate sprite sheet"| MA </mermaid-slide><mermaid-slide data-title="Step 3: Inspect" data-description="The candidate sheet is passed to an algorithmic inspection function — a normal function call that checks for identity drift, pose misalignment, and texture bleed."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST"| API["NestJS Server"] API -->|"202 In Progress"| FE API -->|"Trigger async"| MA["Mastra Agent\n(Claude Sonnet)"] MA -->|"Prompt + ref image + pose sequence"| GEM["Gemini 2.0 Flash"] GEM -->|"Candidate sheet"| MA MA -->|"Run inspection"| INSP["Algorithmic Inspection\n(Drift, Pose, Bleed)"] INSP -->|"Pass / Fail + flags"| MA </mermaid-slide><mermaid-slide data-title="Step 4: Edit Failing Frames" data-description="If inspection fails, the agent sends the existing sheet back to Gemini with targeted feedback. Gemini edits — not regenerates — preserving style consistency."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST"| API["NestJS Server"] API -->|"202 In Progress"| FE API -->|"Trigger async"| MA["Mastra Agent\n(Claude Sonnet)"] MA -->|"Prompt + ref image + pose sequence"| GEM["Gemini 2.0 Flash"] GEM -->|"Candidate sheet"| MA MA -->|"Run inspection"| INSP["Algorithmic Inspection\n(Drift, Pose, Bleed)"] INSP -->|"Pass / Fail + flags"| MA MA -->|"Failing frames + targeted feedback"| GEM GEM -->|"Edited image — style preserved"| MA </mermaid-slide><mermaid-slide data-title="Step 5: Retry Loop" data-description="Generate → Inspect → Edit repeats until all frames pass or the retry cap is hit. Best variants are composited into the final sheet."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST"| API["NestJS Server"] API -->|"202 In Progress"| FE API -->|"Trigger async"| MA["Mastra Agent\n(Claude Sonnet)"] MA -->|"Prompt + ref image + pose sequence"| GEM["Gemini 2.0 Flash"] GEM -->|"Candidate sheet"| MA MA -->|"Run inspection"| INSP["Algorithmic Inspection\n(Drift, Pose, Bleed)"] INSP -->|"Pass or Fail"| MA MA -->|"Fail → targeted feedback"| GEM MA -->|"Pass or retry cap hit"| COMP["Composite Best Variants"] </mermaid-slide><mermaid-slide data-title="Step 6: Engineering Cleanup" data-description="Sheets that pass inspection go through algorithmic cleanup — background removal and other post-processing — before being stored in NestJS's indexed asset folder."> flowchart LR COMP["Composite Best Variants"] -->|"Passed sheet"| CLEAN["Engineering Cleanup\n(Remove BG, etc.)"] CLEAN -->|"Store asset"| STORE["NestJS Indexed Asset Folder"] </mermaid-slide><mermaid-slide data-title="Step 7: Load into Phaser" data-description="The cleaned, stored sheet is fetched via HTTP. Phaser loads it with strict grid alignment, atlas padding, and imageSmoothingEnabled=false for crisp pixel art."> flowchart LR FE["Next.js / Phaser"] -->|"HTTP POST"| API["NestJS Server"] API -->|"202 In Progress"| FE API -->|"Trigger async"| MA["Mastra Agent\n(Claude Sonnet)"] MA -->|"Prompt + ref image + pose sequence"| GEM["Gemini 2.0 Flash"] GEM -->|"Candidate sheet"| MA MA -->|"Run inspection"| INSP["Algorithmic Inspection\n(Drift, Pose, Bleed)"] INSP -->|"Pass or Fail"| MA MA -->|"Fail → targeted feedback"| GEM MA -->|"Pass or retry cap hit"| COMP["Composite Best Variants"] COMP -->|"Passed sheet"| CLEAN["Engineering Cleanup\n(Remove BG, etc.)"] CLEAN -->|"Store asset"| STORE["NestJS Indexed Asset Folder"] STORE -->|"GET /assets/:id"| FE FE -->|"Grid-aligned · atlas padding · no smoothing"| RENDER["Crisp Pixel Art in Phaser"] </mermaid-slide></mermaid-diagram-slides> Laid out in its entirety:<flow-diagram data-nodes="[{"data":{"label":"Front End (Next.js / Phaser)"},"id":"fe","position":{"x":268.33272320000003,"y":-47.050700800000016},"type":"input"},{"data":{"label":"NestJS Server"},"id":"nest","position":{"x":270,"y":180},"type":"default"},{"data":{"label":"Mastra Agent (Claude Sonnet)"},"id":"agent","position":{"x":763.845504,"y":-6.153369600000008},"type":"default"},{"data":{"label":"Gemini Image Tool"},"id":"gemini","position":{"x":761.408,"y":167.328},"type":"default"},{"data":{"label":"Algorithmic Inspection (Drift, Pose, Bleed)"},"id":"inspect","position":{"x":760,"y":340},"type":"default"},{"data":{"label":"Pass?"},"id":"pass","position":{"x":504.36800000000005,"y":351.28202239999996},"type":"default"},{"data":{"label":"Engineering Cleanup (Remove BG, etc.)"},"id":"cleanup","position":{"x":270,"y":340},"type":"default"},{"data":{"label":"NestJS Indexed Asset Folder"},"id":"store","position":{"x":268.33272320000003,"y":499.3585664000001},"type":"default"},{"data":{"label":"Load into Phaser"},"id":"phaser","position":{"x":269.74297600000006,"y":645.2560384},"type":"output"}]" data-edges="[{"id":"e-fe-nest","label":"POST /generate-asset","source":"fe","target":"nest","type":"smoothstep"},{"animated":true,"id":"e-nest-fe","label":"202 In Progress","source":"nest","target":"fe","type":"smoothstep"},{"id":"e-nest-agent","label":"Trigger async","source":"nest","target":"agent","type":"smoothstep"},{"id":"e-agent-gemini","label":"Prompt + ref image","source":"agent","target":"gemini","type":"smoothstep"},{"id":"e-gemini-inspect","label":"Candidate sheet","source":"gemini","target":"inspect","type":"smoothstep"},{"id":"e-inspect-pass","source":"inspect","target":"pass","type":"smoothstep"},{"id":"e-pass-cleanup","label":"Yes","source":"pass","target":"cleanup","type":"smoothstep"},{"animated":true,"id":"e-pass-gemini","label":"No: Edit + Feedback","source":"pass","target":"gemini","type":"smoothstep"},{"id":"e-cleanup-store","label":"Store asset","source":"cleanup","target":"store","type":"smoothstep"},{"id":"e-store-phaser","label":"GET /assets/:id","source":"store","target":"phaser","type":"smoothstep"}]" data-height="580"></flow-diagram> <ul><li value="1">Claude Sonnet runs the agent loop — reasoning, tool calls, and validation decisions</li><li value="2">Gemini 2.0 Flash (image preview) generates and edits sprite frames, exposed as a tool to Sonnet</li><li value="3">Algorithmic inspection checks each candidate for identity drift, pose misalignment, and texture bleed</li><li value="4">Edit-not-regenerate — failing frames go back to Gemini with the existing asset + targeted feedback, preserving style consistency</li><li value="5">Retry cap — loop runs until frames pass or hit the limit; best variants are composited into the final sheet</li><li value="6">Engineering cleanup — sheets that pass inspection go through background removal and other algorithmic cleanup before being stored</li></ul> <h2 id='71c977d4-5953-48c2-8ba0-6017dc3ddcbe'>Some code snippets</h2>Agent prompt<file-upload data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=49cc1c5b-ee8b-4d0f-8caa-38aeadf03822" data-file-name="spriteSheetPromptRenderer.ts" data-description="" data-file-type="application/octet-stream"></file-upload>Sprite sheet generation prompt (inside the tool call)<file-upload data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=49be017e-4795-41d3-88f8-541a82b674c5" data-file-name="spriteGenerationPrompt.ts" data-description="" data-file-type="application/octet-stream"></file-upload>Sprite sheet validator<file-upload data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=2a6a0496-f3c7-4b5f-8eb3-b561c4effabc" data-file-name="validateSpriteSheet.ts" data-description="" data-file-type="application/octet-stream"></file-upload>Sprite sheet clean up<file-upload data-src="https://api.productnow-prod.com/storage/getStorageObject?storageObjectId=de4e195d-8161-47ba-ae96-ba439e2e2c93" data-file-name="transparentizeSpriteSheetBackground.ts" data-description="" data-file-type="application/octet-stream"></file-upload> <h2 id='36c28b63-f66a-4187-b771-4d30ae0c4f25'>What worked, what failed, and lessons learned</h2>What worked<ul><li value="1">The end-to-end pipeline actually produced assets — Mastra's iterative agent loop generated and loaded sprite sheets into the game in real time<ul><li value="1"><web-citation data-url="https://mastra.ai/" data-title="Mastra website"></web-citation></li></ul></li><li value="2">Iterative refinement (generate → inspect → edit) caught obvious failures before they reached the canvas, worked surprisingly well</li><li value="3">The Mastra/Phaser split kept responsibilities clean: agent orchestration on the server, real-time rendering in the browser<ul><li value="1"><web-citation data-url="https://phaser.io/" data-title="Phaser website"></web-citation></li></ul></li><li value="4">The engineering harness (retry cap, best-variant compositing) is model-agnostic — swapping in a specialized sprite model could meaningfully improve output quality without rebuilding the pipeline</li></ul> What failed<ul><li value="1">Animation quality was poor: frames drifted in proportions, pose alignment, and fine detail across states</li><li value="2">Sheets at a glance looked acceptable, but broke down once animated — temporal coherence, not per-frame quality, is the real bottleneck</li></ul> <h2 id='f126ccf1-0fa9-48bf-93d1-584b3f5d4bd2'>Closing thoughts</h2>I'm still hopeful someone will figure this out and publish a new kind of game. One where we're only limited by our collective creativity (and I guess the model capability 😅). Until then, I'll keep messing around with this, see what I can figure out next! What do you think? Sound like a potentially fun game genre? Other random thoughts<ul><li value="1">Sequence-aware, frame-by-frame generation seems like the safest bet when consistency matters, might try that next: <web-citation data-url="https://arxiv.org/html/2412.03685v2" data-title=""></web-citation> <web-citation data-url="https://computationalcreativity.net/iccc24/papers/ICCC24_paper_199.pdf" data-title=""></web-citation> <web-citation data-url="https://www.scenario.com/blog/ai-sprite-generator" data-title=""></web-citation></li><li value="2">It seems like to make this a truly player creativity bound sandbox, LLMs need to be seamlessly integrated in, aka I think chat like features aren't enough, we can do so much more than that!<ul><li value="1">At its core we're setting the rules of the world and letting players fill in the details based on their actions</li></ul></li><li value="3">To make this work at scale, caching shared discoveries (aka generated assets) could be great for making the unit economics of the game work, plus add to the shared experience of the sandbox</li><li value="4">Curious if you can extend this to boss battle loops, where the model can generate the boss attack patterns and stuff on the fly</li><li value="5">It'd be super interesting if the entire economy of such a game was player run, including the normal shops<ul><li value="1">I think it'd cause the whole game to feel like a truly living ecosystem where anything could happen, knowing most of the drops and things were some LLM in the background deciding what made sense</li></ul></li><li value="6">Making this a browser-based game (similar to Runescape) would make it wildly accessible and easy to continually deploy updates to<ul><li value="1">That would also force restrictions on performance, which (I think?) means you focus on game loop quality more</li></ul></li></ul>

Powered by ProductNow