On Astra and Axolotls
Improving GPT-6 Astra’s Uncanny Capabilities by Prompting for Analysis and Correction.
Proposed Model Spec Addition. Analysis of Problems: If the assistant receives multiple repetitive instructions, analyze why the previous instructions have not been satisfied, take steps to fix the issue, and report on the process and progress.
NB: The TLDR is at the bottom. GPT-6 Astra’s image and video generation capabilities are outstanding, breathtaking, surprising, and outright glee-inducing. Like millions of others, I am obsessed. Interestingly, the model understands an instruction like “make the axolotl cuter.” But it struggles mightily with an instruction like “make the bottom of the lip more wavy on the y axis, not the z axis.”
On Saturday, September 13, I excitedly showed my 11-year-old daughter Abby how Astra made me a Bagel Bot video game that was actually fun to play on the first try. She liked the game, but what she really wanted to build was an axolotl game in which the animal could inhabit different domestic spaces and work at different occupations. She didn’t want to shoot things, score points, or dodge objects. So GPT-6 Astra immediately gave us an undulating series of basic shapes, and with a few prompts, we got it smoothed out and more three-dimensional in only five minutes. Her list of occupations included Firefighter, and it’s an ocean setting rather than a lake, but there’s a lake and a waterfall in the ocean. Hey, sponges don’t wear square pants either— deal with it. We both had a ton of fun.
But I thought I would surprise her by making her a more lifelike digital animated model of her beloved axolotl Marshmallow so that she could play with him on her iPad, or, when she’s old enough someday, on an iPhone.
So I uploaded a bunch of reference images and things went swimmingly at first. 🥸
But if there is one thing you have to get right about an axolotl, it’s the smile. I found that Astra could do almost every modeling task well with hours of intensive description from me, but when it came to describing the facial features of a cute salamander, or how to change them— how to press the eyes to be more perpendicular, for instance, or how the bottom of the top lip waved— it was absolutely impossible to get GPT-6 Astra to do the job. Every change led to other problems, and the errors compounded upon other errors. About four hours into the process, I thought we were finally getting somewhere. But eight hours after that, I gave up for the day.
So I asked Chat to do a thorough account of our logs and analyze what went wrong. It reported “59 mouth, lip, chin, jaw, expression, and clarification messages, plus 24 related head/snout messages that affect how those instructions are interpreted: 83 entries total.
It owned up to its shortcomings, as you would hope it to do. “The central failure was mine: I kept making small code edits to an incorrect head shape, then describing those edits as though they had achieved your visual instructions. Your increasingly precise corrections did not produce a corresponding improvement in how I worked.” It also seemed frankly immune to any amount of clarification or correction. For example, I showed a reference photograph and asked Astra to reproduce the wavy bottom edge of the top lip. First it made it way too exaggerated. Then it rotated it 90 degrees, so that instead of waving on the y-axis, it was wavy on the z-axis. Eventually I figured out what it had done, described it fully, and requested it changed—many times— but Astra simply never changed it. It said, “Your instruction had become clear; the implementation and visual verification still failed.”
In fairness, what I was asking Astra to do was computationally incredibly complex, even if easy by human standards. But as it put it, the difficulty of the task “does not justify hours of confident, unsuccessful iteration.“ After I asked it to analyze what had gone wrong and report on how to fix the issues, the head and mouth shape improved…but it continued making new mistakes.
I felt better about my version and what I had contributed to the process when I asked GPT-6 Astra to make a model based on an Internet photo search and entirely on its own.
It didn’t do a very good job, so I asked it to redo it, and it was very confident about its performance and their improvements.
I stuck it on my refrigerator.
TLDR: A Model Spec Improvement
Just as the Assistant knows to check its work when doing a math or a programming problem, models could be trained in visual tasks to reason: “does this request duplicate similar past requests?” After, say, the 4th such occurrence, the assistant could reason: “I keep getting similar requests. The user seems unsatisfied. I should check the archive of past requests before executing this one.”
On the second day playing with Astra’s video motion features, I added an instruction to “analyze and fix” problems and saw immediate improvement. My second day, of motion animation went much better, using this one simple technique.
Proposed Spec Addition:
Analysis of Problems: If the assistant receives multiple repetitive instructions, analyze why the previous instructions have not been satisfied, take steps to fix the issue, and report on the process and progress.
A Success!
But despite such obvious shortcomings, it is worth remembering that the project overall was a success. The fact that an amateur with zero experience in digital animation could build that model in a day through only linguistic commands is wondrous. Amazing. The way I was able to sculpt the rest of the body with only words was incredible. And GPT-6 did a really good job with most features. I asked it to prepare a report about one feature and set of interactions that went remarkably well, the design and sculpting of the eyes.
If you are still reading at this point, it is probably because you helped build Astra’s modeling functions, and if that is the case, I just want to say thank you for giving me this amazing new ability and for bringing so much joy to my daughter and me.
