Of Astra and Axolotls.

Astra’s Uncanny Image Modeling Capabilities & Prompting for Analysis and Correction.

Proposed Spec Addition: Analysis of Instructions: If the assistant receives multiple repetitive instructions, analyze why the instructions have not been satisfied and take steps to fix the issue to better comply with user requests.

Astra’s image and video generation capabilities are outstanding, breathtaking, surprising, and outright glee-inducing. Like millions of others, I am obsessed. Interestingly, the model understands an instruction like “make the axolotl cuter.” But it struggles mightily with an instruction like “make the bottom of the lip more wavy on the y axis, not the z axis.”

On Saturday, September 13, I excitedly showed my 11-year-old daughter Abby how Astra made me a Bagel Bot video game that was actually fun to play on the first try. She liked the game, but what she really wanted to build was an axolotl game where the animal could design different domestic spaces and work at different occupations. She didn’t want to shoot things or dodge objects. So Astra immediately gave us an undulating series of basic shapes, and with a little direction, we got it smoothed out and more three-dimensional in only five minutes.

That was fun, but I thought I would surprise her by making her a digital animated model of her beloved axolotl Marshmallow so that she could play with him on her iPad, or, when she’s old enough someday, on an iPhone. So I uploaded a bunch of reference images and things went swimmingly at first. Astra did a marvelous job with simpler shapes— I could tell it to smooth the cylinder of a leg into the larger cylinder of the body and it would. I could even do detailing like scooping out a depression running down the bottom length of the animal. But if there is one thing you have to get right about an axolotl, it’s the smile. That’s basically the animal’s entire raison-d-etre. But I found that Astra could do almost everything well, but when it came to describing the facial features of a gecko, or how to change them— how to press the eyes to be more perpendicular, for instance, or how the bottom of the top lip waved— it was absolutely impossible to get ChatGPT to do the job. Every change led to other problems, and the error compounded with other errors. About four hours into the process, I thought we were finally getting somewhere. But eight hours later, I gave up.

So I asked Chat to do a thorough account of our logs and analyze what went wrong. It reported “59 mouth, lip, chin, jaw, expression, and clarification messages, plus 24 related head/snout messages that affect how those instructions are interpreted: 83 entries total.

It owned up to its shortcomings, as you would hope it to do. “The central failure was mine: I kept making small code edits to an incorrect head shape, then describing those edits as though they had achieved your visual instructions. Your increasingly precise corrections did not produce a corresponding improvement in how I worked.” It also seemed frankly immune to any amount of clarification or correction. For example, I showed a reference photograph, and asked Astra to reproduce the wavy bottom edge of the top lip. First it made it way too exagerated. Then it rotated it 90 degrees, so that instead of waving on the y-axis, it was wavy on the z-axis. Eventually I figured out what it had done, described it fully, and requested it changed— many times— but Astra simply never changed it. It said “Your instruction had become clear; the implementation and visual verification still failed.”

In fairness, what I was asking Astra to do was computationally incredibly complex, even if easy by human standards. But as it put it, the difficulty of the task “does not justify hours of confident, unsuccessful iteration.“ After I asked it to analyze what had gone wrong and report on how to fix the issues, the head and mouth shape improved…but it continued making new mistakes. After smoothing out a blocky chin it improvised for some reason, it added a bit of a beak, well, just because, and a square chin.

A Model Spec Improvement

Just as the Assistant knows to check its work when doing a math or a programming problem, models could be trained to ask “does this visual result match similar requests?” I made 83 requests regarding the mouth; you would think that after, say, the 5th request, the assistant would say, “I keep getting similar requests, maybe I should check the archive of past requests before executing this one and see if they match.” But beyond that, a simple alteration of the model spec could measurably improve performance by saying a version of what I did, at the end of 12 hours of work: read back over the transcript; identify similar requests; see if they have been satisfied, and if not, analyze why not. Then design a method to fix the problem.

That’s what it did, after I forced it to reflect on the process— it analyzed all the reference photos and produced a sketch of the head shape of the animal, then remolded the model according to that study. All it took to improve the behavior was to ask the model to reflect and analyze before modeling. Such reflective prompts could be a feature of the model spec, improving the product without writing a single line of code.

On the second day playing with Astra’s video motion features, I implemented the adjustment and saw immediate improvement. There was no need to make 83 requests— but with a simple addition to the model spec it wouldn’t take 12 hours of failure before a user figured out how to get better results.