From video reference to editable animation: a research notebook

The first dummy moved through the sequence and still felt wrong. It could rotate, land, and punch, but the landing had little weight and the rocks seemed disconnected from the performer. That gap between a technically valid output and a convincing result became the useful part of the experiment.

For Etherbound, I am approaching animation as a data problem: what can I observe, what must I infer, and which assumptions can I test? The study follows a short reference passage through an aerial rotation, a grounded landing, and two rock strikes. Its outputs are editable Blender studies, structured annotations, measurements, and a record of my review decisions.

I am interested in exploring OpenAI’s Astra as a research and coding assistant for this kind of work. The question is whether an assistant can help turn visual criticism into inspectable code and better experiments. The measured annotation calls in this notebook used Gemini 3.8 Flash through OpenRouter; their cost and results are attributed to that component. This single study does not establish a comparative score for Astra.

A plain dummy makes this investigation easier. It removes clothing and material detail from the review, so foot sliding, weak silhouettes, and joint reversals remain visible. The small examples below illustrate the data-handling ideas with synthetic inputs. They are self-contained teaching examples; the production prompts, fitting logic, and orchestration remain private.

We began with screenshots, eventually focusing on frames 116–143. The early reconstruction looked like a jump in place because the specification underrepresented horizontal travel, weight transfer, and the sequence from torso to shoulder to hand. More fluent descriptions would not, by themselves, fix those missing variables.

I separate three kinds of information: observations supported by the image, inferences about hidden geometry or depth, and direction describing the performance I want. A hidden wrist should not acquire a confident image coordinate simply because a downstream solver would find one convenient. A requested deeper stance is creative direction, not a newly discovered fact about the reference.

This reduced record illustrates that distinction. Its values are invented for the example, and its coordinates refer only to normalized image space:

{  "sample_id": "example-001",  "time_s": 0.5,  "observation": {    "landmark": "left_wrist",    "visibility": "occluded",    "image_xy": null  },  "inference": {    "depth": "undetermined"  },  "direction": {    "review_target": "stronger landing compression"  }}

Local tools handle decoding, timestamps, and frame selection. The vision model supplies structured descriptions and landmark estimates. The three recorded annotation requests for run 4 cost approximately $0.104 in provider-reported inference charges. That figure excludes the coding assistant, development time, rendering, and other project costs.

One annotation supplied coordinates for a landmark it also described as hidden. I kept the original response and recorded a correction that cleared the unsupported coordinates. Overwriting the response would have made the record tidier while removing evidence of the failure.

A local SQLite database links annotation runs, feedback, usage, and experiments; the media is stored separately. This gives each review a specific output to refer to. It also helps distinguish an annotation improvement from a change to the animation code, although our day-one iterations changed several variables and cannot isolate causality.

Here is a small Python example of preserving an original observation while producing a corrected one. It demonstrates the principle without exposing the internal annotation schema:

from dataclasses import dataclass, replace
@dataclass(frozen=True)class Observation:    landmark: str    visible: bool    image_xy: tuple[float, float] | None
def clear_unobserved_point(point: Observation) -> Observation:    if not point.visible:        return replace(point, image_xy=None)    return point
raw = Observation("left_wrist", False, (0.42, 0.31))corrected = clear_unobserved_point(raw)
assert raw.image_xy == (0.42, 0.31)assert corrected.image_xy is None

The immutable record makes the difference explicit. The actual workflow also needs provenance, a reason for the correction, and a link to the affected run. Retaining these records supports evaluation; it does not automatically train or fine-tune a model.

My role at each checkpoint was to identify the most consequential failure and turn it into a reviewable requirement. The following summarizes that direction without reproducing the prompts.

• After run 2: the landing appeared to slide backward, the body lacked weight, and the arms did not make the rocks feel connected to the performer. The next revision prioritized stable foot contact, horizontal travel, and clearer coordination between the hands and props.
• After run 3: travel and foot planting improved, but the arms remained unnatural, landing compression was weak, some joints appeared to reverse, and the review lacked the requested audio. The next brief called for valid bend directions, a lower stance before the strikes, body-clear rock paths, and synchronized sound.
• After the base run 4: passing geometry checks did not resolve the choreography. Review called for clearer facing changes, a more legible sweep and launch sequence, and enough shot duration for the second release and its sound. These remain review targets; the later framing revision does not establish that all of them were solved.

A planted foot can be checked against a ground anchor during its contact interval. A knee needs a consistent bend direction as well as a plausible angle. A rock needs clearance from the body along its path. Those requirements make portions of the critique measurable, while timing, force, and readability still require looking at the performance.

The translation layer needs a common clock for visual evidence, editable motion, and audio. For run 4, matching the screenshots located the passage at 9.125–11.875 seconds in the archived source. Several screenshots repeated a frame. The native 24 fps footage supplied 66 distinct frames in that interval, adding temporal evidence without synthesizing intermediate images.

For a constant-rate excerpt, the relationship between source time and study time can be illustrated with exact rational arithmetic. This Python example reproduces the interval arithmetic; a decoder should retain actual presentation timestamps when footage has irregular timing:

from fractions import Fraction
source_start = Fraction(73, 8)  # 9.125 secondsduration = Fraction(11, 4)     # 2.75 secondsfps = 24frame_count = int(duration * fps)
study_times = [Fraction(i, fps) for i in range(frame_count)]source_times = [source_start + t for t in study_times]
assert frame_count == 66assert source_times[0] == source_startassert source_times[-1] < source_start + duration

We extracted the matching source soundtrack, packed it into Blender, and included it in the review exports. The extended run-4 shot changed the excerpt and framing; earlier audio-test scores are not evidence for that revision. Soundtrack placement and whether a landing or strike feels synchronized remain separate questions. This study did not implement automatic sound-event-to-pose inference. The run-5 and run-6 comparisons retain that synchronized source audio so the effects can be heard against the movement. Press Play to listen; the soundtrack is reference evidence, not newly generated sound effects.

The distinction matters: a correctly placed whoosh can expose a late strike more clearly. Audio becomes another review signal in the translation process, rather than a final decoration added after the motion is accepted.

Checking only the authored keys misses failures introduced by interpolation. I sample the evaluated motion between keys for joint constraints, planted-foot error, ground clearance, and rock clearance. Results below belong to the latest verified run-5 revision, rather than carrying forward a passing score from an older run-4 bake.

The following miniature example measures one thing only: maximum distance from a planted-foot anchor over three synthetic samples in meters. The midpoint moves even though the first and last samples agree. The 2 mm tolerance is illustrative, not a reported production threshold:

from math import dist
def max_contact_drift(samples, anchor):    if not samples:        raise ValueError("Contact interval has no samples")    return max(dist(position, anchor) for position in samples)
anchor = (0.0, 0.0, 0.0)samples = [(0.0, 0.0, 0.0), (0.004, 0.0, 0.0), (0.0, 0.0, 0.0)]drift_m = max_contact_drift(samples, anchor)tolerance_m = 0.002
assert drift_m == 0.004assert drift_m > tolerance_mprint(f"Needs review: {drift_m * 1000:.1f} mm contact drift")

A real validator must evaluate the animated scene across the relevant contact intervals and handle coordinate systems, geometry, and tolerances consistently. Dense sampling increases the chance of detecting a failure; it is not a continuous proof. These checks also do not measure balance, biomechanical plausibility, or resemblance to the reference.

A later 3.5-second front-facing shot gave the second launch more time. The storyboard below documents that presentation revision. It has not received the same full validation pass as the base run, and the orthographic camera still limits the impression of rocks approaching the viewer.

Revision: spine-and-balance-fix-02. The latest run-5 checkpoint replaces the rigid torso with a three-segment spine, lets the shoulders tilt through the turn, and finishes on a raised-leg recovery with counterbalancing arms. My review moved from the broad instruction to add weight toward specific relationships: the supporting hip, the lifted leg, the curved torso, and the arms have to belong to the same action.

The comparison also needed to be tested. A straight-on orthographic view had flattened the rocks’ forward travel even though they were moving in the scene. The final comparison uses a perspective camera, and decoded movie panels were checked against that intended view. The source and reconstruction share a 3.4167-second clock; native source frames are held between samples, not synthesized or stretched to hide timing differences.

For this revision, checks at 30, 60, 120, and 480 samples per second found no sampled joint-limit violations or knee/elbow bend reversals. Minimum conservative rock/body clearance was approximately 11.6 cm, maximum planted-ankle error 0.089 mm, and maximum between-key solver deviation approximately 4.61 mm. These figures are tied to the same release manifest as the comparison below.

I accepted this motion as the baseline for the next run, not as a finished physical simulation. The preparatory step still contains a knee-flexion peak around 1,591 degrees per second, and the end-pose balance check uses illustrative segment weights rather than a full dynamic model. Three new annotation requests cost approximately $0.1052 in provider-reported charges; the later camera and spine fixes did not require more annotation requests.

Revision: floating-pebbles-and-open-knee-fix-01 — technical checks passed; visual approval pending. Once run 5 supplied an accepted motion baseline, the next review focused on rocks that looked too uniform and round. Run 6 introduced a small collection of fractured rock and chip meshes. That made it possible to ask a different question: can the assets change without accidentally changing the movement being evaluated?

My subsequent review narrowed the brief to visible pebble trails, a raised knee opening upward and outward, and less linear floating motion before release. This revision adds 18 staggered pebbles, restrained drift and tilt, and an eased launch into a shallow arc. Character curves remain unchanged through 2.96 seconds; the ending right-leg correction is intentional. What began as an asset-only experiment now includes an explicit motion change, which matters when interpreting the result. The review criteria are summarized here; the production instructions remain private.

The latest package now binds the comparison and reports to the same revision: all 54 manifest-listed files matched their SHA-256 digests. Checks at 30, 60, 120, and 480 samples per second found no sampled joint-limit violations or bend reversals. Minimum conservative rock/body clearance was approximately 12.3 cm; maximum planted-ankle error remained 0.089 mm, with approximately 4.61 mm maximum between-key solver deviation. The decoded comparison was also checked against the intended perspective view.

The pebble checks establish projected visibility at selected frames and conservative body clearance—not complete occlusion testing or realism. Fast flexion still peaks around 1,662 degrees per second, so a numerical pass does not settle naturalness. No new annotation calls were needed for this revision. Its checkpoint still awaits visual approval; the comparison below is evidence to review, not a claim that the motion is finished.

The distinction between an authored projectile path and a separate collision experiment remains important. A convincing Blender review does not establish working runtime physics or materials in Unreal. Those integrations need their own checks; they are not implied by an exported asset.

Day one produced an inspectable experiment: preserved inputs, structured annotations, an editable reconstruction, numerical checks, and a sequence of review decisions. Camera calibration, physical scale, hidden depth, and substantial choreography still depend on assumptions or manual choices. Multiple changes between runs mean I cannot attribute the improvement to a single model or intervention.

For a more rigorous exploration of Astra’s capabilities, I would hold the source excerpt and evaluation criteria fixed, then vary one part of the workflow at a time. Useful outcomes would include time to a reviewable revision, contradictions caught in annotations, regressions introduced by code changes, and visual acceptance under the same review brief. A cheaper annotation call and a faster coding iteration answer different questions and should be measured separately.

The most interesting question is whether an assistant helps me formulate a better test after a failure. Run 5 supplied an accepted baseline; run 6 makes the next review more precise: do the floating rocks feel weighted, do the pebbles follow through visibly, and does the opening knee read naturally without disturbing the earlier movement? The latest comparison is the record I will use to judge that progress.