Technology

I think creating a lora for your character would help in that case.

A LORA is good for replicating a style, where there's existing stuff, helps add training data for a particular subject. There are problems that existing generative AIs smack into that that's good at fixing. But it's not a cure-all for all limitations of such systems. The problem I'm referring to is kinda fundamental to how the system works today -- it's not a lack of training data, but simply how the system deals with the world.

The problem is that the LLM-based systems today think of the world as a series of largely-decoupled 2D images, linked only by keywords. A human artist thinks of the world as 3D, can visualize something -- maybe using a model to help with perspective -- and then render it.

So, okay. If you want to create a facial portrait of a kinda novel character, that's something that you can do pretty well with AI-based generators.

But now try and render that character you just created from ten different angles, in unique scenes. That's something that a human is pretty good at. Here's a page from a Spiderman comic:

https://spiderfan.org/images/title/comics/spiderman_amazing/031/18.jpg

Like, try reproducing that page in Stable Diffusion, with the same views. Even if you can eventually get something even remotely approximating that, a human, traditional comic artist is going to be a lot faster at it than someone sitting in front of a Stable Diffusion box.

Is it possible to make some form of art generator that can do that? Yeah, maybe. But it's going to have to have a much more-sophisticated "mental" model of the world, a 3D one, and have solid 3D computer vision to be able to reduce scenes in its training corpus to 3D. And while people are working on it, that has its own extensive set of problems. Look at your training set. The human artist slightly stylized stuff or made errors that human viewers can ignore pretty easily, but a computer vision model that doesn't work exactly like human vision and the computer vision system might go into conniptions over. For example, look at the fifth panel there. The artist screwed up -- the ship slightly overlaps the dock, right above the "THWIP". A human viewer probably wouldn't notice or care. But if you have some kind of computer vision system that looks for line intersections to determine relative 3d positioning -- something that we do ourselves, and is common in computer vision -- it can very easily look at that image and have no idea what the hell is going on there. Or to give another example, the ship's hull isn't the same shape from panel to panel. In panel 4, the curvature goes one way; in panel 5, the other way. Say I'm a computer vision system trying to deal with that. Is what's going on there that there ship is a sort of amorphous thing that is changing shape from frame to frame? Is it important for the shape to change, to create a stylized effect, or is it just the artist doing a good job of identifying what the matters to a human viewer and half-assing what doesn't matter? Does this show two Spidermen in different dimensions, alternating views? Are the views from different characters, who have intentional vision distortions? I mean, understanding what's going on there entails identifying that something is a ship, knowing that ships don't change shape, having some idea of what is important to a human viewer in the image, knowing from context that there's one Spiderman, in one dimension, etc. The viewer and the artist can do it, because the viewer and the artist know about ships in the real world -- the artist can effectively communicate an idea to the viewer because they not only have hardware that processes the thing similarly, but also have a lot of real-world context in common that the LLM-based AI doesn't have.

The proportions aren't exactly consistent from frame to frame, don't perfectly reflect reality, and might be more effective at conveying movement or whatever than an actual rendering of a 3d model would be. That works for human viewers. And existing 2D systems can kind of dodge the problem (as long as they're willing to live with the limitations that intrinsically come with a 2D model) because they're looking at a bunch of already-stylized images, so can make similar-looking images stylized in the same way. But now imagine that they're trying to take stylized images, then reduce them into a coherent 3D world, then learn to re-apply stylization. That may involve creating not just a 3D model, but enough understanding of the objects in that world to understand what stylization is reasonable to create a given emotional effect and be reasonable to a human, and when. People may not care that that ship is doing some impossible geometry, but might care a whole lot about the numbers of limbs that Spiderman has. Is it technically possible to make such a system? Probably. But is it a minor effort to get there from here? No, probably not. You're going to have to make a system that works wildly differently from the way that the existing systems do. That's even though what you're trying to do might seem small from the standpoint of a human observer -- just being able to get arbitrary camera angles of the image being rendered.

The existing generative AIs don't work all that much the way a human does. If you think of them as a "human" in a box, that means that there are some things that they're gonna be pretty impressively good at that a human isn't, but also some things that a human is pretty good at that they're staggeringly godawful at. Some of those things that look minor (or even major) to a human viewer can be worked around with relatively-few changes, or straightforward, mechanical changes. But some of those things that look simple to a human viewer to fix -- because they would be for a human artist, like "just draw the same thing from another angle" -- are really, really hard to improve on.