Op-Ed: Nvidia cranks up the real-time world for training AI agents and 3D worlds

Cunningly hidden in Nvidia’s somewhat overloaded company blog is a very useful gem. This link goes directly to the story about real-time training to save you some time finding it and keeping track of it.

This blog is about the quality of generated environments for AI training, and it’s absolutely crucial. In fact, I’m wondering if Nvidia fully appreciates how high core value this is.

The story goes like this: The problems in real-time AI operations are actual blockers, affecting response times and overall performance quality.

This is a new approach using the do-everything optimizer Artifixer to generate high-quality worlds for AI orientation and practical experience for AI in a real-time environment. Artifixer scale and resolution are good, and can obviously be up-gunned as required.

A very necessary approach

This is pretty ambitious, arguably long overdue, work and it’s essential. To quote the blog:

Whether the output is a game, film, robot or factory digital twin, the goal is the same: expand the canvas of creativity with AI-generated worlds that are grounded in 3D, governed by physics and directed by creators.

To start with, note the expression “AI-generated worlds”. An AI-generated world could be anything. It could be a simulation, or a pixel-perfect rendition of a real environment or a variant. This is heavy-duty data management; it has functional issues galore, and it’s critical to any physical or even depicted operation.

There’s another, much less recognizable, imperative here. The generated environment could also be a chaotic mess full of low quality or misleading information. AI shouldn’t have to work with crayon-level imagery as “information”, and it shouldn’t have to try to train on it. You could argue, in fact, that this decluttering high-res approach could also become a tool for AI to sort information values from imagery and clutter.

That’s particularly the case in 3D environments where bots and robots are expected to function. If you remember how truly lousy and shabby 3D can be, and the shoddy graphics that were so much appreciated in the early days of VR, you’ll see instant value.

You’ll also see a lot more value in the Artifixer upgrades. This is far more important than mere cosmetic fixes for gruesome resolution. This, for a nice change, is a level of functional detail that actually serves a purpose.

AI has to navigate these geo-messes in training and in operation. 4K of crap is 4K of crap. Response time matters, and built-in visual obstacle courses don’t help. A few wrong pixels are simply irrelevant anyway and need to go.

Imagine a bot trying to interpret this clutter, and train itself using the clutter, not the real environment. This is before you get to actual training values.

Rather than rehash a single word of Nvidia’s blog, (you do need to read it) there are some clear future evolutions that need consideration. This approach to training is so fundamental that the ripples will spread far and fast.

There’s a lot more to the Nvidia information, but it’s necessary to focus on the high values of Artifixer here:

Real world physics: There’s a direct correlation between input visual information and performance quality control for both the AI and its environments. AI with an eye for faults and image issues could identify flaws in materials, in the sense that identifying any image that “does not equal required standard” could save a lot of time for production values. Depending on resolution, this could be at nanoscale.

Image anomalies and “disparate data”: In the real-world context, any hard input that doesn’t equate to the baseline image for say, a car park or a retail shopping aisle could be valuable. “Should this rhinoceros be in the toiletries aisle?”, etc.

Gaming: This could be a huge benefit to very demanding, high-production-value gaming on just about any level. Particularly the more demanding types of gaming like Beyond All Reason, role-playing games, or Warhammer 40K to name a near-handful. In these games, visual dysfunction is a real risk. For quality control and bug finds and fixes, the ability to assess images on this basis could be invaluable.

Robot surgery and high-precision work: If delousing imagery is critical for training, it’s much more so in practical applications. This type of work involves multiple dynamics, not least of which is orientation. A human brain orients automatically to a much higher level than AI can at the moment. This may be the bridge. In surgery, living tissue moves around. In laser or other applications, temperatures and other factors change the orientation.

These are pretty simple examples, and they can get exponentially more difficult fast. AI can’t function on the reflexive basis that human brains respond to situational dynamics, at least, not yet. Any change in an environment has to be effectively processed by Pony Express, even if it’s only a few milliseconds. This is a real issue for real-time responsiveness, and the processing logistics are truly clunky. If the clunkiness becomes systematic due to delays or issues with visual processing, aka “latency”, it’s a genuine problem. Safety and successful outcomes may depend on those few milliseconds.

Watch this tech evolve. It won’t get dull.