Why Does AI Keep Getting Your Videos Wrong? New Kling Research Reveals the Problem

Artificial intelligence can generate remarkably realistic video from written instructions, creating convincing environments, detailed lighting and increasingly sophisticated movement. Yet creators experimenting with the technology may encounter an unexpected difficulty when the finished scene looks impressive but fails to show the actions, camera movements or interactions they originally requested.

New research from the team behind Kling AI suggests that the challenge of video generation extends beyond producing convincing images. The more demanding problem involves understanding the relationships between objects, actions and instructions, then preserving those relationships throughout a moving sequence.

Released on 3 November 2026, the technical paper examines how multimodal artificial intelligence interprets information and evaluates generated footage. Its findings offer an insight into why visual realism alone is insufficient for professional filmmaking, where creators require consistency, repeatability and control over the results.

Why can AI create realistic footage but misunderstand instructions?

Consider a filmmaker requesting a scene in which someone enters a café, orders a drink and sits beside a window while the camera follows their movement. An AI generator might produce a convincing location and character while placing the actions in the wrong order, altering an object unexpectedly or introducing a camera movement that was never requested.

These examples illustrate the wider technical problem rather than specific failures documented in Kling's experiments, but they demonstrate why a visually attractive result may still be unsuitable for its intended purpose. Professional production requires more than individual frames that look convincing because the relationship between those frames determines whether the scene communicates the director's intentions.

The Kling researchers identify semantic control as a central challenge, referring to a system's ability to understand and preserve the intended meaning of creative instructions. Their report examines several connected stages of video generation, including training data, the interpretation of prompts, the relationship between visual and textual information, and methods for evaluating the finished material.

One important area involves captioning, which converts visual information into descriptive text that can help a model understand the content and structure of a scene. A basic caption might identify someone walking through a room, while a more detailed description could explain the person's direction, the position of nearby objects, the sequence of actions and the movement of the camera.

The researchers argue that richer descriptions can provide more useful information for video-generation models, particularly when instructions involve complex relationships between visual elements. Their work also examines reference-grounded captioning, which helps connect written descriptions to particular images or other source material that a creator wants the generated footage to reflect.

Prompt enhancement forms another part of the research because a short instruction may not contain enough information to guide a complicated scene reliably. The team describes methods for expanding prompts into more detailed descriptions, including approaches involving reinforcement learning and the retrieval of relevant examples.

The paper also introduces Kling-MLLM, a multimodal foundation model designed to process text, images, video, audio and combinations of those inputs. By connecting different forms of information, the researchers aim to support the interpretation and evaluation of complex video-generation tasks rather than relying on written instructions in isolation.

An interesting observation concerns text encoders, which convert language into representations that other models can process. The researchers argue that stronger general language ability does not automatically produce better results in video generation because understanding the visual significance of an instruction may require training that connects language with perception.

This helps explain why an AI system might be capable of discussing a complex scene accurately while struggling to reproduce it as moving footage. Describing an action in words and generating a sequence that preserves the action's physical and visual relationships are related but distinct technical tasks.

The research also considers evaluation methods, including systems designed to identify specific weaknesses in generated material. Rather than judging only whether a video appears realistic, developers need ways to assess whether it follows instructions, maintains consistency and satisfies the intended creative requirements.

However, the paper was produced by Kling's own research team and should not be interpreted as independent evidence that the company's technology consistently outperforms competing systems. It describes technical approaches and observations rather than demonstrating that every problem involving prompt interpretation has been resolved.

Why could better control become more important than realism?

The distinction between impressive footage and useful production material becomes particularly important when AI is incorporated into professional creative workflows. A filmmaker may need a precise sequence of actions, consistent character appearances or camera movements that connect naturally with surrounding scenes, making unpredictable results difficult to accommodate.

If a generator repeatedly produces footage that looks attractive but fails to meet those requirements, the creator may need to produce numerous alternatives before finding something usable. The research does not quantify how much time professional filmmakers currently spend repeating generations, but it explains why greater control could be valuable even when visual quality is already high.

This also raises questions about the benchmarks used to evaluate AI video technology because a model that performs well on visual realism may not necessarily provide the consistency required for filmmaking. Developers who concentrate primarily on producing attractive demonstration clips could overlook problems that become apparent when users attempt to direct detailed sequences.

The Kling paper discusses evaluation methods intended to identify these weaknesses more effectively, including approaches that provide detailed criticism of particular aspects of an output. Such methods could help developers understand whether a problem originated in the interpretation of instructions, the generation process or the assessment of the finished scene.

For entertainment producers, improved control could eventually make AI video more practical for storyboarding, visual experimentation and certain production tasks. However, the paper does not establish that these methods reduce costs, replace conventional filmmaking processes or deliver measurable improvements across commercial productions.

The wider development of synthetic video also creates questions about authenticity and public understanding. Ofcom's Online Nation 2025 research reported that 22% of UK adult internet users had encountered fake or deceptive images or videos, including deepfakes, during the preceding four weeks, with the figure rising to 28% among adults aged 18 to 34.

The report also found that 85% of adults considered it important for online platforms to make users aware when content had been made or edited using artificial intelligence. These findings concern the wider online environment rather than Kling specifically, but they demonstrate why increasing technical sophistication should be accompanied by consideration of transparency and disclosure.

As AI-generated footage becomes more convincing, audiences may find it increasingly difficult to distinguish synthetic scenes from authentic recordings. That concern is particularly relevant when generated material resembles news footage or documentary evidence, where viewers may reasonably expect images to represent real events.

For entertainment, the distinction is different because audiences routinely accept fictional scenes, visual effects and digitally created environments. Nevertheless, transparency about production methods may remain important when AI-generated performances or imagery are presented in ways that could affect viewers' understanding of how a programme was made.

The Kling research also reinforces the continuing importance of human creative judgement because accurately following instructions does not automatically produce a compelling scene. A filmmaker must still decide whether the result supports the narrative, communicates the intended emotion and works alongside the performances and images surrounding it.

The most significant development in AI video may therefore be a system that allows creators to specify, revise and reproduce scenes consistently rather than one that merely produces the most impressive isolated demonstration. Greater control could make the technology more useful within established production processes, although independent testing will be needed to determine how reliably those improvements translate into professional work.

For creators frustrated by unpredictable results, the research offers an explanation of the technical challenges developers are attempting to address. It also suggests that the future of AI filmmaking may depend as much on understanding creative instructions as on generating increasingly realistic imagery.

Research and sources: Kling AI research (2026) | Ofcom Online Nation 2025