In the previous article, I wrote about a problem that becomes more visible as AI makes software development faster: the amount of code we can produce is growing faster than our ability to fully understand it.
At first, I thought this was mainly a code-quality problem. Better reviews, stronger tests, better prompts, stricter architecture rules, and better tooling all seemed like possible answers. Those things are still important, but over time I realized that they do not completely address the issue.
Sometimes the code is actually fine.
A feature works, the tests pass, the structure looks reasonable, and the diff is understandable. In that moment, there is no obvious reason to be concerned. The problem appears later, especially when I return to the same area after some time and realize that I can still read the implementation, but I no longer have a clear picture of the larger design.
I may understand each function while being less certain about why one module was separated from another, why data moves through a particular path, which assumptions shaped the implementation, or how a new dependency changed the system around it.
That distinction became important to me.
A software system can work correctly while the person responsible for it has only a partial mental model of how it works.
Working Software and Human Understanding Are Different Outcomes
Software engineering gives us many signals of success.
The program compiles. Tests are green. Static analysis passes. The API returns the expected result. The user interface behaves correctly. The deployment succeeds.
These are valuable signals because they tell us something about the state of the software. They help us build confidence that an implementation behaves as expected.
But they do not directly tell us what the human team understands.
This was already true before AI. Large systems have always contained parts that no single engineer completely understands. Frameworks, third-party dependencies, distributed teams, legacy systems, and years of accumulated changes make complete understanding unrealistic.
AI changes the speed of this problem.
A coding agent can create a new module, update several interfaces, introduce a dependency, modify tests, change configuration, and refactor related code in one development session. The resulting code may be correct and even well structured, but the repository can move much further than the human mental model during the same period.
This creates a gap between software progress and human comprehension.
That gap matters because responsibility does not disappear when code generation becomes easier. Someone still needs to maintain the system, review future changes, investigate failures, make architectural decisions, explain trade-offs, and decide whether the software is safe to evolve.
The engineer may no longer need to manually type every line, but the engineer still needs enough understanding to make responsible decisions about the system.
This Problem Is Familiar in Education
The same distinction appears in education.
A student can produce the correct answer without having a deep understanding of the concept behind it. A learner may memorize a procedure, repeat a familiar pattern, and perform well on exercises that look similar to the examples used during practice.
From the outside, this can look like successful learning.
The limitation becomes visible when the context changes.
A slightly different problem may require the student to apply the same concept in a new way. If the student only learned the procedure, performance may suddenly drop. The knowledge was useful, but it was not flexible enough to transfer to a new situation.
This is one reason education cannot be evaluated only by whether a learner produced the expected output.
Good learning involves building a mental model that supports explanation, prediction, connection, and adaptation.
I think this provides a useful way to look at AI-assisted software development.
Developers may become very effective at producing successful outputs through a cycle such as:
describe
→ generate
→ inspect
→ run
→ fix
→ mergeThe process can be extremely productive. There is nothing inherently wrong with that.
But productivity and understanding are different dimensions.
A developer may become faster at producing working software without becoming equally better at explaining how the system is structured or predicting the impact of the next change.
In educational terms, we may be improving performance without necessarily improving the underlying mental model at the same rate.
Recognition Can Feel Like Understanding
Another connection with learning is the difference between recognizing an explanation and reconstructing knowledge independently.
A well-written explanation often feels easy to understand while it is in front of us. The same happens with a good lecture, a tutorial, or a worked example. Everything appears clear because the structure of the reasoning has already been prepared.
The difficulty becomes more visible later, when the learner needs to explain the concept without the original explanation, apply it in a different situation, or connect it with something else.
AI explanations of software can create the same effect.
A coding assistant can produce an excellent explanation of a module, a design decision, or a complex change. While reading it, the system may feel completely understandable. But that does not guarantee that the explanation has become part of the engineer’s own mental model.
The difference is subtle but important.
The AI may be able to explain the system.
The human may be able to follow that explanation.
Neither statement necessarily means the human can independently reason about the system later.
This does not make AI-generated explanations less useful. It simply means they should be treated as learning support rather than proof that learning has occurred.
That is also how good educational technology should work. A useful learning tool should not only make information easier to consume. It should help the learner build knowledge that can be used later without depending completely on the tool.
Understanding Is Something We Construct
Many learning theories share a general idea that learners actively construct understanding rather than simply receive it.
A teacher can explain a concept. A diagram can make relationships visible. A simulation can allow experimentation. An example can reduce abstraction. All of these things can support learning, but the learner still has to connect those pieces into an internal model.
I think software understanding works in a similar way.
Code browsers, dependency graphs, architecture diagrams, Git history, test reports, static analysis, AI explanations, and generated documentation can all help. They provide valuable representations of the system.
But providing more information is not automatically the same as helping someone understand.
A developer may already have access to thousands of files, hundreds of commits, dependency graphs, traces, test output, and documentation. Adding another large summary may increase the amount of available information without reducing the cognitive work required to understand what matters.
This changes the problem.
Instead of treating software understanding mainly as an information-retrieval problem, it may be more useful to treat part of it as a learning problem.
The system should help the human identify important concepts, connect them, understand how they changed, and gradually build a useful mental model.
Code Review May Become Part of the Learning Process
This perspective also changes how I think about code review.
Traditionally, review is mostly discussed as a quality-control activity. We check correctness, maintainability, security, tests, naming, consistency, and design.
Those responsibilities remain important.
But when a large part of the implementation is produced with AI assistance, review may also become one of the main moments when humans reconnect with the system.
A review can help the team understand not only which lines changed, but whether the change introduced a new abstraction, altered an architectural boundary, changed an important data flow, or created a dependency that future work needs to consider.
Some changes are too small to require any meaningful update to the team’s mental model.
Others are not.
A five-line change can sometimes alter an important invariant, while a large mechanical refactor may change very little conceptually. This is another reason why counting files or lines is not enough to describe the significance of a software change.
The more useful unit may be something closer to a change in engineering meaning.
This idea later becomes important for makna, because understanding software evolution requires more than storing textual diffs.
Documentation Is Necessary but Not Sufficient
Documentation is an obvious part of the solution.
Good documentation reduces the cost of understanding a system and preserves knowledge that would otherwise stay only in people’s heads.
But documentation has the same limitation as an explanation in education: the existence of a correct explanation does not guarantee that the reader has integrated it into a useful mental model.
The same is true for architecture diagrams.
The same is true for generated summaries.
The same is true for an AI conversation.
These are all valuable representations, but they should support comprehension rather than be confused with comprehension itself.
This distinction is important for makna.
I do not want makna to become only a system that produces another layer of documentation or another AI-generated summary of a repository. There are already many tools that can generate information about software.
What is harder is helping a human understand which information matters at a particular moment.
For someone joining a project, the important task may be learning the major components and flows.
For someone reviewing a change, the important task may be understanding which parts of the previous architecture have changed.
For someone debugging a failure, the important task may be following evidence from observed behavior back toward relevant engineering decisions.
The information is related, but the learning journey is different.
From Software Explanation to Human Comprehension
This gradually changed the direction of makna.
The early idea was strongly connected to observation. If AI systems are doing more development work, perhaps we need a better way to observe what they do and connect their activity with changes in the repository.
That is still important.
But observation alone is not enough.
A complete trace of every command, edit, test, and interaction could become enormous. It might tell us everything that happened while still leaving the human unsure about what is actually important.
So the system needs another layer.
It needs to move from evidence toward engineering knowledge, and from engineering knowledge toward human comprehension.
This is why the architecture of makna eventually separates three broad concerns:
Observation
What happened?
Knowledge
What does the evidence imply about the software?
Comprehension
What does the human need to understand?The distinction is deliberate.
Raw evidence is valuable because it gives us something trustworthy to return to.
Engineering knowledge helps organize that evidence into entities, relationships, changes, architecture, and assurance.
The comprehension layer exists because neither raw evidence nor structured engineering knowledge automatically tells a person what they need to learn.
The Mental Model Should Evolve With the Software
One idea became especially interesting to me as this architecture developed.
A developer does not need to relearn the entire system after every change.
Usually, we already have some understanding of the software.
What we need is to know which part of that understanding is no longer sufficient.
If a new service is introduced, that may require adding a concept to the mental model.
If two components are merged, part of the old model may need to be removed.
If responsibility moves from one module to another, the model may need to be revised.
If implementation changes without changing the important architecture, perhaps very little needs to change in the human’s understanding.
This is close to the idea that later became ComprehensionDelta in makna.
The purpose is not simply to explain the current system from zero every time.
The more useful goal may be to identify the difference between what a person reasonably understood before and what they need to understand now.
That idea connects software evolution directly with learning.
The repository evolves through engineering changes.
The human mental model should evolve with it.
Keeping the Human Cognitively Connected
I do not think the answer to AI-assisted development is to return to manually writing every line of code.
The productivity gains are real, and they are useful.
I also do not think engineers need to memorize every implementation detail. Software engineering has always depended on abstraction precisely because complete knowledge of every detail is impossible.
The goal is to keep the human cognitively connected to the system at the level required for responsible engineering.
That means maintaining enough understanding of the major components, important boundaries, key data flows, assumptions, recent changes, uncertainties, and evidence behind important claims.
AI can help write the software.
AI can help explain the software.
makna is exploring a different part of the problem: how to help humans maintain meaningful understanding as the software keeps changing.
This is why “it works” is not enough.
Working software is an engineering outcome.
Understanding that software is a human learning outcome.
As AI makes the first outcome much faster to achieve, I think we need to become much more deliberate about the second.