This is not a benchmark and not a product review. It is a documented personal point of view based on an engineering project, a detailed task history, and intensive experience working with the Agentic IDE IBM Bob over 14 days.
Maybe some of the observations in this post will confirm your own experience. Others may be different. This is one documented personal perspective from an intensive 14-day engineering project.
Over 14 calendar days, I worked more intensively with an Agentic IDE on a “small” software engineering project.
My goal was not to test IBM Bob as a product. I wanted to experience what happens when I use an Agentic IDE consistently across one project and combine it with Spec-Driven Development, custom skills, MCP integrations, documentation, verification, observability, task handoffs, architecture reviews, and Engineering Provenance.
IBM Bob was used in this project. Its detailed task history gave me evidence about how the work evolved across 48 exported sessions. The project also produced a substantial amount of code, tests, documentation, and recorded agent activity. I will look at these numbers in more detail later.
But the numbers are not the main result.
The most important learning was how my own engineering behavior changed. Some of these effects are already discussed in AI-assisted software engineering, but experiencing them during an intensive project made them much more concrete for me.
“The better the Agentic IDE worked, the more I trusted it. The more I trusted it, the more I worked at a higher abstraction level. I focused more on requirements, architecture, evaluation, and outcomes, while inspecting fewer implementation details myself.“
I call this effect my Navigation System Syndrome.
The project also deepened ideas I had already explored in earlier posts about review, governance, Engineering Provenance, and agent harnesses.
The question I now find more interesting than “How much code can AI generate?” is:
How can we work with increasingly capable Agentic IDEs without gradually losing understanding, verification, collaboration, and engineering control?
Table of Contents
- Why I Did This Detailed Project Documentation
- The Project in Short
- Evidence Behind My Experience
- How I Extended and Used the Agentic IDE Bob
- What the Task History Shows
- Spec-Driven Development: Starting Is Easier Than Continuing
- My Navigation System Syndrome
- A Good Result Is Not Always Exactly My Result
- Verification Does Not Scale at the Same Speed as Generation
- AgentOps Helped Me Look Beyond the Final Result
- Documentation Is Not Obsolete
- External Memory and Engineering Provenance
- Using AI to Review AI
- What Happens to Teamwork?
- My Main Learnings
- Agentic Coding changes developer behavior
- Trust can grow faster than my verification capacity
- Spec-Driven Development helps maintain control
- The Agentic IDE becomes an engineering environment
- Documentation and Provenance remain important
- Human responsibility remains
- Final Thoughts
- Evidence, Scope, and Limitations
- Earlier articles that this experience report builds on
1. Why I Did This Detailed Project Documentation
Over the last months, I have explored several related questions in earlier posts: how AI-generated software can be reviewed, how agents increasingly participate in engineering decisions, why Engineering Provenance becomes important, and how an agent harness connects models with context, tools, execution, state, and feedback.
Many of the ideas behind this project were not new to me.
What was new was the intensity: applying them together, consistently, over 14 calendar days.
I wanted to take a relatively small project and apply these ideas consistently over several consecutive days.
For the size of the project, the amount of specification, testing, documentation, handoff information, evaluation, and review was clearly oversized.
That was intentional.
I wanted to find out:
What happens when I really maintain these engineering practices while an Agentic IDE keeps accelerating the implementation?
2. The Project in Short
The application itself is not the main topic of this article. It primarily provided a realistic engineering environment for the experiment.
The project was a containerized Python application that processed text and different file types, created structured context for AI processing, used object storage, and integrated capabilities from IBM watsonx Orchestrate.
The project was complex enough to include:
- Integration work
- Storage and container concerns
- Changing requirements
- Agent execution
- Evaluation
- Observability
- Runtime behavior
- Architecture decisions
The original project documentation records the development process, test evolution, specification history, handoffs, and agent-related provenance mechanisms in detail.
The final documented project state contained:
| Category | Files | Approx. lines |
|---|---|---|
| Application code | 67 | ~17,700 |
| Tests | 30 | ~11,510 |
| Documentation and configuration | 66 | ~14,000 |
| Total | 163 | ~43,200 |
It also contained 742 automated tests: 439 service tests and 303 evaluation tests.
Approximately 32% of the repository lines were documentation and configuration.
That unusually high share was intentional.
Specifications, prompts, agent definitions, configuration, and handoff information were treated as engineering artifacts, not only as documentation added afterward.
3. Evidence Behind My Experience
I did not want the conclusions to be based only on memory.
The quantitative analysis covers 48 exported Bob sessions across the 14-day project span.
| Metric | Value |
|---|---|
| Exported sessions | 48 |
| Total recorded tokens | 6,208,894 |
| Average tokens per session | 129,352 |
| Maximum | 231,843 |
| Minimum | 38,273 |
The token values are taken directly from the exported Bob task history. IBM does not officially document how these values are calculated, so I do not interpret them as input, output, or cached tokens. I use them only as a rough indicator of session size, not as a model-consumption or cost benchmark. Bob’s billing is based on Bobcoins, not on these values. See the Bobcoins docs.
The repository contained approximately 0.65 lines of test code for every line of application code. This ratio does not measure code coverage, and the number of tests does not prove their quality.
However, together these figures show how strongly automated verification was emphasized in this experiment.
4. How I Extended and Used the Agentic IDE Bob
In most of the 48 exported sessions, I used the normal Agent mode. The table below shows a high-level summary of the recorded usage. It is important to keep in mind that using IBM Bob does not mean that I did not write documentation or code myself, or create and modify diagrams.
| Mode | Sessions |
|---|---|
agent | 37 |
sdd (custom) | 9 |
architecture drawing (custom) | 1 |
arch-review (custom) | 1 |
But modes were only one part of the environment.
I extended the workspace with project-specific skills, MCP integrations, memory, project rules, task tooling, lifecycle hooks, and specialized modes. These covered areas such as Spec-Driven Development, architecture review, security, scalability, maintainability, documentation, and configuration analysis.
I will not explain every extension here. One official IBM resource for extending IBM Bob is the Bob+ documentation (from the IBM Build Engineering Team I am a part of).
The important point is the change in the working environment.
It was no longer simply:
Human → Model → Code
It was closer to:
Human → Agentic IDE → Modes + Skills + MCP + Rules + Memory + Tools → Engineering Work
This made my earlier investigation into the concept of an agent harness much more concrete. I had described the harness as the operational layer around the model that coordinates context, tools, execution, state, and feedback.
I also used local Qwen models through Ollama to help prepare and refine some structured prompts before using them in the Agentic IDE. My project documentation (not public) documents this pattern for the initial architecture prompt.
For me, this confirmed an important observation:
The model itself was only one part of the engineering environment.
5. What the Task History Shows
One advantage of Bob for this case study was the detailed task history.
The most frequently used tools included:
| Tool | Invocations |
|---|---|
read_file | 2,054 |
execute_command | 1,500 |
apply_diff | 823 |
update_todo_list | 357 |
grep | 249 |
list_files | 240 |
write_file | 172 |
| Total for the tools shown | 5,395 |
The activity was strongly concentrated around reading repository state, executing commands, applying changes, and continuing the engineering workflow.
For me, this suggests a simple pattern:
Read → Execute → Change → Verify
This is more interesting than only asking how many lines of code an AI generated.
The activity shows that the Agentic IDE repeatedly worked with existing repository state, executed commands, applied changes, and continued from previous work.
That is much closer to an iterative engineering process than to one-shot code generation.
I would not read too much into the tool counts.
For example, read_file was called far more often than write_file, but that does not tell us how much code was read or written.
The data supports the workflow pattern, not a productivity benchmark.
6. Spec-Driven Development: Starting Is Easier Than Continuing
One of my strongest learnings was not that specifications are useful.
I already believed that.
The real learning was how difficult it is to keep the specification alive.
At the beginning, it is relatively easy to define requirements, constraints, architecture, and acceptance criteria.
Later, the interaction becomes much more direct:
Fix this.
Add this.
Change that.
Run the tests.
Continue.
The implementation moves quickly.
This is exactly when the specification can fall behind.
In this project, requirements, acceptance criteria, status information, and implementation details were repeatedly synchronized with the living specification.
That required discipline.
For me, Spec-Driven Development is therefore not mainly about writing a specification before coding.
It is about maintaining a control loop between intention and implementation.
This is something I already understood. But I experienced its importance much more strongly during these 14 days.
7. My Navigation System Syndrome
This was probably my strongest personal learning from the project.
I call it my Navigation System Syndrome.
This is my own term. It is not an established scientific or engineering concept.
I have used this analogy before in talks and discussions, but this project made me experience it much more intensely.
Think about driving with a navigation system.
Without navigation, we actively pay attention to roads, signs, directions, and landmarks.
When navigation works reliably, we gradually stop doing some of this ourselves.
We trust the system.
Most of the time, that trust is justified.
But after a while, we may reach the destination without really knowing how we got there.
I experienced something similar with Agentic Coding.
The Agentic IDE produced useful results again and again.
That changed my behavior. I increasingly focused on capabilities, requirements, architecture, evaluation, and outcomes. At the same time, I paid less attention to individual variables, implementation details, and internal control-flow decisions.
This created a very productive flow.
Once I was inside that flow, interrupting it to inspect every changed file manually became increasingly unattractive.
That is the core of my Navigation System Syndrome: the more reliably the system guided me, the easier it became to stop checking every part of the route myself.
8. A Good Result Is Not Always Exactly My Result
From time to time, I stopped and inspected larger implementation changes more closely.
Usually, the result was good.
But sometimes I found details such as:
- More hard-coded values than I would normally prefer
- An abstraction placed differently from my own preference
- An implementation choice that technically fulfilled the requirement but was not exactly what I had expected
- Something I had simply not specified precisely enough
This distinction matters.
The problem was not always:
The AI made a mistake.
Sometimes the real problem was:
I had not specified what I actually wanted precisely enough.
This changes the role of the developer.
The more implementation freedom I delegate, the more clearly I have to decide:
- What must be specified precisely
- What can be delegated
- Where explicit verification is necessary
- Where I intentionally accept implementation freedom.
This fits closely with my earlier argument that the most important shift is not only who writes the code, but who participates in engineering decisions.
9. Verification Does Not Scale at the Same Speed as Generation
During this project, the Agentic IDE could inspect, change, test, and refactor more material than I could manually verify in the same amount of time.
For me, full line-by-line inspection therefore became increasingly unrealistic as the pace of generated changes increased.
This is related to the review problem I had already described in Who Reviews AI-Generated Software?: code generation can scale faster than complete human understanding and review.
That does not mean verification becomes less important.
It means verification has to happen at several levels.
In this project, I relied on:
- Acceptance criteria
- Automated tests
- Runtime validation
- Evaluation
- Architecture reviews
- Task history
- Observability
- Provenance
- Targeted manual inspection
One project incident illustrated this well: functionality that worked in unit tests still failed in the complete runtime environment and had to be corrected afterward.
So one lesson remains simple:
A passing unit test is not the same as a working system.
And 742 automated tests are not a production-readiness certificate.
10. AgentOps Helped Me Look Beyond the Final Result
One capability I found particularly useful was AgentOps in IBM watsonx Orchestrate.
I used it while building, testing, and evaluating the agentic part of the application because the final output alone did not always tell me enough about what happened during execution.
Sometimes I need visibility into the execution itself:
- What happened
- Where unexpected behavior occurred
- How a run related to an evaluation
- Where weaknesses appeared
In this project, I used the AgentOps observability and evaluation capabilities around watsonx Orchestrate to better understand agent execution.
This reinforced another practical rule for me:
Do not only inspect the result. Inspect how the agent reached the result.
The project also propagated watsonx Orchestrate conversation IDs through results and evaluation information so individual executions could be traced back to governance and inspection tooling.
For me, this links observability directly to Engineering Provenance.
11. Documentation Is Not Obsolete
I recently had a discussion with someone who expressed this perspective:
The source code is the documentation. With an Agentic IDE, documentation can always be generated when needed.
I partly agree.
AI can inspect a repository and reconstruct useful technical documentation.
But there is a difference between:
What exists?
and:
Why does it exist?
Source code does not automatically preserve:
- rejected alternatives,
- deferred decisions,
- assumptions,
- changing requirements,
- architectural intent,
- and shared project understanding.
That is one reason why maintained high-level documentation remained useful during this project.
The more I worked at a higher abstraction level, the more valuable it became to have a current project description without having to reconstruct everything from source code first.
This also connects to my earlier Engineering Provenance work, where I argued that handoffs and recorded decisions can become part of the evidence behind an AI-assisted artifact.
12. External Memory and Engineering Provenance
One artifact became particularly valuable: tasks-handoff.md. It became a form of external engineering memory. An AI conversation context is temporary. A version-controlled engineering artifact can persist across sessions, developers, and tools.
The project also maintained:
- A living specification
- Requirement IDs
- Task history
- Versioned prompts
- Versioned agent definitions
- Evaluation artifacts
- Test history
- Execution identifiers
Within the 48 analyzed sessions, I found no unexplained gaps, missing handoffs, or orphaned tasks.
In this YouTube video, I show how I exported the Bob tasks.
Within the scope of this project, the recorded task history gave me a coherent and traceable engineering record. I did not evaluate it against a formal audit standard.
This was not an automatic property of Agentic Coding.
Engineering Provenance has to be designed.
13. Using AI to Review AI
There is an obvious contradiction in this experiment.
I also used the Agentic IDE to analyze and review parts of the project that the Agentic IDE itself had helped to build.
I had already discussed this recursive problem in Who Reviews AI-Generated Software?
This is not independent verification, but it can still be useful.
An agent can inspect large parts of a repository, aggregate history, compare artifacts, calculate statistics, and identify possible weaknesses.
My conclusion is therefore not:
“The AI reviewed it, so it is correct.”
It is:
AI can extend the reach of engineering review, but humans still need to interpret, verify, and challenge the result.
The architecture review confirmed this.
It found several strong areas, but also important weaknesses around security, scalability, concurrency, and deployment readiness.
The fact that these weaknesses existed alongside 742 tests was useful evidence in itself.
14. What Happens to Teamwork?
The more capabilities I moved into the Agentic IDE, the fewer reasons I had to leave the environment. Questions I might previously have discussed with a colleague could often be explored directly with the agent.
For an individual developer, that can feel highly productive.
But what happens at the team level?
This connects closely to “The End of Teamwork?“.
The article argues that software is built not only from code but from agreement and introduces the term Collaboration Debt: individual throughput can increase while the conversations through which teams develop shared understanding may decrease.
I find this especially relevant because I experienced the first part directly:
I could work increasingly independently with the Agentic IDE.
But this project does not provide evidence that this damages teamwork.
That is an important distinction.
What it gave me was a new question:
If developers solve more engineering problems individually with agents, how do teams preserve shared understanding?
Questions such as functional decomposition, responsibility, shared architecture knowledge, and coordination between multiple humans and agents deserve a separate investigation.
I would not expand on this further in this post.
15. My Main Learnings
After these 14 intensive days, my main conclusions are:
1. Agentic Coding changes developer behavior
The effect is not only technical. It changes the level at which I work and how much detail I inspect personally.
2. Trust can grow faster than my verification capacity
When the system produced useful results repeatedly, trusting it became increasingly natural. At the same time, manually verifying every implementation detail became less realistic. This is the basis of my Navigation System Syndrome.
3. Spec-Driven Development helps maintain control
The difficult part is not creating the first specification. It is keeping intention and implementation synchronized.
4. The Agentic IDE becomes an engineering environment
The model is only one component. Modes, skills, MCP integrations, memory, rules, tools, execution, and observability increasingly shape how engineering work is performed.
5. Documentation and Provenance remain important
AI makes reconstruction easier, but decisions, intent, handoffs, and engineering history still benefit from persistent artifacts.
6. Human responsibility remains
An agent can perform a large amount of engineering work, but somebody still has to answer the final engineering question:
Is this really the system we wanted to build?
16. Final Thoughts
This project did not fundamentally change what I had written about before. It deepened those ideas.
Earlier I had asked how AI-generated software can be reviewed, argued that governance becomes more important as AI participates in engineering decisions, and explored why Engineering Provenance may be needed when agents become part of the development process.
During these 14 days, I experienced these questions much more intensely.
The most interesting learning was not about IBM Bob itself.
It was about how my own working behavior changed once the Agentic IDE became reliable and capable enough that I trusted it deeply.
The more complete the environment became, the easier it was to stay in the flow.
The more I stayed in the flow, the less attractive it became to inspect every implementation detail.
That is productive.
But it also creates a new engineering risk.
This is my Navigation System Syndrome.
For me, the important question is therefore no longer:
How much code can AI generate?
It is:
How do we create an engineering environment in which humans and agents can move quickly without losing understanding, traceability, verification, collaboration, and responsibility?
Two parts of my engineering environment were particularly useful: IBM Bob’s detailed task history helped me reconstruct the development process, while AgentOps capabilities around watsonx Orchestrate helped me inspect agent execution more deeply.
It is an experience report about working intensively with an Agentic IDE.
And my strongest conclusion is:
The more capable Agentic IDEs become, the more important engineering discipline becomes — exactly because these tools make it increasingly easy to stop looking at every detail.
17. Evidence, Scope, and Limitations
This experience report also builds on my earlier posts about AI-generated software review, governance, Engineering Provenance, and agent harnesses.
The teamwork section uses The End of Teamwork? only as an external perspective. That article argues that reduced collaboration can create what the author calls “Collaboration Debt” by weakening shared understanding. My project itself does not establish that effect in teams.
The limitations are important:
This information is based on project artifacts, including the source repository, Bob task history, tasks-handoff.md, the living specification, automated tests, evaluation artifacts, versioned agent definitions, architecture-review results, and execution information.
- This is one project, not a benchmark
- IBM Bob was the environment used, not the object being compared
- Token values from the task export are not officially documented and are not a measure of model consumption or cost
- Tool invocation counts do not measure engineering effort
- Lines of test code are not equivalent to code coverage
- Architecture-review results reflect assessment judgments
- The application was a pilot, not a production-ready system
- The teamwork implications remain an open question
18. Earlier articles that this experience report builds on
[1] Thomas Suedbroecker, Who Reviews AI-Generated Software?, April 2026.
https://suedbroecker.net/2026/04/03/who-reviews-ai-generated-software/
[2] Thomas Suedbroecker, From AI Coding Assistants to Autonomous Engineering Systems, June 2026.
https://suedbroecker.net/2026/06/08/from-ai-coding-assistants-to-autonomous-engineering-systems/
[3] Thomas Suedbroecker, From Idea to Responsibility: Why AI Coding Needs Engineering Provenance, August 2026.
https://suedbroecker.net/2026/08/12/from-idea-to-responsibility-why-ai-coding-needs-engineering-provenance/
[4] Thomas Suedbroecker, Agent Harness in 2026: Hype Word, Industry Term, or Useful Technical Concept?, August 2026.
https://suedbroecker.net/2026/08/30/agent-harness-in-2026-hype-word-industry-term-or-useful-technical-concept/
[5] Thomas Suedbroecker, Coordinating Multiple Agentic IDEs with a Shared Handoff File?, June 2026.
https://suedbroecker.net/2026/06/20/coordinating-multiple-agentic-ides-with-a-shared-handoff-file/
Products and technical documentation
[6] IBM, IBM Bob Documentation.
https://bob.ibm.com/docs/ide
[7] IBM, IBM Bob Tutorials.
https://bob.ibm.com/docs/ide/tutorials/introduction
[8] IBM, watsonx Orchestrate.
https://www.ibm.com/products/watsonx-orchestrate
[9] IBM, watsonx Orchestrate Documentation.
https://www.ibm.com/docs/en/watsonx/watson-orchestrate
[10] Ollama, official project website.
https://ollama.com/
[11] Qwen, official project website.
https://qwen.ai/
[12] Bob+ — IBM Build Engineering
https://ibm-self-serve-assets.github.io/building-blocks-docs/
External perspective on teamwork
[13] The End of Teamwork? — Fun Machines
https://funmachines.substack.com/p/the-end-of-teamwork
Note: This post reflects my own ideas and experience. AI was used as a writing and thinking aid to structure and check the arguments, not to define them.
#AgenticAI, #AgenticCoding, #AICoding, #AIEngineering, #SoftwareEngineering, #AgenticIDE, #EngineeringProvenance, #SpecDrivenDevelopment, #SoftwareArchitecture, #AIObservability, #AgentOps, #HumanInTheLoop, #DeveloperExperience, #AITools, #FutureOfSoftwareDevelopment
Thomas Suedbroecker's Blog 
Leave a comment