Code RAG: Indexing Code with Tree-sitter and AST

Codebase indexing: why code search does not work

Naive code search relies on splitting files into fixed-size chunks — by number of lines or characters, as general-purpose semantic search systems do.

In a monorepo, this scheme breaks immediately: a forty-line function falls apart into three pieces, context is lost, and the results include the middle of a class without a signature and without a call site.

Result relevance also drops at the level of vector representations. A fragment vector does not know that it precedes a service method rather than a config or test, or that a variable name here is a domain type rather than a string.

Search confidently returns chunks similar by word set, but not those related to the meaning of the task. The developer gets a dozen files and manually determines which one is even relevant.

Then the language model steps in and finishes the picture: on incomplete context, it confidently fills in non-existent methods, invents parameters, and references classes that are not in the project.

Verifying such a suggestion takes longer than writing the code yourself — the tool turns into a source of false signals, and trust in it drops to zero.

The root of the problem is not the model or the index size, but the fact that code is a structured artifact, yet it is indexed as plain text. Parsing a syntax tree provides boundaries that match real entities: function, class, module, scope. Below is how this changes search quality.

What syntax parsing provides: a measurable effect

When indexing without syntax parsing, a file is cut into fixed-size windows, and the results include fragments: half a function, someone else's signature, a piece of a condition.

Based on such a fragment, the model answers imprecisely or fills in what is missing — hence false matches and references to code that is not in the project. Tree-sitter builds a parse tree, and the indexing unit becomes a whole function, class, or method.

Result precision increases because the match is tied to a meaningful node rather than to a line. Searching by method name in different components yields different nodes with different surroundings, and call context filters out false matches before they reach the model. There is almost no garbage from broken blocks at the top of the results.

The speed of finding the needed fragment is navigation through the tree: we go straight to the definition and its calls instead of manually paging through files.

The number of references to the source drops because the engineer less often needs to open the repository and check what the model meant.

A separate class of errors — "the model referenced non-existent code" — is closed by the fact that each node stores coordinates in the file. The reference can be verified, and a fragment that does not exist does not appear in the results.

Syntax parsing moves code search from "looks like that" to "this method in this file."

Code parsing methods: comparison of approaches

The choice of parsing method determines which questions repository search can address at all.

Codebase indexing is always a trade-off between structural precision and robustness to heterogeneous code: strict parsing provides relationships between entities but breaks on dialects and generated files, while fuzzy parsing works everywhere but returns noise.

The granularity of splitting is decided separately: part of a file, a function, a class, or a whole module. The larger the unit, the more context ends up in the answer and the higher the chance that the needed detail drowns in it.

A hybrid scheme usually delivers a working result: candidates are gathered by lexical and vector search, and block boundaries are refined using the syntax tree.

Quality is checked by the share of answers with a correct reference to file and line — this is the criterion by which approaches should be compared on your code.

Implementation stages: from repository audit to handover

Codebase indexing is not a one-off script but a process that lives alongside repositories.

The quality of the assistant's answers is determined by how the code is split into fragments and how fresh the results are, so we carry out the work in stages and record a verifiable result at each one.

Work sequence

1. Repository audit. We count the number of repositories, languages, and code volume, separate generated and vendor directories, and mark abandoned branches: garbage in the index causes false positives.
2. Parse scheme design. We define the unit of meaning — function, class, module. Splitting is based on the syntax tree while preserving relationships between calls, not on a fixed number of lines.
3. Index build. Initial build across the entire codebase with commit binding: any result can be mapped to a specific code version.
4. Update setup. The index is rebuilt on an event in the version control system — selectively, by changed files, not entirely.
5. Quality measurements. We prepare a reference set of questions for the codebase and calculate the share of hits in the right module and the completeness of context in the answer.
6. Access control. Permissions are inherited from repositories and branches: search does not return code that an employee is not supposed to see.
7. Documentation handover. We provide update procedures, the parse scheme, and metrics by which the team itself controls index quality.

After handover, the index remains manageable: it is clear what to fix when quality drops and how to connect new repositories without rebuilding from scratch.

Implementation artifacts: schemes, access rules, procedures

Implementation ends not with a demonstration of search on prepared questions, but at the moment when the client's team connects a new repository, changes access rules, and updates the index without us.

That is why we hand over not a "working service" but a set of artifacts, each covering its own area of operation.

• Index scheme. Which entities are extracted from code (module, class, function), how they are linked by calls and inheritance, where metadata and vector representations of fragments are stored. It shows why search finds a function definition from an indirect description.
• Parse map. The list of languages and grammars that are parsed into a syntax tree, and what enters the index as plain text. Areas of incomplete parsing are marked separately — so that search is not expected to do the impossible.
• Access rules. Which repositories, branches, and directories are visible to each team, at which layer the permission is checked, and how the restriction is reflected in results. Permissions are inherited from existing groups rather than created as a second list.
• Update procedure. What triggers reindexing (merge into the main branch, schedule, manual run), how long it takes, what happens to the index when code is deleted and files are renamed. Plus the procedure for a full rebuild if the parse scheme changes.
• Quality measurement log. A reference set of questions for the codebase with expected answers and a report on the share of hits before and after changes. Changes to weights and parsing are verified against it, not against feelings.
• Developer instructions. How to formulate a query so that search returns the needed fragment, how to add a new repository, where to look in logs when a result is incorrect.
• Operational materials. Configurations, deployment scheme, monitoring points, and the procedure for rolling back to a previous index version — this is enough for an on-duty shift to bring the system up after a failure without us.

Case study: monorepo search before and after

A monorepo with several thousand packages and tens of millions of lines: answering the question "where is this defined and who calls it" took the engineer up to seven minutes — with switching between directories, reading entities with similar names, and manual verification.

Full-text line search caught matches in comments, fixtures, and generated code, so fewer than a third of the top 5 hits were relevant.

What we changed

We added structural parsing via Tree-sitter: file traversal, parse tree construction, and an entity index — modules, classes, methods, calls, and imports tied to line ranges.

The query runs not over text but over a relationship graph: "who calls this method," "where is the interface implemented," "what will a signature change affect." Reindexing is incremental: after a commit, only the affected tree branch is updated; a full pass is not required.

Metrics before and after

Access control is inherited from the version control system: the index stores the same file permissions as the repository itself, so closed modules do not surface in results for those who have not been granted access to them.

Roles and groups are not duplicated in a separate list — when an employee moves between teams, search permissions change together with code permissions.

How is incremental indexing different from a full rebuild?

A full rebuild traverses the entire codebase: every file is parsed again, split into fragments, and vectorized.

The time of such an operation depends on repository size, not on the size of the edit — in a monorepo this is tens of minutes, and during all this time search returns yesterday's code snapshot.

Incremental update starts from changes in a commit or branch merge.

Parsing via Tree-sitter produces a parse tree, and it shows which functions, classes, and modules are actually affected: only changed nodes are recalculated, while the remaining fragments and their vectors are reused as is.

Incremental updates have limits of applicability, and they matter more than the optimization itself.

Updating by changes is not allowed when the vectorization model changes, the fragment splitting scheme changes, or a new language is added: old and new vectors end up in inconsistent spaces, and proximity no longer means anything.

Deleted files are tracked by tombstone records — otherwise results return code that is no longer in the repository.

The difference in load is also noticeable: an incremental update appends new segments instead of rebuilding the entire storage, so peak writes happen in the background and do not compete with search. Periodic scheduled compaction removes garbage from deleted and rewritten files.

In practice, we maintain a hybrid: an incremental update after every merge into the main branch, a full rebuild on a schedule and necessarily when the model or parse scheme changes. Result freshness is then measured in minutes, not hours.

Technical concerns before starting a project

Before starting, the concerns are usually the same: will a shared index become a permissions hole, what happens after history rewriting, can we handle the volume and languages, will we have to break something that already works. Below is how these questions are addressed in practice.

What happens during history rewriting and a forced branch overwrite?

Indexing is tied to commits, not to the "current state of files." When history is rewritten, only the affected paths are rebuilt; we do not touch the rest.

Outdated fragment versions are marked as stale and deleted according to the retention rule — search does not return code that is no longer in the main branch.

We have several languages and a large monorepo — can it handle it?

Parsing is done via a syntax tree for each language separately, so relationships between functions and classes are not blurred on mixed code. Files without a parser are processed by a fallback method — by sections and indentation.

The monorepo is indexed in parts: active branches have priority, and the rest catches up in a queue.

We already have our own code search deployed — will everything have to be redone?

No. We compare the coverage of the existing solution, connect it as a source, and supplement it where structural parsing is insufficient. A complete replacement is needed only if the current search does not return fragments tied to file and line.

Let's discuss the task: we'll return with an estimate and plan

Our task definition starts with inputs, not with a commercial proposal: the more precise the source data, the fewer assumptions in estimating the scope of work and the more realistic the stage sequence. For an initial review, the following set is sufficient:

• access to the repository or an export, and the list of languages that need to be covered by codebase indexing;
• directory structure and build rules — they show how the code is divided into modules and services;
• examples of questions the solution should answer: function search, call tracing between modules, explanation of an unfamiliar area;
• access restrictions: which directories, branches, and files must not enter the index;
• what is unsatisfactory about the current code search — speed, result completeness, lack of context;
• desired points where your stage approval is needed.

On a call, we will discuss the boundaries of indexing, the frequency of reindexing when branches change, requirements for result quality, and how to connect to the team's working tools.

Based on the results, the client receives an indexing scheme with code parsing into a syntax tree, a staged work sequence, and an implementation plan — with approval points and a measurable result for each stage.

Describe the task in two or three paragraphs — we will return with an estimate of the scope of work and an implementation plan.