#!/usr/bin/env python3 """The language-neutral half of the `code`-facts ingest. An ingest has two halves. One READS a tree and knows a language: what a function is, what a call looks like, where a dependency is declared. The other turns whatever was read into a DKE Python program that records it — subject ids, the row tables, the header, the CLI, the stderr report. Only the first half is language-specific, and this module is the second half. A reader supplies a `Language` and a `collect` function; everything below here is the same whatever the reader read. Adding a language is therefore a walker, not a second copy of the emitter — which matters because the parts that are easy to get subtly wrong (a subject that changes between runs, a Call.callee that is not a Function subject, an optional cell written as an empty string) live HERE, are guarded once, and cannot drift per language. Nothing in this file talks to the network and nothing spawns a process. The readers must hold to that too; `test_ingest.py` asserts it over every module in this directory rather than over one of them. """ from __future__ import annotations import argparse import hashlib import re import sys import time from dataclasses import dataclass, field from pathlib import Path, PurePosixPath from typing import Callable # The kind names the `code` module declares. A module-qualified path is # written `code.Function..`; the ingest never writes an # unqualified kind, so these facts cannot land on some other module's schema. MODULE = "code" # Exit codes. 0 is an ingest that read something; the rest are the ways it # can produce an empty program, each distinguishable from the outside. EXIT_OK = 0 EXIT_NO_SUCH_PATH = 2 EXIT_NOTHING_TO_READ = 3 # The identity half of the fact-writing layer lives in `dke.factwriter`, which # knows nothing about code. Re-exported here because the readers import both # names from this module and there is no reason to make them learn a second # import for a move that changes nothing they can see. # (INGEST-R1-EXTRACT-THE-FACT-WRITING-LAYER) from dke.factwriter import FactProgram, Ids, sanitize # noqa: E402,F401 @dataclass class Function: qual: str name: str module: str lines: int branches: int loops: int is_test: bool = False # a test function, by the reader's conventions @dataclass class Klass: qual: str name: str module: str methods: int @dataclass class Call: caller_qual: str callee_qual: str site: str @dataclass class UnresolvedCall: """A call whose TARGET COULD NOT BE NAMED — the inventory of the hole. CODE-MODULE-UNRESOLVED-CALL-SITES. Dropping an unresolvable edge is right and does not change: a missing edge is a known gap, a guessed edge is a wrong answer the queries cannot tell from a fact. But dropping the edge also dropped the observation that a dispatch site EXISTS, so `uncalled()`'s caveat — a call we could not see leaves its callee looking uncalled — was prose a reader met once and could never query. This records the site WITHOUT inventing an edge: we assert that a call happened here and that its target was not determined. That is a fact, as true as a resolved call. ONLY the unnameable case belongs here, and the distinction is load-bearing. A walker drops a call for two different reasons: the target could not be named at all (dispatch through a variable, a handler table, `getattr`), or the target WAS named and simply lies outside the ingested set (a stdlib or third-party call). Only the first can make an INGESTED function look uncalled, because only an unnameable target might have been one of them — an external call resolves to a name that is not a Function subject and can never suppress one. Recording the second here would inflate the inventory with sites that say nothing about how far `uncalled()` can be trusted, which is the one question this kind exists to answer. """ caller_qual: str site: str text: str # the callee expression as written, so a reader can see WHY @dataclass class Rationale: """A reason the author WROTE DOWN, attached to what it explains. CODE-MODULE-RATIONALE-FACTS. Every other kind this producer emits is something a reader could recompute from the source — structure. The reason the structure is shaped that way is written down in most codebases and reached the store nowhere, which is a strange gap in a product whose whole argument is that reasoning should be explicit and inspectable. STATED, never inferred. `Counterpart` is the precedent: two functions that have to agree is "a claim about intent, and only a reader knows it", so the reader states it and the engine holds the consequences. The comparable that motivated this reaches for an LLM to infer intent instead; a stated fact does not rot silently and an inferred one does. THE MARKER SET IS A DECISION, not a discovery, and it is OURS rather than the module's — a customer's producer chooses its own. `WHY:` and `NOTE:` and nothing else, on the precedent of CODE-COMPLEXITY-COUNTING-UNDEFINED, where what counts as a branch was made the producer's stated decision rather than a hidden one. Both require the author to have typed a prefix, which is what makes this opt-in: a convention wide enough to sweep ordinary comments would turn every line in the tree into a fact and the query into noise, and that is the failure mode worth more than the coverage. `TODO:` is deliberately NOT a marker. It records an intention, not a reason, and this tree's own CLAUDE.md bans bare TODOs outright — reading them as rationale would import a convention the codebase rejects. """ subject: str # the SUBJECT of the Function or Class it explains marker: str # WHY | NOTE — as written, minus the colon text: str # the comment body, one line site: str # file:line # The marker convention, defined ONCE for every language this producer reads. # Two readers with their own copy is how the Python and C++ halves would come to # disagree about what counts, and a customer comparing two halves of one # polyglot store would have no way to see it. # # Anchored at the START of the comment body: a `WHY:` in the middle of a # sentence is prose about markers, not a marker. Case-sensitive for the same # reason — `why:` at the head of an ordinary sentence is common, `WHY:` is not # something anyone types by accident, and being typed on purpose is the entire # signal this kind carries. _RATIONALE = re.compile(r"^(WHY|NOTE)\s*:\s*(\S.*)$") def rationale_marker(comment_body: str) -> tuple[str, str] | None: """(marker, text) if this comment body opens with a marker, else None. Takes the body with its delimiters already stripped, because what those look like is the language's business and what counts as a marker is not. """ m = _RATIONALE.match(comment_body.strip()) if m is None: return None return m.group(1), m.group(2).strip() def enclosing_definition(line: int, spans: list[tuple[str, int, int]]) -> str | None: """The qualified name whose body contains `line`, innermost first. Attribution is by CONTAINMENT and nothing else. A comment sitting above a definition reads to a human as being about it, and attaching it that way was considered and rejected: the line before a `def` is just as often the tail of the previous one, and a rule that is right most of the time produces facts nobody can tell from the wrong ones. Unattached rationale is dropped rather than guessed at, which is the same call the call graph makes about an edge. Innermost wins so a marker inside a nested function belongs to that function rather than to the one around it. """ best: str | None = None best_size = None for qual, start, end in spans: if start <= line <= end: size = end - start if best_size is None or size < best_size: best, best_size = qual, size return best @dataclass class Import: importer: str # module qualified name imported: str # module name as written @dataclass class Dependency: name: str # the distribution name version: str # exact version when known; empty when only a range was declared direct: bool | None # True when declared by this project; None when transitive pinned: bool # declared exactly, so the version is not a guess @dataclass class Facts: functions: list[Function] = field(default_factory=list) classes: list[Klass] = field(default_factory=list) calls: list[Call] = field(default_factory=list) # CODE-MODULE-UNRESOLVED-CALL-SITES — sites whose target could not be # named. Absent-tolerant: a producer that records none leaves the store # working exactly as it does today. unresolved: list[UnresolvedCall] = field(default_factory=list) # CODE-MODULE-RATIONALE-FACTS — reasons the author wrote down, attached to # what they explain. Absent-tolerant in the same way: record none and every # other answer is unchanged. rationale: list[Rationale] = field(default_factory=list) imports: list[Import] = field(default_factory=list) deps: list[Dependency] = field(default_factory=list) dep_files: list[str] = field(default_factory=list) seen_deps: set[str] = field(default_factory=set) calls_seen: int = 0 calls_resolved: int = 0 # Candidate source files the walker FOUND (extension matched, not # excluded) versus the ones it went on to read. The pair is what makes # "this tree has no source I can read" distinguishable from "this tree # has source and no functions in it" — before they existed, both printed # a row of zeros and exited 0. files_found: int = 0 files_excluded: int = 0 files_parsed: int = 0 files_failed: list[tuple[str, str]] = field(default_factory=list) # qualified function name → the name of a test that reaches it. Filled by # reached_by_tests() after call resolution, so it can only ever be as good # as the call graph — which is a lower bound, and so is this. tested: dict[str, str] = field(default_factory=dict) test_files: int = 0 @dataclass class Language: """What a reader is, from the emitter's side. `tag` is the value written to `code.Function.language` and `code.Class.language`. It is what lets one store hold facts about more than one language and still be sliceable — and what lets a reader of those facts see that two producers were involved before comparing numbers that may be on different scales. """ tag: str # `python` — the stored value name: str # `Python` — for prose prog: str # `dke-ingest-python` — argv[0] and messages extensions: tuple[str, ...] # `(".py",)` — what the walker looks for caveats: list[str] # header prose about THIS language's limits collect: Callable[[list[Path], list[str]], Facts] def path_in_test_dir(path: str) -> bool: """True when some directory on the way to `path` is `test` or `tests`. Shared because it is a directory-layout convention rather than a language one. Each reader layers its own file-NAME and callable-name rules on top, which are the parts a language actually decides. """ p = PurePosixPath(path.replace("\\", "/")) return any(part in ("test", "tests") for part in p.parent.parts) def reached_by_tests(facts: Facts) -> dict[str, str]: """Qualified function name → the name of a test that reaches it. TRANSITIVE over the recorded call graph, not direct-only. A test almost never calls the function under test at depth 1 — it calls an entry point which calls it — so a direct-only rule would leave `untested` nearly as universal as writing nothing, which is the defect this exists to fix. The claim being recorded is therefore "a test was seen to reach this", NOT "this is covered". Reaching is not executing: a branch may never be taken, and the call graph under it is a lower bound, so a function reported as untested may simply be reached a way the source does not show. That is why `untested` is a candidate list, in the same register as `uncalled()`. Deterministic: roots and the frontier are processed in sorted order, so the test recorded for a function reached by several is always the same one. """ edges: dict[str, list[str]] = {} for c in facts.calls: edges.setdefault(c.caller_qual, []).append(c.callee_qual) tested: dict[str, str] = {} for root in sorted(f.qual for f in facts.functions if f.is_test): test_name = root.rsplit(".", 1)[-1].rsplit("::", 1)[-1] # Breadth-first from this test. `tested` doubles as the visited set: # a function already attributed to an earlier test keeps that # attribution, which is what makes the result order-independent. frontier = [root] while frontier: nxt: list[str] = [] for q in sorted(frontier): if q in tested: continue tested[q] = test_name nxt.extend(edges.get(q, ())) frontier = nxt return tested def emit(facts: Facts, ids: Ids, lang: Language, *, stamp: int | None = None, load_key: str | None = None) -> str: """Render the facts as a DKE Python program. `stamp` and `load_key` belong to the load marker (`FactProgram.load_marker`) and both default: the stamp to the wall clock in milliseconds, the key to this reader's language TAG — `python`, `cpp` — which is the same token the facts themselves carry in `language`, so a reader asking whether the last Python load finished spells it exactly as they spell everything else. It defaulted to `name` for one commit, which is the prose spelling (`Python`, `C++`): every unit test passed, because they supply their own key, and what caught it was running the producer over a real tree and finding a cell the published README's own example does not name. Keying on the LANGUAGE rather than on the tree is deliberate and is the weaker half of this: a `code` store holds one codebase that may be several languages, so "did the last Python load finish" is the question a reader has, and two Python trees loaded into one store share a marker. The key is a parameter so a caller that knows the tree can say so; `emit` is not given the root and will not infer one. """ if stamp is None: stamp = time.time_ns() // 1_000_000 resolved_pct = (100.0 * facts.calls_resolved / facts.calls_seen if facts.calls_seen else 0.0) # The generic half — the write verb, the row form, the escaping, the # program shell — is FactProgram's. What stays here is everything that # knows what `code` MEANS. The preamble below is this reader's own prose, # which is why it writes into prog.lines directly. prog = FactProgram(MODULE) L = prog.lines L.append(f"# Facts about a {lang.name} codebase, for the `code` reasoning module.") L.append("#") L.append(f"# Generated by {lang.prog}. This file records what was found;") L.append("# `code` supplies what to make of it. Submit this, then ask `code`.") L.append("#") L.append(f"# language {lang.tag}" f" (recorded on every Function and Class)") L.append(f"# files parsed {facts.files_parsed}" f" (of {facts.files_found} {'/'.join(lang.extensions)} file(s) found)") L.append(f"# functions {len(facts.functions)}") L.append(f"# classes {len(facts.classes)}") L.append(f"# imports {len(facts.imports)}") _direct = sum(1 for d in facts.deps if d.direct) L.append(f"# dependencies {len(facts.deps)}" f" ({_direct} declared, {len(facts.deps) - _direct} transitive)" + (f" (from {', '.join(facts.dep_files)})" if facts.dep_files else " (none declared)")) L.append(f"# call sites seen {facts.calls_seen}") L.append(f"# call edges recorded {facts.calls_resolved}" f" ({resolved_pct:.0f}% of sites)") L.append(f"# dispatch sites {len(facts.unresolved)}" f" (call happened, target not determined)") L.append("#") _tested = len(facts.tested) _tested_pct = (100.0 * _tested / len(facts.functions)) if facts.functions else 0.0 L.append(f"# test files {facts.test_files}") L.append(f"# functions a test reaches {_tested}" f" ({_tested_pct:.0f}% of functions)") L.append("#") L.append("# The call graph is a LOWER BOUND. Every edge recorded is real;") L.append("# edges are missing — a call this reader could not resolve from") L.append("# the source alone is NOT recorded, because a guessed edge is a") L.append("# wrong answer the queries cannot tell from a fact. `uncalled()`") L.append("# over this graph lists CANDIDATES.") L.append("#") L.append("# Where the bound is loose is not left to your imagination: each") L.append("# missing edge is recorded as an UnresolvedCall — the site, and") L.append("# the source text whose target could not be named. `dispatch_sites()`") L.append("# lists them, so the caveat above is a number for THIS codebase") L.append("# rather than a warning about every codebase.") for line in lang.caveats: L.append(f"# {line}" if line else "#") L.append("#") L.append("# `Function.test` names a test SEEN TO REACH that function, over") L.append("# that same graph — so it inherits the same lower bound, and it") L.append("# is not a coverage measurement. Reaching is not executing: a") L.append("# branch may never be taken. The `untested` rule therefore marks") L.append("# CANDIDATES too — a function only a dynamic call reaches, or one") L.append("# exercised by a test runner without a call in source, is marked") L.append("# untested here. Read it as `no test was seen to reach this`.") L.append("#") L.append("# `untested` is a standing RULE, not a query you call: it sets") L.append("# `Function.untested` on every function it holds for. To see them,") L.append("# read the field — `subjects(Function.untested, True)` — or call") L.append("# `critical()`, which is the same set narrowed to the complicated") L.append("# ones. Until 2026-08-25 this paragraph wrote that name with") L.append("# call parentheses after it, and there has never been a script by") L.append("# that name to call.") if facts.files_failed: L.append("#") L.append(f"# {len(facts.files_failed)} file(s) did not parse and contributed nothing:") for rel, why in facts.files_failed[:10]: L.append(f"# {rel} ({why})") if len(facts.files_failed) > 10: L.append(f"# … and {len(facts.files_failed) - 10} more") prog.open() # Discipline 3: this load says whether it finished. DKE commits per claim, # so a run that dies part-way leaves a store that looks coherent and is # missing rows — recoverable by re-running, and until this existed not # detectable at all. prog.load_marker(sanitize(load_key or lang.tag), stamp) # The claims travel as DATA, not as statements. # # This used to emit one `remember` per claim, and loading one mid-sized # C++ library took about 1,100 of them across six submissions. A path segment naming a # variable in scope resolves to that variable's VALUE, so the subject can # come out of a row and one written `remember` serves the whole table. The # claims are identical either way; what changes is that the program stops # growing with the codebase. Measured at 3,000 subjects × 3 fields, the # row form is ~3.5x smaller as source — and the response is bounded rather # than not, because the wire caps repeated executions of ONE line while # N distinct written lines each keep their own transcript entry. # The three disciplines are FactProgram's now; these bindings keep every # call site below reading exactly as it did, which is the point — this # extraction is not allowed to change a byte of the emitted program. q = prog.q put = prog.put table = prog.table # The test that was seen to reach this function rides as the row's last # field, and an unreached function carries "". The cell is still WRITTEN # only when there is one — `code.untested` reads the ABSENCE of this cell, # so an empty string stored for every unreached function would silence the # rule for all of them instead of firing for the reached ones. The empty # string is the row's placeholder, never a stored value. # Which functions something calls, recorded on the CALLEE. The same edges # go out as `Call` rows below; this is the other end of each one. # # It is here rather than derived because a rule ranges over one kind # (SPEC §15.1 — every name it uses is the loop variable), so nothing in the # module can conclude "no Call names me" from the Call kind. Written on the # callee, the question becomes an ordinary absence on Function and # `code.uncalled` is a rule instead of a scan over every function. # # A self-call counts, because it did before: the query this replaces # matched Call.callee without excluding the caller, and this is a # performance change, not a semantic one. called_quals = {c.callee_qual for c in facts.calls} # CODE-MODULE-FAN-IN-RANKING — degree, recorded on the subject it describes # for exactly the reason `called` is: a rule ranges over ONE kind, so # "how many Calls name me?" is a question the module cannot ask of the Call # kind. Counting it in a query instead is the shape `uncalled()`'s own # comment records as having failed — work proportional to the codebase for # an answer proportional to the answer, and on a 144-function store that # was the query that overran the wire. # # DISTINCT neighbours, not edges. Fan-in answers "how many things depend on # me", and a caller that calls me three times is one dependency, not three; # counting call SITES would rank a function by how chatty its callers are. # The comparable that motivated this reports raw degree, which is the one # place this deliberately departs from it. fan_in: dict[str, set[str]] = {} fan_out: dict[str, set[str]] = {} for c in facts.calls: fan_in.setdefault(c.callee_qual, set()).add(c.caller_qual) fan_out.setdefault(c.caller_qual, set()).add(c.callee_qual) fn_rows = [] for f in sorted(facts.functions, key=lambda x: x.qual): s = ids.get(f.qual) fn_rows.append(f"({q(s)}, {q(f.name)}, {q(f.module)}, {f.lines}, " f"{f.branches}, {f.loops}, {q(facts.tested.get(f.qual, ''))}, " f"{q('1' if f.qual in called_quals else '')}, " f"{len(fan_in.get(f.qual, ()))}, " f"{len(fan_out.get(f.qual, ()))})") table("functions", fn_rows, [ put("Function.s.name", "r[1]"), put("Function.s.module", "r[2]"), put("Function.s.lines", "r[3]"), put("Function.s.branches", "r[4]"), put("Function.s.loops", "r[5]"), # A constant for the whole run, so it is written as a literal rather # than carried in every row. What it buys is a polyglot store: two # producers writing into one store are distinguishable afterwards, # which matters most where their numbers are on different scales. put("Function.s.language", q(lang.tag)), # NOT named `test`: a path segment that names an in-scope variable # resolves to that variable's VALUE, so binding `test` here would turn # the ATTRIBUTE `test` into the test's name and every function would # read as untested. Nothing rejects it — the ambiguity guard covers a # class declared in this module, and `code.Function` is not one. 'reached_by = r[6]', 'if reached_by != "":', " " + put("Function.s.test", "reached_by"), # Same write-only-when-present shape as `test` directly above, and for # the same reason: `code.uncalled` reads the ABSENCE of this cell, so a # value recorded for every function — even `False` — would silence the # rule everywhere instead of firing it where nothing calls. 'is_called = r[7]', 'if is_called != "":', " " + put("Function.s.called", "True"), # Written for EVERY function, zero included — the opposite of `called` # and `test` directly above, and the difference is which way the rule # reads them. Those two are read by their ABSENCE, so a value everywhere # would silence the rule everywhere. `callers` is read by COMPARISON, # and a function nothing calls has a fan-in of zero, which is a fact # rather than a gap. Leaving it absent would also make `hub` and # `uncalled` disagree about the same function. put("Function.s.callers", "r[8]"), put("Function.s.callees", "r[9]"), ]) cls_rows = [ f"({q(ids.get(c.qual))}, {q(c.name)}, {q(c.module)}, {c.methods})" for c in sorted(facts.classes, key=lambda x: x.qual) ] table("classes", cls_rows, [ put("Class.s.name", "r[1]"), put("Class.s.module", "r[2]"), put("Class.s.methods", "r[3]"), put("Class.s.language", q(lang.tag)), ]) # Call.caller and Call.callee hold Function SUBJECT IDS, not display # names — `uncalled()` matches Call.callee against the subjects of # Function, so a display name here would make every function look # uncalled. The `code` module states this contract. seen_edges: set[tuple[str, str]] = set() call_rows = [] for i, c in enumerate(sorted(facts.calls, key=lambda x: (x.caller_qual, x.callee_qual, x.site))): caller = ids.get(c.caller_qual) callee = ids.get(c.callee_qual) if (caller, callee) in seen_edges: continue # one edge per pair; the site records the first occurrence seen_edges.add((caller, callee)) call_rows.append(f"({q(f'e{i}')}, {q(caller)}, {q(callee)}, {q(c.site)})") table("calls", call_rows, [ put("Call.s.caller", "r[1]"), put("Call.s.callee", "r[2]"), put("Call.s.site", "r[3]"), ]) # CODE-MODULE-UNRESOLVED-CALL-SITES — the inventory of what the walker # could not see. Deduplicated on (caller, site) rather than on the callee # text: one source line is one site, and two spellings at the same line are # the same place a reader would go to look. seen_sites: set[tuple[str, str]] = set() unres_rows = [] for i, u in enumerate(sorted(facts.unresolved, key=lambda x: (x.caller_qual, x.site, x.text))): caller = ids.get(u.caller_qual) if (caller, u.site) in seen_sites: continue seen_sites.add((caller, u.site)) unres_rows.append(f"({q(f'u{i}')}, {q(caller)}, {q(u.site)}, {q(u.text)})") table("unresolved_calls", unres_rows, [ put("UnresolvedCall.s.caller", "r[1]"), put("UnresolvedCall.s.site", "r[2]"), put("UnresolvedCall.s.text", "r[3]"), ]) # CODE-MODULE-RATIONALE-FACTS. Two tables from one list, because the module # asks two different questions of it. The Rationale rows carry the TEXT — # what the author said, which only a human reads. `Function.explained` is # the boolean a RULE can range over: "is this complicated, untested thing # also unexplained?" is a question about Function, and a rule ranges over # ONE kind, so it cannot be asked of Rationale. Same shape as `called` and # for the same reason, including that it is written ONLY where a rationale # exists — the rule reads its absence. rat_rows = [] explained: set[str] = set() # `explained` is a field on Function, so only a rationale attached to a # FUNCTION may set it. A marker in a class body outside any method attaches # to the class, which is a real fact worth recording and would be a wrong # one written here. fn_quals = {f.qual for f in facts.functions} for i, r in enumerate(sorted(facts.rationale, key=lambda x: (x.subject, x.site, x.text))): subj = ids.get(r.subject) if subj is None: continue if r.subject in fn_quals: explained.add(subj) rat_rows.append(f"({q(f'w{i}')}, {q(subj)}, {q(r.marker)}, " f"{q(r.text)}, {q(r.site)})") table("rationale", rat_rows, [ put("Rationale.s.subject", "r[1]"), put("Rationale.s.marker", "r[2]"), put("Rationale.s.text", "r[3]"), put("Rationale.s.site", "r[4]"), ]) # NOT named `explained`, for the reason the `reached_by` binding above is # not named `test`: a path segment naming an in-scope variable resolves to # that variable's VALUE, so a table called `explained` turns the ATTRIBUTE # `explained` into the table itself. That is not a subtle failure — it is a # typecheck error naming `list>` where a string belongs — but # it is the same trap, two tables apart, and the comment there did not stop # it being walked into a second time. table("with_reason", [f"({q(s)},)" for s in sorted(explained)], [ put("Function.s.explained", "True"), ]) seen_imports: set[tuple[str, str]] = set() imp_rows = [] for i, im in enumerate(sorted(facts.imports, key=lambda x: (x.importer, x.imported))): if (im.importer, im.imported) in seen_imports: continue seen_imports.add((im.importer, im.imported)) # The top-level package, so `import requests.adapters` counts as using # `requests`. A dependency query matching on the full dotted name would # call a submodule-only dependency unused. imp_rows.append(f"({q(f'i{i}')}, {q(im.importer)}, {q(im.imported)}, " f"{q(import_root(im.imported))})") table("imports", imp_rows, [ put("Import.s.importer", "r[1]"), put("Import.s.imported", "r[2]"), put("Import.s.root", "r[3]"), ]) # As with `Function.test`, the optional cells stay optional: a transitive # dependency records NO `direct` cell, because absence is the honest # encoding for "nobody asked for this directly" and a stored False would # claim the project actively declined it. The row carries False so the # column has one type; the `if` decides whether it is ever written. dep_rows = [ f"({q(ids.get('dep.' + d.name))}, {q(d.name)}, {q(d.version or '')}, " f"{'True' if d.direct else 'False'}, {'True' if d.pinned else 'False'})" for d in sorted(facts.deps, key=lambda x: x.name.lower()) ] table("dependencies", dep_rows, [ put("Dependency.s.name", "r[1]"), # Not named `version`, for the same reason `reached_by` is not `test`. 'ver = r[2]', 'if ver != "":', " " + put("Dependency.s.version", "ver"), 'if r[3]:', " " + put("Dependency.s.direct", "True"), put("Dependency.s.pinned", "r[4]"), ]) return prog.render() # The three Function fields `code`'s standing rules read as an ABSENCE, and the # queries each one flattens when nothing is recorded in it. # # `code.dpy` documents the near miss on `Function.test` — record it only where a # test really reaches, because a value everywhere silences the rule everywhere. # This is the OTHER half of that trap: a producer that records the field NOWHERE # makes the rule fire EVERYWHERE, and the result looks exactly like a real # answer. It was met for real (CODE-ABSENCE-READ-QUERIES-COLLAPSE-SILENTLY-ON-A- # SILENT-PRODUCER): ingesting a tree without its tests put 592 of 592 functions # in `untested`, and `hotspot`, `critical` and `unexplained` all returned the # same 55 functions — three questions collapsed onto one answer, with nothing # saying so. # # The STORE-side detection exists: `checkup` reports `unmet . # no-values` for exactly this, including for a cell read through `absent()` # (measured). This warning is the PRODUCER-side half, which `checkup` cannot be: # checkup speaks after the facts are in a store and someone thinks to ask, while # the producer knows at write time, with the person who chose the paths still # watching. _ABSENCE_READ_FIELDS = ( ("test", "untested, critical() and unexplained()", ""), ("called", "uncalled()", ""), ("explained", "unexplained()", " In particular unexplained() cannot narrow below critical(), which is" " the whole of what it offers over it."), ) def absence_coverage(facts: Facts) -> list[tuple[str, int, int, str, str]]: """(field, recorded, total functions, queries flattened at zero, extra note). Counted from `facts` rather than from the emitted program, and counted the same way the tables write: `test` only where a test name is non-empty, `called` only where some call resolves to this function, `explained` only where a rationale attaches to a FUNCTION (one on a class is a real fact and not this one). """ fn_quals = {f.qual for f in facts.functions} total = len(fn_quals) called = {c.callee_qual for c in facts.calls} & fn_quals explained = {r.subject for r in facts.rationale} & fn_quals tested = {q for q, name in facts.tested.items() if name and q in fn_quals} have = {"test": len(tested), "called": len(called), "explained": len(explained)} return [(f, have[f], total, qs, note) for f, qs, note in _ABSENCE_READ_FIELDS] def import_root(imported: str) -> str: """The top-level name an import reaches for. `requests.adapters` uses `requests`; `boost/asio.hpp` uses `boost`. A dependency query matching on the full name would call a dependency used only through a submodule unused. Splitting on both separators keeps this one function rather than one per language, since the answer is the same idea in each. """ return imported.replace("\\", "/").split("/")[0].split(".")[0] or imported def run(lang: Language, argv: list[str] | None = None) -> int: """The shared CLI: walk, emit, report, and say plainly what was read.""" ap = argparse.ArgumentParser( prog=lang.prog, description=f"Turn a {lang.name} codebase into `code`-module facts.") ap.add_argument("paths", nargs="+", type=Path, help="files or directories to read") ap.add_argument("-o", "--out", type=Path, help="write here instead of stdout (`-` is stdout)") ap.add_argument("--exclude", action="append", default=[], help="skip paths containing this substring (repeatable)") args = ap.parse_args(argv) for p in args.paths: if not p.exists(): print(f"{lang.prog}: no such path: {p}", file=sys.stderr) return EXIT_NO_SUCH_PATH facts = lang.collect(args.paths, args.exclude) # NOTHING TO READ, stated rather than implied. # # A walker handed a tree with none of its extensions in it used to print a # row of zeros, write a program with no facts in it, and exit 0. The zeros # were there, but an exit code of 0 and no sentence saying otherwise read # as success — and the next thing the reader does is import `code` and find # every query answering []. So the two empty cases now say which one they # are, and neither exits 0. if facts.files_found == 0: wanted = ", ".join(lang.extensions) where = ", ".join(str(p) for p in args.paths) print(f"{lang.prog}: found no {wanted} file(s) under {where} — " f"this reader reads {lang.name}, and there is none here. " f"Nothing was written.", file=sys.stderr) if facts.files_excluded: print(f"{lang.prog}: {facts.files_excluded} file(s) matched " f"{wanted} but were skipped by --exclude", file=sys.stderr) return EXIT_NOTHING_TO_READ if facts.files_parsed == 0: wanted = ", ".join(lang.extensions) print(f"{lang.prog}: found {facts.files_found} {wanted} file(s) and " f"read none of them — every one failed. Nothing was written.", file=sys.stderr) for rel, why in facts.files_failed[:10]: print(f"{lang.prog}: {rel} ({why})", file=sys.stderr) return EXIT_NOTHING_TO_READ text = emit(facts, Ids(), lang) # `-o -` MEANS STDOUT, the way it does in every other tool that takes an # output path. Without this line it writes a FILE named `-`, which is both # surprising and unpleasant to undo: `rm -` parses the name as options, so # removing it takes `rm ./-` and a reader who does not know that is stuck # looking at a file they cannot delete. One reached this repository that # way and sat in the published tree's source directory for six days. if args.out and str(args.out) != "-": args.out.write_text(text, encoding="utf-8") else: sys.stdout.write(text) rate = (100.0 * facts.calls_resolved / facts.calls_seen if facts.calls_seen else 0.0) print(f"{lang.prog}: {facts.files_parsed} file(s), " f"{len(facts.functions)} function(s), {len(facts.classes)} class(es), " f"{facts.calls_resolved}/{facts.calls_seen} call sites resolved " f"({rate:.0f}%), {len(facts.unresolved)} dispatch site(s), " f"{len(facts.deps)} dependency(ies) " f"({sum(1 for d in facts.deps if d.direct)} declared)", file=sys.stderr) _t = len(facts.tested) _tpct = (100.0 * _t / len(facts.functions)) if facts.functions else 0.0 print(f"{lang.prog}: {facts.test_files} test file(s); a test was seen " f"to reach {_t}/{len(facts.functions)} function(s) ({_tpct:.0f}%) — " f"reachability over the call graph, a lower bound, not coverage", file=sys.stderr) for field, have, total, queries, note in absence_coverage(facts): if total and not have: print(f"{lang.prog}: WARNING: 0 of {total} function(s) recorded " f"`{field}`, and `code` reads that field as an absence. It " f"is therefore absent everywhere, which flattens {queries}: " f"each answers about every function instead of " f"narrowing.{note} A collapsed " f"answer is indistinguishable from a real one. The usual " f"cause is the paths you passed; `checkup` reports the same " f"thing from the store side.", file=sys.stderr) if facts.files_failed: print(f"{lang.prog}: {len(facts.files_failed)} file(s) did not " f"parse and contributed nothing", file=sys.stderr) return EXIT_OK