Repository navigation
Commit 637dfa7
committed
perf(tools): derive the changed-line span from bytes, not from per-line slices
Reported by @jatmn.
changedLineSpan split both file versions into []string, so it allocated one
string header per line for EACH version — work that scales with the size of the
file rather than the size of the edit. countLines then split a third time just to
take a length. All of it ran before RecordEdit had checked whether there was a
tracked observation to update at all.
He measured 32,036,640 bytes for two 2 MB versions of a million short lines, and
extrapolated roughly 1.6 GB of headers for a 100 MB file before counting the
content, the updated bytes and the hashes already live beside them.
The memory was not the sharp end. edit_file writes the updated bytes BEFORE
calling RecordEdit, so an out-of-memory kill lands after the user's file has
changed and before the tracker baseline catches up: the file and the record of it
disagree, and nothing says so. His instruction was to fix the allocation source
rather than add a size guard, and that a guard — if used at all — would have to
reject before os.WriteFile rather than during post-write re-baselining. Taken as
written: there is no new limit, and the same edits are accepted as before.
Two identical byte prefixes share every line that ends inside them, so counting
newlines in the common prefix counts the unchanged leading lines directly. Same
for the suffix, with one rule that the line-array version got for free: the line
a shared tail STARTS in is only shared when the tail begins on a line boundary in
both versions, or its opening bytes differ and it is a changed line that merely
ends the same way.
Measured on his own scenario, two 2 MB versions:
changedLineSpan 32,036,640 bytes -> 0 allocations
countLines per-line slice -> 0 allocations
## The equivalence is tested, not asserted
A rewrite of a diff primitive is only safe if it is behaviour-preserving, so the
replaced implementation is kept in the test file as an oracle and both are asked
the same questions. That caught 206 mismatches in my first attempt at the byte
scan — empty versions, and trailing lines that are only partially shared. Neither
appears in the hand-written cases, and I would have shipped it.
Two regressions: the oracle comparison, and an absolute allocation ceiling rather
than a ratio, so a future rewrite that reintroduces per-line slices fails rather
than merely getting slower. Mutation: restoring strings.Split in either function
allocates 3.2 MB and 3.6 MB against a 4 KB bound.
splitLinesForTracking lost its last caller and is removed — the same obsolete-
helper lint failure that broke Windows CI on #911, caught here by running
make lint-static before pushing rather than after. 0 issues.
Rebased onto ad34dc8, 0 behind. go test -race ./internal/tools/: clean.
Pre-existing here and on main: TestRunDoctorFormatsRedactedProviderDiagnostics
and TestRunDoctorConnectivityProbesProvider exit 3 in this environment.
Origin-Session: local-c962d7 | Claude Code | 17 prompts
Origin-Snapshot: a599377c09e01 parent 6d59074 commit 637dfa7
2 files changed
Lines changed: 189 additions & 26 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
264 | 265 | | |
265 | 266 | | |
266 | 267 | | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
267 | 286 | | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
280 | 341 | | |
281 | 342 | | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
282 | 350 | | |
283 | 351 | | |
284 | 352 | | |
| |||
294 | 362 | | |
295 | 363 | | |
296 | 364 | | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | | - | |
303 | | - | |
304 | 365 | | |
305 | 366 | | |
306 | 367 | | |
| |||
311 | 372 | | |
312 | 373 | | |
313 | 374 | | |
314 | | - | |
315 | | - | |
316 | | - | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
317 | 379 | | |
318 | | - | |
319 | | - | |
320 | | - | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
321 | 401 | | |
322 | | - | |
| 402 | + | |
323 | 403 | | |
324 | 404 | | |
325 | 405 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
153 | 154 | | |
154 | 155 | | |
155 | 156 | | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
0 commit comments