Repository navigation
Commit f204338
perf: pre-size fileSource reads (close the InitFrom io.ReadAll churn) (#10)
* perf: pre-size fileSource reads too (close the InitFrom churn)
Follow-up to #9. That PR made webSource and tarGzSource size-aware, but
fileSource was left returning a bare *os.File, so readAll's knownSize()
returned false for file reads and fell back to io.ReadAll's doubling growth.
geo loads its cached ~75MB MaxMind mmdb from disk via
Runner.InitFrom(FromFile(path)) at startup, so that read churned through the
75->60->48->38->30->24MB realloc staircase on every boot (confirmed via
alloc_space pprof on a lantern-box v0.0.91 box: io.ReadAll 373MB under
keepcurrent.readAll <- syncOnce <- InitFrom <- geo.FromWeb).
Make fileSource.Fetch return a sizedReadCloser carrying the stat size when
there's no preprocessor, so the read is a single pre-sized allocation. The
preprocessor path keeps the io.ReadAll fallback (a preprocessor can change
the byte count). Also close the file on the early-return/error paths, which
previously leaked the fd.
go build/vet/test green; adds TestFileSourceIsSizeAware.
* review: fix inverted fileSource modified-since check, close-couple preprocessor stream, hermetic web test
- fileSource.Fetch returned ErrUnmodified exactly when the file WAS newer
than the cutoff (and fetched when unchanged). Invert to match the
Source.Fetch contract and byteSource.Fetch.
- couple a preprocessor's returned stream's Close to the underlying *os.File
so io.NopCloser-style preprocessors don't leak the descriptor.
- add TestFileSourceUnmodified covering the ErrUnmodified path.
- make TestUpdateFromWeb hermetic via httptest; the live httpbin.org
dependency intermittently hung and timed out CI.
---------
Co-authored-by: Ilya Yakelzon <reflog@getlantern.org>1 parent 0364abd commit f204338
3 files changed
Lines changed: 93 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
| 12 | + | |
| 13 | + | |
11 | 14 | | |
12 | 15 | | |
13 | 16 | | |
| |||
103 | 106 | | |
104 | 107 | | |
105 | 108 | | |
106 | | - | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
107 | 116 | | |
108 | 117 | | |
109 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
10 | 13 | | |
11 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
12 | 59 | | |
13 | 60 | | |
14 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
160 | 176 | | |
161 | 177 | | |
162 | 178 | | |
| |||
174 | 190 | | |
175 | 191 | | |
176 | 192 | | |
| 193 | + | |
177 | 194 | | |
178 | 195 | | |
179 | | - | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
180 | 201 | | |
181 | 202 | | |
182 | | - | |
183 | | - | |
184 | 203 | | |
185 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
186 | 207 | | |
| 208 | + | |
187 | 209 | | |
188 | 210 | | |
189 | | - | |
190 | | - | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
191 | 221 | | |
0 commit comments