CPU load steadily increasing

[eth2353]

System information

Geth version: 1.16.7-stable @ b9f3a3d964ed3d31e710ec7dd66da9181477ecb2 (official amd64 Docker image)
CL client & version: Prysm 7.0.0
OS & Version: Linux
Commit hash : (if develop)

Expected behaviour

CPU usage remains stable.

Actual behaviour

CPU load is steadily increasing:

Steps to reproduce the behaviour

Unclear. We have two other nodes running the same version that are unaffected.

On the affected node, we updated from Geth v1.15.10 to v1.16.7, on December 2nd. The issue seems to have started about a day later, on December 3rd (multiple hours before the Fusaka upgrade so probably unrelated to the upgrade itself).

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.

Logs

Recent logs

INFO [12-08|10:58:24.511] Chain head was updated                   number=23,967,639 hash=f46b7c..34d8d6 root=ad5734..c8a8ed elapsed="624.365µs"
WARN [12-08|10:58:25.086] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:58:25.256] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:58:25.810] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:58:37.131] Imported new potential chain segment     number=23,967,640 hash=849967..cb706f blocks=1 txs=208  mgas=59.998 elapsed=367.116ms    mgasps=163.430  triediffs=198.59MiB triedirty=252.66MiB
INFO [12-08|10:58:37.307] Chain head was updated                   number=23,967,640 hash=849967..cb706f root=c46854..04fc01 elapsed=23.114574ms
WARN [12-08|10:58:37.833] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:58:38.006] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:58:38.608] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:58:48.771] Imported new potential chain segment     number=23,967,641 hash=c84f53..38c4b2 blocks=1 txs=313  mgas=59.563 elapsed=286.072ms    mgasps=208.209  triediffs=199.63MiB triedirty=253.07MiB
INFO [12-08|10:58:48.945] Chain head was updated                   number=23,967,641 hash=c84f53..38c4b2 root=fded16..cb9c1b elapsed=13.684429ms
WARN [12-08|10:58:49.507] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:58:49.671] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:58:50.166] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:59:00.783] Imported new potential chain segment     number=23,967,642 hash=392525..33f506 blocks=1 txs=123  mgas=7.169  elapsed=52.113ms     mgasps=137.567  triediffs=198.25MiB triedirty=253.51MiB
INFO [12-08|10:59:01.285] Chain head was updated                   number=23,967,642 hash=392525..33f506 root=6d6b97..ace335 elapsed=1.484048ms
WARN [12-08|10:59:01.848] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:59:02.021] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:59:02.539] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:59:13.069] Imported new potential chain segment     number=23,967,643 hash=a325d4..b8f71c blocks=1 txs=283  mgas=56.850 elapsed=239.692ms    mgasps=237.179  triediffs=199.55MiB triedirty=253.73MiB
INFO [12-08|10:59:13.247] Chain head was updated                   number=23,967,643 hash=a325d4..b8f71c root=e5a6f8..87754c elapsed=16.097421ms
WARN [12-08|10:59:13.807] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:59:13.970] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:59:14.460] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:59:15.952] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,643 processed=1         remaining=0          elapsed=1.980s
INFO [12-08|10:59:15.952] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,643 processed=1         elapsed=1.980s
INFO [12-08|10:59:25.112] Imported new potential chain segment     number=23,967,644 hash=9e7205..2114b6 blocks=1 txs=164  mgas=12.851 elapsed=82.604ms     mgasps=155.575  triediffs=199.52MiB triedirty=253.93MiB
INFO [12-08|10:59:25.346] Chain head was updated                   number=23,967,644 hash=9e7205..2114b6 root=d17dfc..db9467 elapsed=2.003288ms
WARN [12-08|10:59:25.914] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:59:26.076] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:59:26.555] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:59:37.314] Imported new potential chain segment     number=23,967,645 hash=302999..4dd775 blocks=1 txs=81   mgas=7.546  elapsed=114.349ms    mgasps=65.988   triediffs=198.31MiB triedirty=254.18MiB
INFO [12-08|10:59:37.517] Chain head was updated                   number=23,967,645 hash=302999..4dd775 root=37864b..d5b973 elapsed="755.25µs"
WARN [12-08|10:59:38.082] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:59:38.242] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:59:38.763] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|10:59:48.722] Imported new potential chain segment     number=23,967,646 hash=4d53fa..e5d770 blocks=1 txs=184  mgas=24.716 elapsed=163.236ms    mgasps=151.411  triediffs=198.93MiB triedirty=254.40MiB
INFO [12-08|10:59:48.886] Chain head was updated                   number=23,967,646 hash=4d53fa..e5d770 root=3c730d..352188 elapsed=3.137784ms
WARN [12-08|10:59:49.452] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|10:59:49.617] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|10:59:50.108] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:00:00.565] Imported new potential chain segment     number=23,967,647 hash=866b70..500fef blocks=1 txs=207  mgas=59.997 elapsed=305.707ms    mgasps=196.257  triediffs=199.42MiB triedirty=254.65MiB
INFO [12-08|11:00:00.739] Chain head was updated                   number=23,967,647 hash=866b70..500fef root=a54f9a..19d555 elapsed=18.950235ms
WARN [12-08|11:00:01.288] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:00:01.448] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:00:02.050] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:00:13.669] Imported new potential chain segment     number=23,967,648 hash=e71394..dfcbc9 blocks=1 txs=332  mgas=21.140 elapsed=189.094ms    mgasps=111.799  triediffs=200.03MiB triedirty=255.04MiB
INFO [12-08|11:00:13.858] Chain head was updated                   number=23,967,648 hash=e71394..dfcbc9 root=be6fab..7e5539 elapsed=3.40876ms
WARN [12-08|11:00:14.416] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:00:14.600] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:00:15.145] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:00:25.350] Imported new potential chain segment     number=23,967,649 hash=27a708..ec06fd blocks=1 txs=190  mgas=59.985 elapsed=379.045ms    mgasps=158.252  triediffs=200.94MiB triedirty=255.11MiB
INFO [12-08|11:00:25.557] Chain head was updated                   number=23,967,649 hash=27a708..ec06fd root=ba7da8..f1c220 elapsed=21.60134ms
WARN [12-08|11:00:26.118] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:00:26.277] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:00:26.780] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:00:38.388] Imported new potential chain segment     number=23,967,650 hash=d066d7..1ce827 blocks=1 txs=263  mgas=22.159 elapsed=185.499ms    mgasps=119.456  triediffs=200.19MiB triedirty=255.58MiB
INFO [12-08|11:00:38.547] Chain head was updated                   number=23,967,650 hash=d066d7..1ce827 root=400fca..1d67c7 elapsed=3.010628ms
WARN [12-08|11:00:39.108] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:00:39.268] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:00:39.766] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:00:48.957] Imported new potential chain segment     number=23,967,651 hash=d34768..f9cb28 blocks=1 txs=232  mgas=20.009 elapsed=479.011ms    mgasps=41.771   triediffs=200.39MiB triedirty=255.78MiB
INFO [12-08|11:00:49.241] Chain head was updated                   number=23,967,651 hash=d34768..f9cb28 root=f00d0c..6ed731 elapsed=3.64448ms
WARN [12-08|11:00:49.831] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:00:49.998] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:00:50.529] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:01:01.670] Imported new potential chain segment     number=23,967,652 hash=e3299c..36f4dc blocks=1 txs=299  mgas=22.627 elapsed=149.918ms    mgasps=150.928  triediffs=201.18MiB triedirty=255.97MiB
INFO [12-08|11:01:01.670] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=27h20m9s   timestamp=1,765,290,071
INFO [12-08|11:01:01.892] Chain head was updated                   number=23,967,652 hash=e3299c..36f4dc root=d4db75..8c059c elapsed=6.524061ms
WARN [12-08|11:01:02.475] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:01:02.661] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:01:03.305] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:01:12.751] Imported new potential chain segment     number=23,967,653 hash=fc545f..f66302 blocks=1 txs=267  mgas=28.641 elapsed=182.095ms    mgasps=157.286  triediffs=200.98MiB triedirty=0.00B
INFO [12-08|11:01:12.956] Chain head was updated                   number=23,967,653 hash=fc545f..f66302 root=56664b..3f74eb elapsed=14.444158ms
WARN [12-08|11:01:13.489] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:01:13.696] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:01:14.281] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:01:17.248] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,653 processed=1         remaining=0          elapsed=2.963s
INFO [12-08|11:01:17.249] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,653 processed=1         elapsed=2.963s
INFO [12-08|11:01:25.734] Imported new potential chain segment     number=23,967,654 hash=f2a869..8bcc00 blocks=1 txs=123  mgas=42.292 elapsed=275.105ms    mgasps=153.730  triediffs=200.51MiB triedirty=1.66MiB
INFO [12-08|11:01:26.442] Chain head was updated                   number=23,967,654 hash=f2a869..8bcc00 root=b71884..2555f0 elapsed=15.360839ms
WARN [12-08|11:01:27.013] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:01:27.206] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:01:27.721] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:01:36.572] Imported new potential chain segment     number=23,967,655 hash=b41dcf..096a20 blocks=1 txs=206  mgas=20.914 elapsed=223.725ms    mgasps=93.479   triediffs=200.68MiB triedirty=3.05MiB
INFO [12-08|11:01:36.971] Chain head was updated                   number=23,967,655 hash=b41dcf..096a20 root=3c4c15..2522e7 elapsed=5.192487ms
WARN [12-08|11:01:37.504] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:01:37.689] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:01:38.243] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:01:48.945] Imported new potential chain segment     number=23,967,656 hash=8ca12a..44a38e blocks=1 txs=200  mgas=47.385 elapsed=520.432ms    mgasps=91.050   triediffs=201.37MiB triedirty=3.91MiB
INFO [12-08|11:01:49.116] Chain head was updated                   number=23,967,656 hash=8ca12a..44a38e root=069631..8ac645 elapsed=14.995417ms
WARN [12-08|11:01:49.679] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:01:49.850] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:01:50.333] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:02:00.408] Imported new potential chain segment     number=23,967,657 hash=35630e..d68607 blocks=1 txs=72   mgas=4.201  elapsed=69.491ms     mgasps=60.455   triediffs=200.55MiB triedirty=4.86MiB
INFO [12-08|11:02:00.784] Chain head was updated                   number=23,967,657 hash=35630e..d68607 root=bfc6ce..ea5538 elapsed="897.895µs"
WARN [12-08|11:02:01.367] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:02:01.529] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:02:02.027] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:02:02.034] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,657 processed=1         remaining=0          elapsed=1.249s
INFO [12-08|11:02:02.034] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,657 processed=1         elapsed=1.249s
INFO [12-08|11:02:12.820] Imported new potential chain segment     number=23,967,658 hash=b38289..02d983 blocks=1 txs=297  mgas=54.002 elapsed=321.958ms    mgasps=167.730  triediffs=201.52MiB triedirty=5.92MiB
INFO [12-08|11:02:13.000] Chain head was updated                   number=23,967,658 hash=b38289..02d983 root=fe84eb..29b4da elapsed=15.468539ms
WARN [12-08|11:02:13.501] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:02:13.663] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:02:14.151] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:02:23.280] Imported new potential chain segment     number=23,967,659 hash=9b9c13..e608f9 blocks=1 txs=59   mgas=3.141  elapsed=56.140ms     mgasps=55.956   triediffs=200.32MiB triedirty=6.92MiB
INFO [12-08|11:02:23.689] Chain head was updated                   number=23,967,659 hash=9b9c13..e608f9 root=16e1be..3a0c00 elapsed="532.634µs"
WARN [12-08|11:02:24.167] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:02:24.327] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:02:24.882] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:02:37.811] Imported new potential chain segment     number=23,967,660 hash=d76e55..b0dc24 blocks=1 txs=151  mgas=59.932 elapsed=329.112ms    mgasps=182.101  triediffs=200.14MiB triedirty=7.92MiB
INFO [12-08|11:02:37.989] Chain head was updated                   number=23,967,660 hash=d76e55..b0dc24 root=d71b20..985763 elapsed=19.846737ms
WARN [12-08|11:02:38.544] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:02:38.711] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:02:39.256] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:02:40.528] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,660 processed=1         remaining=0          elapsed=2.538s
INFO [12-08|11:02:40.528] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,660 processed=1         elapsed=2.538s
INFO [12-08|11:03:01.447] Imported new potential chain segment     number=23,967,661 hash=32e2cb..54f309 blocks=1 txs=181  mgas=9.657  elapsed=85.504ms     mgasps=112.938  triediffs=200.99MiB triedirty=8.06MiB
INFO [12-08|11:03:01.804] Chain head was updated                   number=23,967,661 hash=32e2cb..54f309 root=fdbcf8..09914c elapsed=1.250032ms
WARN [12-08|11:03:02.358] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:03:02.525] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:03:03.105] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:03:13.709] Imported new potential chain segment     number=23,967,662 hash=5785fa..bab4d2 blocks=1 txs=287  mgas=22.570 elapsed=132.658ms    mgasps=170.133  triediffs=201.30MiB triedirty=9.22MiB
INFO [12-08|11:03:13.872] Chain head was updated                   number=23,967,662 hash=5785fa..bab4d2 root=8d2255..a0fc5c elapsed=3.063265ms
WARN [12-08|11:03:14.431] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:03:14.593] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:03:15.071] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:03:25.558] Imported new potential chain segment     number=23,967,663 hash=bbc6aa..27583b blocks=1 txs=205  mgas=59.376 elapsed=690.310ms    mgasps=86.013   triediffs=201.05MiB triedirty=10.32MiB
INFO [12-08|11:03:25.737] Chain head was updated                   number=23,967,663 hash=bbc6aa..27583b root=f120f4..92ea30 elapsed=18.46557ms
WARN [12-08|11:03:26.305] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:03:26.473] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:03:27.008] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:03:31.980] Imported new potential chain segment     number=23,967,661 hash=aea030..17274e blocks=1 txs=3    mgas=0.232  elapsed=8.752ms      mgasps=26.562   triediffs=201.13MiB triedirty=10.32MiB
INFO [12-08|11:03:36.858] Imported new potential chain segment     number=23,967,664 hash=8d4aad..0868b8 blocks=1 txs=394  mgas=36.958 elapsed=205.188ms    mgasps=180.115  triediffs=203.34MiB triedirty=10.91MiB
INFO [12-08|11:03:37.076] Chain head was updated                   number=23,967,664 hash=8d4aad..0868b8 root=7588c4..95caa3 elapsed=7.589701ms
WARN [12-08|11:03:37.562] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:03:37.727] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:03:38.293] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:03:48.120] Imported new potential chain segment     number=23,967,665 hash=0c2b9e..193584 blocks=1 txs=126  mgas=15.563 elapsed=113.111ms    mgasps=137.588  triediffs=202.87MiB triedirty=11.71MiB
INFO [12-08|11:03:48.289] Chain head was updated                   number=23,967,665 hash=0c2b9e..193584 root=abb3db..087263 elapsed=1.615254ms
WARN [12-08|11:03:48.875] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:03:49.049] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:03:49.569] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:04:00.498] Imported new potential chain segment     number=23,967,666 hash=6c5f6b..9228b6 blocks=1 txs=237  mgas=31.589 elapsed=133.720ms    mgasps=236.232  triediffs=203.73MiB triedirty=12.28MiB
INFO [12-08|11:04:00.663] Chain head was updated                   number=23,967,666 hash=6c5f6b..9228b6 root=6165cf..06ebe4 elapsed=7.555767ms
WARN [12-08|11:04:01.229] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:04:01.392] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:04:01.911] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:04:13.804] Imported new potential chain segment     number=23,967,667 hash=e93a57..bd8274 blocks=1 txs=206  mgas=23.315 elapsed=128.782ms    mgasps=181.040  triediffs=204.51MiB triedirty=12.78MiB
INFO [12-08|11:04:13.804] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=27h16m57s  timestamp=1,765,290,071
INFO [12-08|11:04:14.147] Chain head was updated                   number=23,967,667 hash=e93a57..bd8274 root=693971..5d8105 elapsed=6.613458ms
WARN [12-08|11:04:14.709] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:04:14.871] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:04:15.357] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:04:24.667] Imported new potential chain segment     number=23,967,668 hash=d79365..e74f23 blocks=1 txs=187  mgas=58.591 elapsed=307.935ms    mgasps=190.270  triediffs=205.08MiB triedirty=13.13MiB
INFO [12-08|11:04:24.844] Chain head was updated                   number=23,967,668 hash=d79365..e74f23 root=d74a6f..3aa986 elapsed=19.305747ms
WARN [12-08|11:04:25.418] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:04:25.601] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:04:26.130] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:04:27.689] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,668 processed=1         remaining=0          elapsed=2.086s
INFO [12-08|11:04:27.689] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,668 processed=1         elapsed=2.086s
INFO [12-08|11:04:36.470] Imported new potential chain segment     number=23,967,669 hash=7289a9..9ca05a blocks=1 txs=157  mgas=19.012 elapsed=160.217ms    mgasps=118.664  triediffs=204.81MiB triedirty=14.21MiB
INFO [12-08|11:04:36.664] Chain head was updated                   number=23,967,669 hash=7289a9..9ca05a root=dd9506..792ef3 elapsed=2.557651ms
WARN [12-08|11:04:37.235] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:04:37.397] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:04:37.916] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:04:48.727] Imported new potential chain segment     number=23,967,670 hash=1ab007..809658 blocks=1 txs=76   mgas=10.880 elapsed=128.349ms    mgasps=84.771   triediffs=204.52MiB triedirty=15.08MiB
INFO [12-08|11:04:48.980] Chain head was updated                   number=23,967,670 hash=1ab007..809658 root=cecdb3..4f92ae elapsed=1.88111ms
WARN [12-08|11:04:49.538] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:04:49.715] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:04:50.225] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:00.740] Imported new potential chain segment     number=23,967,671 hash=e07601..e72bd5 blocks=1 txs=240  mgas=59.921 elapsed=390.287ms    mgasps=153.530  triediffs=205.98MiB triedirty=15.37MiB
INFO [12-08|11:05:01.026] Chain head was updated                   number=23,967,671 hash=e07601..e72bd5 root=bd8d31..3849ab elapsed=18.601003ms
WARN [12-08|11:05:01.628] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:05:01.814] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:05:02.326] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:02.351] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,671 processed=1         remaining=0          elapsed=1.324s
INFO [12-08|11:05:02.351] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,671 processed=1         elapsed=1.324s
INFO [12-08|11:05:12.557] Imported new potential chain segment     number=23,967,672 hash=43efe5..2557fa blocks=1 txs=163  mgas=21.848 elapsed=270.178ms    mgasps=80.866   triediffs=206.14MiB triedirty=16.11MiB
INFO [12-08|11:05:12.725] Chain head was updated                   number=23,967,672 hash=43efe5..2557fa root=43cc14..9dbf0d elapsed=3.183158ms
WARN [12-08|11:05:13.284] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:05:13.449] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:05:13.943] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:25.575] Imported new potential chain segment     number=23,967,673 hash=8f870d..d477cf blocks=1 txs=188  mgas=11.034 elapsed=98.781ms     mgasps=111.699  triediffs=205.18MiB triedirty=17.17MiB
INFO [12-08|11:05:25.765] Chain head was updated                   number=23,967,673 hash=8f870d..d477cf root=99e069..cf4ab7 elapsed=1.617398ms
WARN [12-08|11:05:26.321] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:05:26.503] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:05:26.993] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:37.326] Imported new potential chain segment     number=23,967,674 hash=6d8ed4..1715c9 blocks=1 txs=171  mgas=58.586 elapsed=309.036ms    mgasps=189.578  triediffs=205.38MiB triedirty=17.66MiB
INFO [12-08|11:05:37.515] Chain head was updated                   number=23,967,674 hash=6d8ed4..1715c9 root=77932a..63cd5f elapsed=32.322693ms
WARN [12-08|11:05:38.073] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:05:38.243] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:05:38.754] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:48.582] Imported new potential chain segment     number=23,967,675 hash=9ddf90..e6c568 blocks=1 txs=236  mgas=26.025 elapsed=225.439ms    mgasps=115.440  triediffs=207.22MiB triedirty=17.86MiB
INFO [12-08|11:05:48.755] Chain head was updated                   number=23,967,675 hash=9ddf90..e6c568 root=1e091d..67c649 elapsed=4.86657ms
WARN [12-08|11:05:49.316] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:05:49.498] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:05:49.996] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:05:50.012] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,675 processed=1         remaining=0          elapsed=1.255s
INFO [12-08|11:05:50.012] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,675 processed=1         elapsed=1.256s
INFO [12-08|11:06:01.228] Imported new potential chain segment     number=23,967,676 hash=8c1c4a..748976 blocks=1 txs=164  mgas=14.563 elapsed=87.010ms     mgasps=167.373  triediffs=205.48MiB triedirty=19.23MiB
INFO [12-08|11:06:01.426] Chain head was updated                   number=23,967,676 hash=8c1c4a..748976 root=949a92..ae3c85 elapsed=3.23735ms
WARN [12-08|11:06:01.926] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:06:02.091] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:06:02.583] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:06:02.620] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,676 processed=1         remaining=0          elapsed=1.193s
INFO [12-08|11:06:02.621] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,676 processed=1         elapsed=1.193s
INFO [12-08|11:06:12.651] Imported new potential chain segment     number=23,967,677 hash=9fc60a..0309aa blocks=1 txs=120  mgas=57.362 elapsed=273.418ms    mgasps=209.794  triediffs=205.23MiB triedirty=19.83MiB
INFO [12-08|11:06:12.819] Chain head was updated                   number=23,967,677 hash=9fc60a..0309aa root=cd7933..7cea3f elapsed=18.928585ms
WARN [12-08|11:06:13.321] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:06:13.491] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:06:14.005] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:06:15.622] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,677 processed=1         remaining=0          elapsed=1.611s
INFO [12-08|11:06:15.622] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,677 processed=1         elapsed=1.612s
INFO [12-08|11:06:24.807] Imported new potential chain segment     number=23,967,678 hash=681ed8..6fdffe blocks=1 txs=231  mgas=27.716 elapsed=158.630ms    mgasps=174.719  triediffs=206.27MiB triedirty=20.34MiB
INFO [12-08|11:06:24.976] Chain head was updated                   number=23,967,678 hash=681ed8..6fdffe root=e42890..3311b0 elapsed=3.923439ms
WARN [12-08|11:06:25.542] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:06:25.706] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:06:26.214] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:06:36.655] Imported new potential chain segment     number=23,967,679 hash=141ecd..699b60 blocks=1 txs=180  mgas=13.675 elapsed=80.909ms     mgasps=169.011  triediffs=206.06MiB triedirty=21.07MiB
INFO [12-08|11:06:36.912] Chain head was updated                   number=23,967,679 hash=141ecd..699b60 root=a777ba..e52144 elapsed=2.20871ms
WARN [12-08|11:06:37.470] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:06:37.631] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:06:38.122] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:06:48.506] Imported new potential chain segment     number=23,967,680 hash=c1c335..10ab1d blocks=1 txs=222  mgas=16.598 elapsed=87.403ms     mgasps=189.904  triediffs=206.13MiB triedirty=21.72MiB
INFO [12-08|11:06:48.770] Chain head was updated                   number=23,967,680 hash=c1c335..10ab1d root=94a1ce..795dfd elapsed=3.775513ms
WARN [12-08|11:06:49.254] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:06:49.414] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:06:49.975] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:01.695] Imported new potential chain segment     number=23,967,681 hash=037a27..1183f8 blocks=1 txs=201  mgas=16.123 elapsed=101.481ms    mgasps=158.874  triediffs=206.31MiB triedirty=22.32MiB
INFO [12-08|11:07:01.966] Chain head was updated                   number=23,967,681 hash=037a27..1183f8 root=652268..2f079c elapsed=3.133666ms
WARN [12-08|11:07:02.497] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:07:02.685] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:07:03.262] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:12.788] Imported new potential chain segment     number=23,967,682 hash=584cef..76cb8c blocks=1 txs=164  mgas=59.979 elapsed=363.009ms    mgasps=165.228  triediffs=205.76MiB triedirty=22.96MiB
INFO [12-08|11:07:13.054] Chain head was updated                   number=23,967,682 hash=584cef..76cb8c root=e80901..fe67c3 elapsed=22.856672ms
WARN [12-08|11:07:13.542] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:07:13.722] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:07:14.301] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:24.809] Imported new potential chain segment     number=23,967,683 hash=84e054..05a2e8 blocks=1 txs=226  mgas=31.852 elapsed=261.133ms    mgasps=121.978  triediffs=206.44MiB triedirty=23.49MiB
INFO [12-08|11:07:24.809] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=27h13m46s  timestamp=1,765,290,071
INFO [12-08|11:07:24.983] Chain head was updated                   number=23,967,683 hash=84e054..05a2e8 root=3dcf10..65487c elapsed=5.168052ms
WARN [12-08|11:07:25.558] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:07:25.742] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:07:26.287] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:37.692] Imported new potential chain segment     number=23,967,684 hash=8b0d99..94c5f3 blocks=1 txs=172  mgas=15.986 elapsed=115.640ms    mgasps=138.236  triediffs=206.96MiB triedirty=23.81MiB
INFO [12-08|11:07:37.974] Chain head was updated                   number=23,967,684 hash=8b0d99..94c5f3 root=5553ef..fdd37e elapsed=4.516346ms
WARN [12-08|11:07:38.527] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:07:38.687] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:07:39.221] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:49.705] Imported new potential chain segment     number=23,967,685 hash=45d972..f903a7 blocks=1 txs=169  mgas=59.537 elapsed=307.130ms    mgasps=193.849  triediffs=206.87MiB triedirty=24.42MiB
INFO [12-08|11:07:49.888] Chain head was updated                   number=23,967,685 hash=45d972..f903a7 root=85d83d..78e851 elapsed=20.461104ms
WARN [12-08|11:07:50.465] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:07:50.639] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:07:51.145] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:07:52.754] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,685 processed=1         remaining=0          elapsed=2.114s
INFO [12-08|11:07:52.754] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,685 processed=1         elapsed=2.114s
INFO [12-08|11:08:00.363] Imported new potential chain segment     number=23,967,686 hash=ab8484..f701f5 blocks=1 txs=168  mgas=15.760 elapsed=123.435ms    mgasps=127.675  triediffs=206.86MiB triedirty=25.03MiB
INFO [12-08|11:08:00.669] Chain head was updated                   number=23,967,686 hash=ab8484..f701f5 root=1b824d..afb5b3 elapsed=2.093246ms
WARN [12-08|11:08:01.248] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:08:01.432] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:08:01.956] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:08:12.741] Imported new potential chain segment     number=23,967,687 hash=f6b110..47fa6b blocks=1 txs=190  mgas=17.660 elapsed=115.934ms    mgasps=152.324  triediffs=207.43MiB triedirty=25.41MiB
INFO [12-08|11:08:12.960] Chain head was updated                   number=23,967,687 hash=f6b110..47fa6b root=79fafb..0ebe95 elapsed=3.651162ms
WARN [12-08|11:08:13.448] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:08:13.621] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:08:14.191] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:08:37.359] Imported new potential chain segment     number=23,967,688 hash=238690..d91b4b blocks=1 txs=197  mgas=27.525 elapsed=187.928ms    mgasps=146.464  triediffs=207.16MiB triedirty=26.21MiB
INFO [12-08|11:08:37.601] Chain head was updated                   number=23,967,688 hash=238690..d91b4b root=351328..ca9f3c elapsed=6.825845ms
WARN [12-08|11:08:38.181] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:08:38.346] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:08:38.886] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:08:48.672] Imported new potential chain segment     number=23,967,689 hash=b77b5b..67e855 blocks=1 txs=233  mgas=57.358 elapsed=311.278ms    mgasps=184.267  triediffs=208.61MiB triedirty=26.33MiB
INFO [12-08|11:08:48.869] Chain head was updated                   number=23,967,689 hash=b77b5b..67e855 root=3e7e33..64ae10 elapsed=17.462349ms
WARN [12-08|11:08:49.356] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:08:49.538] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:08:50.103] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:00.434] Imported new potential chain segment     number=23,967,690 hash=d33752..340c84 blocks=1 txs=173  mgas=21.737 elapsed=133.699ms    mgasps=162.579  triediffs=208.19MiB triedirty=27.08MiB
INFO [12-08|11:09:00.614] Chain head was updated                   number=23,967,690 hash=d33752..340c84 root=4978e0..e05756 elapsed=2.408152ms
WARN [12-08|11:09:01.199] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:09:01.378] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:09:01.879] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:01.893] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,690 processed=1         remaining=0          elapsed=1.278s
INFO [12-08|11:09:01.893] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,690 processed=1         elapsed=1.278s
INFO [12-08|11:09:12.701] Imported new potential chain segment     number=23,967,691 hash=ef4b40..e69dee blocks=1 txs=210  mgas=27.316 elapsed=248.124ms    mgasps=110.088  triediffs=208.37MiB triedirty=27.78MiB
INFO [12-08|11:09:13.160] Chain head was updated                   number=23,967,691 hash=ef4b40..e69dee root=d3e9f6..cc3653 elapsed=4.971525ms
WARN [12-08|11:09:13.713] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:09:13.895] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:09:14.511] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:24.344] Imported new potential chain segment     number=23,967,692 hash=0f8585..71210e blocks=1 txs=169  mgas=13.982 elapsed=80.647ms     mgasps=173.375  triediffs=208.06MiB triedirty=28.45MiB
INFO [12-08|11:09:24.722] Chain head was updated                   number=23,967,692 hash=0f8585..71210e root=796f36..c79484 elapsed=2.721408ms
WARN [12-08|11:09:25.320] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:09:25.483] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:09:26.029] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:36.917] Imported new potential chain segment     number=23,967,693 hash=50d79d..cda5b2 blocks=1 txs=151  mgas=22.776 elapsed=105.636ms    mgasps=215.609  triediffs=207.59MiB triedirty=28.94MiB
INFO [12-08|11:09:37.076] Chain head was updated                   number=23,967,693 hash=50d79d..cda5b2 root=212d64..ab8285 elapsed=6.995661ms
WARN [12-08|11:09:37.583] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:09:37.754] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:09:38.362] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:39.973] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,693 processed=1         remaining=0          elapsed=1.609s
INFO [12-08|11:09:39.973] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,693 processed=1         elapsed=1.610s
INFO [12-08|11:09:48.416] Imported new potential chain segment     number=23,967,694 hash=1aa653..f95ca4 blocks=1 txs=160  mgas=59.982 elapsed=349.642ms    mgasps=171.551  triediffs=207.40MiB triedirty=29.39MiB
INFO [12-08|11:09:48.615] Chain head was updated                   number=23,967,694 hash=1aa653..f95ca4 root=07ef88..53a385 elapsed=23.262789ms
WARN [12-08|11:09:49.199] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:09:49.359] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:09:49.844] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:09:49.871] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,694 processed=1         remaining=0          elapsed=1.254s
INFO [12-08|11:09:49.872] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,694 processed=1         elapsed=1.255s
INFO [12-08|11:10:01.683] Imported new potential chain segment     number=23,967,695 hash=f81c7d..7c7133 blocks=1 txs=198  mgas=21.526 elapsed=135.716ms    mgasps=158.612  triediffs=207.93MiB triedirty=29.91MiB
INFO [12-08|11:10:01.964] Chain head was updated                   number=23,967,695 hash=f81c7d..7c7133 root=e51f43..6992fe elapsed=5.813076ms
WARN [12-08|11:10:02.467] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:10:02.647] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:10:03.202] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:10:12.512] Imported new potential chain segment     number=23,967,696 hash=3e3a64..671194 blocks=1 txs=193  mgas=17.060 elapsed=62.698ms     mgasps=272.086  triediffs=209.12MiB triedirty=30.08MiB
INFO [12-08|11:10:12.779] Chain head was updated                   number=23,967,696 hash=3e3a64..671194 root=b9fb31..95d636 elapsed=2.986722ms
WARN [12-08|11:10:13.343] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:10:13.511] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:10:14.023] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:10:26.344] Imported new potential chain segment     number=23,967,697 hash=03ee9d..f21be6 blocks=1 txs=131  mgas=59.935 elapsed=319.013ms    mgasps=187.875  triediffs=208.14MiB triedirty=30.86MiB
INFO [12-08|11:10:26.344] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=27h10m45s  timestamp=1,765,290,071
INFO [12-08|11:10:26.536] Chain head was updated                   number=23,967,697 hash=03ee9d..f21be6 root=9025e6..45692d elapsed=21.302584ms
WARN [12-08|11:10:27.098] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:10:27.287] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:10:27.828] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:10:36.490] Imported new potential chain segment     number=23,967,698 hash=6216e3..d42a8c blocks=1 txs=188  mgas=20.456 elapsed=165.113ms    mgasps=123.890  triediffs=207.88MiB triedirty=31.77MiB
INFO [12-08|11:10:36.710] Chain head was updated                   number=23,967,698 hash=6216e3..d42a8c root=0fa653..ccb041 elapsed=4.346008ms
WARN [12-08|11:10:37.222] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:10:37.387] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:10:37.932] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:10:39.636] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,967,698 processed=1         remaining=0          elapsed=1.703s
INFO [12-08|11:10:39.637] Log index head rendering finished        firstblock=21,592,441 lastblock=23,967,698 processed=1         elapsed=1.704s
INFO [12-08|11:10:49.779] Imported new potential chain segment     number=23,967,699 hash=db7f10..95e021 blocks=1 txs=194  mgas=18.567 elapsed=135.585ms    mgasps=136.938  triediffs=208.09MiB triedirty=32.22MiB
INFO [12-08|11:10:49.978] Chain head was updated                   number=23,967,699 hash=db7f10..95e021 root=9710dd..ac8c5a elapsed=3.785993ms
WARN [12-08|11:10:50.543] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:10:50.704] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:10:51.245] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-08|11:11:01.131] Imported new potential chain segment     number=23,967,700 hash=c689cf..7a1931 blocks=1 txs=137  mgas=27.431 elapsed=124.366ms    mgasps=220.563  triediffs=207.84MiB triedirty=32.68MiB
INFO [12-08|11:11:01.293] Chain head was updated                   number=23,967,700 hash=c689cf..7a1931 root=3c732a..45360a elapsed=8.191862ms
WARN [12-08|11:11:01.855] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-08|11:11:02.015] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-08|11:11:02.515] Failed to decode block body              block=21,582,685 error=EOF

[rjl493456442]

Looks like your node database is somehow corrupted. Can you please enable the DEBUG log with these instructions:

geth attach <path-to-your-geth.ipc>
debug.vmodule("core/rawdb=4")

[eth2353]

Thanks, yeah, here are a few minutes of DEBUG logs:

Logs

INFO [12-09|11:23:36.983] Imported new potential chain segment     number=23,974,899 hash=c82c45..448804 blocks=1 txs=188  mgas=15.787 elapsed=104.764ms    mgasps=150.685  triediffs=246.35MiB triedirty=81.50MiB
INFO [12-09|11:23:37.195] Chain head was updated                   number=23,974,899 hash=c82c45..448804 root=aeb691..22ea11 elapsed=2.847141ms
WARN [12-09|11:23:37.748] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:23:37.913] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:23:38.406] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:23:38.420] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,899 processed=1         remaining=0          elapsed=1.224s
INFO [12-09|11:23:38.420] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,899 processed=1         elapsed=1.225s
INFO [12-09|11:23:49.034] Imported new potential chain segment     number=23,974,900 hash=7de98d..9883b4 blocks=1 txs=185  mgas=18.343 elapsed=129.309ms    mgasps=141.850  triediffs=245.94MiB triedirty=82.10MiB
INFO [12-09|11:23:49.433] Chain head was updated                   number=23,974,900 hash=7de98d..9883b4 root=fc3826..9e5d7d elapsed=3.034131ms
WARN [12-09|11:23:49.938] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:23:50.103] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:23:50.722] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:24:01.108] Imported new potential chain segment     number=23,974,901 hash=215f18..52b125 blocks=1 txs=139  mgas=58.418 elapsed=310.614ms    mgasps=188.071  triediffs=245.36MiB triedirty=82.50MiB
INFO [12-09|11:24:01.287] Chain head was updated                   number=23,974,901 hash=215f18..52b125 root=813f18..ab76f7 elapsed=21.342707ms
WARN [12-09|11:24:01.811] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:24:01.980] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:24:02.590] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:24:12.205] Unindexed transactions                   blocks=0 txs=0    tail=21,624,902 elapsed=10.918s
INFO [12-09|11:24:12.496] Imported new potential chain segment     number=23,974,902 hash=c5bb10..88cb07 blocks=1 txs=166  mgas=19.199 elapsed=93.189ms     mgasps=206.024  triediffs=245.05MiB triedirty=82.96MiB
INFO [12-09|11:24:12.695] Chain head was updated                   number=23,974,902 hash=c5bb10..88cb07 root=f5a510..18ab6c elapsed=2.39035ms
WARN [12-09|11:24:13.240] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:24:13.399] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:24:13.881] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:24:23.483] Unindexed transactions                   blocks=0 txs=0    tail=21,624,903 elapsed=10.788s
INFO [12-09|11:24:24.796] Imported new potential chain segment     number=23,974,903 hash=f8cfdd..0158ba blocks=1 txs=188  mgas=14.207 elapsed=98.398ms     mgasps=144.383  triediffs=245.83MiB triedirty=83.19MiB
INFO [12-09|11:24:24.979] Chain head was updated                   number=23,974,903 hash=f8cfdd..0158ba root=da30a5..d3e5e2 elapsed=2.370403ms
WARN [12-09|11:24:25.553] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:24:25.715] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:24:26.206] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:24:35.917] Unindexed transactions                   blocks=0 txs=0    tail=21,624,904 elapsed=10.938s
INFO [12-09|11:24:37.641] Imported new potential chain segment     number=23,974,904 hash=3b3d89..090e6a blocks=1 txs=217  mgas=58.002 elapsed=326.028ms    mgasps=177.905  triediffs=245.49MiB triedirty=83.67MiB
INFO [12-09|11:24:37.824] Chain head was updated                   number=23,974,904 hash=3b3d89..090e6a root=c5d4a0..70bed8 elapsed=22.52293ms
WARN [12-09|11:24:38.380] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:24:38.561] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:24:39.039] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:24:40.637] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,904 processed=1         remaining=0          elapsed=2.075s
INFO [12-09|11:24:40.637] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,904 processed=1         elapsed=2.075s
INFO [12-09|11:24:48.696] Imported new potential chain segment     number=23,974,905 hash=ebb337..cf7293 blocks=1 txs=170  mgas=17.420 elapsed=325.192ms    mgasps=53.568   triediffs=244.49MiB triedirty=84.40MiB
INFO [12-09|11:24:48.956] Chain head was updated                   number=23,974,905 hash=ebb337..cf7293 root=c3b73c..d57d8c elapsed=4.624498ms
DEBUG[12-09|11:24:49.225] Unindexed transactions                   blocks=0 txs=0    tail=21,624,905 elapsed=11.400s
DEBUG[12-09|11:24:49.350] Ancient blocks frozen already            threshold=23,974,828 frozen=23,974,829
INFO [12-09|11:25:02.332] Imported new potential chain segment     number=23,974,906 hash=4ec337..8658bb blocks=1 txs=197  mgas=17.063 elapsed=140.656ms    mgasps=121.313  triediffs=243.68MiB triedirty=85.30MiB
INFO [12-09|11:25:02.332] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=2h56m9s    timestamp=1,765,290,071
INFO [12-09|11:25:02.524] Chain head was updated                   number=23,974,906 hash=4ec337..8658bb root=9f5f18..e63d09 elapsed=4.006535ms
WARN [12-09|11:25:03.094] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:25:03.273] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:25:03.823] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:25:13.234] Imported new potential chain segment     number=23,974,907 hash=3ba576..28e0a0 blocks=1 txs=187  mgas=28.845 elapsed=713.185ms    mgasps=40.445   triediffs=243.10MiB triedirty=86.06MiB
INFO [12-09|11:25:13.569] Chain head was updated                   number=23,974,907 hash=3ba576..28e0a0 root=a81b41..2c126c elapsed=24.060047ms
DEBUG[12-09|11:25:14.117] Unindexed transactions                   blocks=0 txs=0    tail=21,624,907 elapsed=11.591s
INFO [12-09|11:25:26.774] Imported new potential chain segment     number=23,974,908 hash=7f70b9..276ad5 blocks=1 txs=145  mgas=59.992 elapsed=334.508ms    mgasps=179.343  triediffs=242.51MiB triedirty=86.42MiB
INFO [12-09|11:25:26.963] Chain head was updated                   number=23,974,908 hash=7f70b9..276ad5 root=cc2142..00d99c elapsed=23.927391ms
WARN [12-09|11:25:27.561] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:25:27.733] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:25:28.288] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:25:38.119] Unindexed transactions                   blocks=0 txs=0    tail=21,624,909 elapsed=11.155s
INFO [12-09|11:25:38.177] Imported new potential chain segment     number=23,974,909 hash=dbee25..5bf6d7 blocks=1 txs=197  mgas=16.040 elapsed=466.907ms    mgasps=34.353   triediffs=242.49MiB triedirty=86.75MiB
INFO [12-09|11:25:38.346] Chain head was updated                   number=23,974,909 hash=dbee25..5bf6d7 root=997515..c80425 elapsed=2.600432ms
WARN [12-09|11:25:38.904] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:25:39.071] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:25:39.559] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:25:48.517] Imported new potential chain segment     number=23,974,910 hash=9d3d4a..662b76 blocks=1 txs=157  mgas=20.678 elapsed=245.762ms    mgasps=84.140   triediffs=241.82MiB triedirty=87.21MiB
INFO [12-09|11:25:48.734] Chain head was updated                   number=23,974,910 hash=9d3d4a..662b76 root=f1053c..727bc8 elapsed=8.255332ms
DEBUG[12-09|11:25:49.194] Unindexed transactions                   blocks=0 txs=0    tail=21,624,910 elapsed=10.847s
DEBUG[12-09|11:25:49.351] Ancient blocks frozen already            threshold=23,974,828 frozen=23,974,829
INFO [12-09|11:26:01.831] Imported new potential chain segment     number=23,974,911 hash=a54bda..05cabd blocks=1 txs=318  mgas=55.573 elapsed=270.627ms    mgasps=205.348  triediffs=240.81MiB triedirty=87.95MiB
INFO [12-09|11:26:02.002] Chain head was updated                   number=23,974,911 hash=a54bda..05cabd root=af160d..24daa5 elapsed=18.918435ms
WARN [12-09|11:26:02.575] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:26:02.741] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:26:03.278] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:26:03.657] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,911 processed=1         remaining=0          elapsed=1.654s
INFO [12-09|11:26:03.657] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,911 processed=1         elapsed=1.654s
DEBUG[12-09|11:26:12.920] Unindexed transactions                   blocks=0 txs=0    tail=21,624,912 elapsed=10.917s
INFO [12-09|11:26:25.038] Imported new potential chain segment     number=23,974,912 hash=a381cd..38fb33 blocks=1 txs=184  mgas=17.008 elapsed=88.932ms     mgasps=191.241  triediffs=240.48MiB triedirty=88.55MiB
INFO [12-09|11:26:25.325] Chain head was updated                   number=23,974,912 hash=a381cd..38fb33 root=0ba0b9..9d9af3 elapsed=2.188273ms
WARN [12-09|11:26:25.922] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:26:26.083] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:26:26.612] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:26:36.140] Unindexed transactions                   blocks=0 txs=0    tail=21,624,913 elapsed=10.814s
INFO [12-09|11:26:37.997] Imported new potential chain segment     number=23,974,913 hash=ee99bd..97dcf9 blocks=1 txs=127  mgas=20.460 elapsed=133.548ms    mgasps=153.205  triediffs=240.47MiB triedirty=88.91MiB
INFO [12-09|11:26:38.157] Chain head was updated                   number=23,974,913 hash=ee99bd..97dcf9 root=3f5d40..46c726 elapsed=2.8459ms
WARN [12-09|11:26:38.761] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:26:38.922] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:26:39.427] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:26:49.105] Imported new potential chain segment     number=23,974,914 hash=c66c6a..b1653d blocks=1 txs=395  mgas=20.526 elapsed=384.891ms    mgasps=53.329   triediffs=242.37MiB triedirty=88.96MiB
DEBUG[12-09|11:26:49.357] Ancient blocks frozen already            threshold=23,974,828 frozen=23,974,829
INFO [12-09|11:26:49.380] Chain head was updated                   number=23,974,914 hash=c66c6a..b1653d root=df7804..7be145 elapsed=7.861828ms
DEBUG[12-09|11:26:49.457] Unindexed transactions                   blocks=0 txs=0    tail=21,624,914 elapsed=11.299s
INFO [12-09|11:27:00.885] Imported new potential chain segment     number=23,974,915 hash=8fa5b6..2879d7 blocks=1 txs=408  mgas=57.919 elapsed=379.409ms    mgasps=152.657  triediffs=242.51MiB triedirty=89.44MiB
INFO [12-09|11:27:01.079] Chain head was updated                   number=23,974,915 hash=8fa5b6..2879d7 root=4066c1..170c61 elapsed=17.745747ms
WARN [12-09|11:27:01.673] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:27:01.861] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:27:02.395] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:27:11.820] Unindexed transactions                   blocks=0 txs=0    tail=21,624,916 elapsed=10.740s
INFO [12-09|11:27:13.830] Imported new potential chain segment     number=23,974,916 hash=9114ee..c98e3d blocks=1 txs=241  mgas=22.116 elapsed=89.544ms     mgasps=246.986  triediffs=241.95MiB triedirty=90.23MiB
INFO [12-09|11:27:13.987] Chain head was updated                   number=23,974,916 hash=9114ee..c98e3d root=93ed4f..80d7db elapsed=2.862711ms
WARN [12-09|11:27:14.554] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:27:14.716] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:27:15.196] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:27:24.760] Unindexed transactions                   blocks=0 txs=0    tail=21,624,917 elapsed=10.772s
INFO [12-09|11:27:25.211] Imported new potential chain segment     number=23,974,917 hash=11fbca..2419ab blocks=1 txs=168  mgas=15.364 elapsed=180.668ms    mgasps=85.037   triediffs=243.04MiB triedirty=90.35MiB
INFO [12-09|11:27:25.408] Chain head was updated                   number=23,974,917 hash=11fbca..2419ab root=02209a..29cfff elapsed=2.984378ms
WARN [12-09|11:27:25.965] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:27:26.138] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:27:26.646] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:27:36.125] Unindexed transactions                   blocks=0 txs=0    tail=21,624,918 elapsed=10.717s
INFO [12-09|11:27:36.703] Imported new potential chain segment     number=23,974,918 hash=d4f48e..14746b blocks=1 txs=186  mgas=59.998 elapsed=409.830ms    mgasps=146.396  triediffs=242.05MiB triedirty=90.95MiB
INFO [12-09|11:27:36.883] Chain head was updated                   number=23,974,918 hash=d4f48e..14746b root=81f0d1..d72229 elapsed=20.712421ms
WARN [12-09|11:27:37.360] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:27:37.539] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:27:38.115] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:27:39.494] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,918 processed=1         remaining=0          elapsed=1.374s
INFO [12-09|11:27:39.494] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,918 processed=1         elapsed=1.374s
DEBUG[12-09|11:27:47.613] Unindexed transactions                   blocks=0 txs=0    tail=21,624,919 elapsed=10.729s
DEBUG[12-09|11:27:49.357] Ancient blocks frozen already            threshold=23,974,828 frozen=23,974,829
INFO [12-09|11:27:49.431] Imported new potential chain segment     number=23,974,919 hash=0d6442..91ee23 blocks=1 txs=217  mgas=24.822 elapsed=164.694ms    mgasps=150.713  triediffs=241.88MiB triedirty=91.50MiB
INFO [12-09|11:27:49.601] Chain head was updated                   number=23,974,919 hash=0d6442..91ee23 root=b3cc02..8f31cb elapsed=3.666319ms
WARN [12-09|11:27:50.079] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:27:50.242] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:27:50.826] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:28:00.268] Unindexed transactions                   blocks=0 txs=0    tail=21,624,920 elapsed=10.666s
INFO [12-09|11:28:01.622] Imported new potential chain segment     number=23,974,920 hash=bb55e0..de2ac6 blocks=1 txs=203  mgas=20.893 elapsed=121.078ms    mgasps=172.560  triediffs=242.31MiB triedirty=91.94MiB
INFO [12-09|11:28:01.828] Chain head was updated                   number=23,974,920 hash=bb55e0..de2ac6 root=fc5005..f5e510 elapsed=2.904038ms
WARN [12-09|11:28:02.405] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:28:02.587] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:28:03.077] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:28:12.541] Unindexed transactions                   blocks=0 txs=0    tail=21,624,921 elapsed=10.712s
INFO [12-09|11:28:13.142] Imported new potential chain segment     number=23,974,921 hash=ab9881..8ac6c3 blocks=1 txs=243  mgas=20.741 elapsed=123.651ms    mgasps=167.736  triediffs=242.98MiB triedirty=92.21MiB
INFO [12-09|11:28:13.142] Ready for fork activation                fork=BPO1  date="09 Dec 25 14:21 UTC" remaining=2h52m58s   timestamp=1,765,290,071
INFO [12-09|11:28:13.362] Chain head was updated                   number=23,974,921 hash=ab9881..8ac6c3 root=bc8631..aa74c9 elapsed=2.538075ms
WARN [12-09|11:28:13.885] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:28:14.047] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:28:14.554] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:28:14.567] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,921 processed=1         remaining=0          elapsed=1.204s
INFO [12-09|11:28:14.567] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,921 processed=1         elapsed=1.204s
DEBUG[12-09|11:28:24.212] Unindexed transactions                   blocks=0 txs=0    tail=21,624,922 elapsed=10.849s
INFO [12-09|11:28:24.960] Imported new potential chain segment     number=23,974,922 hash=a747bc..0a95d9 blocks=1 txs=83   mgas=59.996 elapsed=333.237ms    mgasps=180.039  triediffs=242.63MiB triedirty=92.44MiB
INFO [12-09|11:28:25.385] Chain head was updated                   number=23,974,922 hash=a747bc..0a95d9 root=95426f..b37841 elapsed=34.805826ms
WARN [12-09|11:28:25.900] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:28:26.070] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:28:26.632] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:28:36.698] Imported new potential chain segment     number=23,974,923 hash=8d9596..412e66 blocks=1 txs=111  mgas=7.188  elapsed=131.691ms    mgasps=54.584   triediffs=240.78MiB triedirty=93.04MiB
DEBUG[12-09|11:28:36.757] Unindexed transactions                   blocks=0 txs=0    tail=21,624,923 elapsed=11.372s
INFO [12-09|11:28:36.996] Chain head was updated                   number=23,974,923 hash=8d9596..412e66 root=bad7be..7836e9 elapsed=1.526849ms
WARN [12-09|11:28:37.586] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:28:37.751] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:28:38.261] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:28:48.179] Unindexed transactions                   blocks=0 txs=0    tail=21,624,924 elapsed=11.174s
INFO [12-09|11:28:48.817] Imported new potential chain segment     number=23,974,924 hash=f9475e..b26ac0 blocks=1 txs=351  mgas=29.510 elapsed=108.803ms    mgasps=271.221  triediffs=241.41MiB triedirty=93.49MiB
INFO [12-09|11:28:48.975] Chain head was updated                   number=23,974,924 hash=f9475e..b26ac0 root=2581e9..00a859 elapsed=4.177633ms
WARN [12-09|11:28:49.547] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:28:49.730] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:28:50.361] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:28:50.373] Deep froze chain segment                 blocks=32 elapsed=1.015s       number=23,974,860 hash=8a23e2..bb06c4
DEBUG[12-09|11:29:00.048] Unindexed transactions                   blocks=0  txs=0    tail=21,624,925 elapsed=11.072s
INFO [12-09|11:29:00.521] Imported new potential chain segment     number=23,974,925 hash=dde78c..e392bd blocks=1  txs=291  mgas=27.613 elapsed=105.888ms    mgasps=260.777  triediffs=241.81MiB triedirty=93.84MiB
INFO [12-09|11:29:00.678] Chain head was updated                   number=23,974,925 hash=dde78c..e392bd root=cc7931..df9832 elapsed=3.031215ms
WARN [12-09|11:29:01.286] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:29:01.449] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:29:01.988] Failed to decode block body              block=21,582,685 error=EOF
INFO [12-09|11:29:02.004] Log index head rendering in progress     firstblock=21,592,441 lastblock=23,974,925 processed=1         remaining=0          elapsed=1.325s
INFO [12-09|11:29:02.004] Log index head rendering finished        firstblock=21,592,441 lastblock=23,974,925 processed=1         elapsed=1.325s
DEBUG[12-09|11:29:11.671] Unindexed transactions                   blocks=0  txs=0    tail=21,624,926 elapsed=10.992s
INFO [12-09|11:29:12.662] Imported new potential chain segment     number=23,974,926 hash=91371a..84398c blocks=1  txs=73   mgas=3.111  elapsed=24.211ms     mgasps=128.510  triediffs=240.38MiB triedirty=94.31MiB
INFO [12-09|11:29:13.046] Chain head was updated                   number=23,974,926 hash=91371a..84398c root=863791..85aeba elapsed="688.043µs"
WARN [12-09|11:29:13.609] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:29:13.800] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:29:14.331] Failed to decode block body              block=21,582,685 error=EOF
DEBUG[12-09|11:29:24.418] Unindexed transactions                   blocks=0  txs=0    tail=21,624,927 elapsed=11.371s
INFO [12-09|11:29:24.721] Imported new potential chain segment     number=23,974,927 hash=2be4b6..39967b blocks=1  txs=243  mgas=59.966 elapsed=263.813ms    mgasps=227.304  triediffs=240.27MiB triedirty=94.84MiB
INFO [12-09|11:29:24.891] Chain head was updated                   number=23,974,927 hash=2be4b6..39967b root=f26f4c..1c8dbb elapsed=18.826323ms
WARN [12-09|11:29:25.473] Failed to decode block body              block=21,582,681 error=EOF
WARN [12-09|11:29:25.650] Failed to decode block body              block=21,582,682 error=EOF
WARN [12-09|11:29:26.184] Failed to decode block body              block=21,582,685 error=EOF

I can resync it, no problem, but I'm also happy to help investigate if you want to dig deeper.

[fjl]

Do you have access to the metric system_cpu_goroutines emitted by geth? If it shows a steady increase in the same period, it would give us a hint.

[eth2353]

Yes, I do. The number of goroutines is stable around 200 over the same timeframe (with infrequent spikes to about 420):

Before the update (so while still running v1.15.10) the coroutine appears to have been slightly lower, around 180.

[fjl]

OK, so it's not a leak at least.

[rjl493456442]

The issue stems from the corrupted database with block bodies of 21,582,681, 21,582,682 and 21,582,685 missing.
In the tx indexer, it tries to unindex the transactions in old blocks in order and the missing block bodies prevent this procedure forever.

Unfortunately this database corruption is not detected in the tx unindexer, which means it will keep reading the older blocks and place them in the memory, discarding them by the interruption signal and redo it again.

I feel we can fix this edge case by placing an "empty set" for the missing block bodies in tx unindexing. It will leak the transaction indexes of these blocks in the key value store as the junks, but it's much better than the potential oom issue.

Or alternatively, we can explicitly abort the unindexing operation is any of the block is missing.

[fjl]

Can't we also just continue somehow? At least for tx indexing, it might be better.

[rjl493456442]

Yes, it's the first option I mentioned, we can ignore the missing block body and continue, the downside is these transactions are not accessible and can confuses users a bit.

But I guess it's the best we can do in case of the corrupted database.