regression: duplicate `billing` entries since upgrading to 11.2.0

[calestyo]

Hey.

Since upgrading from 10.2.18 to 11.2.0 our billing logs get flooded with duplicate entries (often more than just 2).

/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.19 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
0
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.20 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
0
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.21 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
0
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.22 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
397036
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.23 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
800468
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.24 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
308485
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.25 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
269064
/var/lib/dcache/billing_text/2026/01# cat billing-2026.01.26 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l
458608

(the upgrade happened on the 22nd)

There were no config changes on my side for billing between the upgrades.

dcache.conf

# grep billing dcache.conf
billing.format.json=true
billing.text.format.door-request-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$drMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$clientChain$]$\\t$$$$transactionTime$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"]
billing.text.format.pool-hit-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$phMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$protocol$]$\\t$$$$if(cached)$cached$else$not-cached$endif$$$$\\t$[$transferPath$]$\\t$[$rc$:"$message$"]
billing.text.format.storage-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$sMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$$$$transferTime$ms$\\t$[$rc$:"$message$"]
billing.text.format.mover-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$mMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$protocol$]$\\t$[$initiator$]$\\t$$$$if(p2p)$p2p$else$no-p2p$endif$$$$\\t$$$$if(created)$upload$else$download$endif$$$$\\t$$$$transferred$B$\\t$$$$meanReadBandwidth$B/s$\\t$$$$meanWriteBandwidth$B/s$\\t$$$$connectionTime$ms$\\t$$$$readActive$ms$\\t$$$$readIdle$ms$\\t$$$$writeActive$ms$\\t$$$$writeIdle$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"]
billing.text.format.remove-file-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$rfMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$rc$:"$message$"]
billing.text.format.warning-pnfs-file-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$wpfMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"]
billing.parser.format!door-request-info-message=${billing.text.format.door-request-info-message}
billing.parser.format!pool-hit-info-message=${billing.text.format.pool-hit-info-message}
billing.parser.format!storage-info-message=${billing.text.format.storage-info-message}
billing.parser.format!mover-info-message=${billing.text.format.mover-info-message}
billing.parser.format!remove-file-info-message=${billing.text.format.remove-file-info-message}
billing.parser.format!warning-pnfs-file-info-message=${billing.text.format.warning-pnfs-file-info-message}

The layout has:

[billing]
[billing/billing]

[billing_text]
billing.format.json=false
billing.text.dir=/var/lib/dcache/billing_text
[billing_text/billing]

This also makes the billing files considerably larger. :-(

Cheers,
Chris.

[mksahakyan]

Hi Christoph, There has been changes made in billing for adding kafka logs. But these changes should not have brought this effect. Let us check it ASAP thank you Marina From: "Christoph Anton Mitterer" ***@***.***> To: "dCache" ***@***.***> Cc: "Subscribed" ***@***.***> Sent: Tuesday, 27 January, 2026 01:05:18 Subject: [dCache/dcache] regression: duplicate `billing` entries since upgrading to 11.2.0 (Issue #8008) calestyo created an issue [ #8008 | (dCache/dcache#8008) ] Hey. Since upgrading from 10.2.18 to 11.2.0 our billing logs get flooded with duplicate entries (often more than just 2). /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.19 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 0 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.20 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 0 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.21 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 0 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.22 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 397036 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.23 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 800468 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.24 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 308485 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.25 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 269064 /var/lib/dcache/billing_text/2026/01# cat billing-2026.01.26 | uniq -c | sed 's/^ *//' | grep -E '^[^1]( |[0-9])' | wc -l 458608 (the upgrade happened on the 22nd) There were no config changes on my side for billing between the upgrades. dcache.conf # grep billing dcache.conf billing.format.json=true billing.text.format.door-request-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$drMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$clientChain$]$\\t$$$$transactionTime$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"] billing.text.format.pool-hit-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$phMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$protocol$]$\\t$$$$if(cached)$cached$else$not-cached$endif$$$$\\t$[$transferPath$]$\\t$[$rc$:"$message$"] billing.text.format.storage-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$sMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$$$$transferTime$ms$\\t$[$rc$:"$message$"] billing.text.format.mover-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$mMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$protocol$]$\\t$[$initiator$]$\\t$$$$if(p2p)$p2p$else$no-p2p$endif$$$$\\t$$$$if(created)$upload$else$download$endif$$$$\\t$$$$transferred$B$\\t$$$$meanReadBandwidth$B/s$\\t$$$$meanWriteBandwidth$B/s$\\t$$$$connectionTime$ms$\\t$$$$readActive$ms$\\t$$$$readIdle$ms$\\t$$$$writeActive$ms$\\t$$$$writeIdle$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"] billing.text.format.remove-file-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$rfMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$rc$:"$message$"] billing.text.format.warning-pnfs-file-info-message=$date; format="${lmu.miscellaneous.date-time-format}"$$$$\\t$wpfMsg:$\\t$[$cellType$:$cellName.cell$@$cellName.domain$:$type$]$\\t$[$session$]$\\t$[$pnfsid$:$path$]$\\t$$$$filesize$B$\\t$[$if(storage)$$$$storage.storageClass$@$storage.hsm$$$$else$<unknown>$endif$]$\\t$[$subject.loginName$]$\\t$[$subject.dn$]$\\t$[[$subject.primaryFqan$]:[$subject.fqans; separator="|"$]]$\\t$[$subject.userName$]$\\t$[$subject.uid$]$\\t$[$subject.primaryGid$:$subject.gids; separator="|"$]$\\t$$$$queuingTime$ms$\\t$[$transferPath$]$\\t$[$rc$:"$message$"] billing.parser.format!door-request-info-message=${billing.text.format.door-request-info-message} billing.parser.format!pool-hit-info-message=${billing.text.format.pool-hit-info-message} billing.parser.format!storage-info-message=${billing.text.format.storage-info-message} billing.parser.format!mover-info-message=${billing.text.format.mover-info-message} billing.parser.format!remove-file-info-message=${billing.text.format.remove-file-info-message} billing.parser.format!warning-pnfs-file-info-message=${billing.text.format.warning-pnfs-file-info-message} The layout has: [billing] [billing/billing] [billing_text] billing.format.json=false billing.text.dir=/var/lib/dcache/billing_text [billing_text/billing] This also makes the billing files considerably larger. :-( Cheers, Chris. — Reply to this email directly, [ #8008 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/ADR34RCUULQMU7HLEERISPD4I2TT5AVCNFSM6AAAAACS7CERS2VHI2DSMVQWIX3LMV43ASLTON2WKOZTHA2TQMRQHAZTAMQ | unsubscribe ] . You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[mksahakyan]

Hi @calestyo ,

could you please send an example from the billing.log.

we do not see duplicates so far.

thank you

[calestyo]

Hey @mksahakyan.

Since the billing logs may contain private information, I've created a ticket with a download link for the logs in the internal RT (ID there is 10775).

Oh and I've noted that the JSON output (I store both, with two different billing services) is also affected.

Cheers,
Chris.

[kofemann]

Confirmed! I can see the duplets on my test system.

[kofemann]

The issue pops up when I run multiple dedicated core domains. Is the same for you?

[calestyo]

I also have three core domains, yes.