Does not work with delta as a pager

[vogler]

delta used as a pager breaks the output of daff:

Config: vogler/dotfiles@a65b3ff#diff-3cbe50069f7ffe4d55b4d1fe7a72fb79e9c9545af55d8326a728b98f00481bf2R120-R124

I tried to set GIT_PAGER=less only for daff (see link), but that didn't work.
How can I use both?