lseek errors on XFS after upgrading to 0.4.42

[brooksprumo]

Hi folks,

I am a developer on https://github.com/anza-xyz/agave/ and we've released our v2.1 somewhat recently. This version upgrades the tar crate from 0.4.41 to 0.4.42.

We've been getting reports of these errors when our snapshot process runs, which brings down the process:

lseek(SEEK_DATA) did not advance. Did the file change while appending?

I see that this was added in #375. I haven't gone through the code sufficiently yet to understand, but did want to open this issue early to see if there's any known issues or debugging help available.

The agave code that is seeing the errors is here:
https://github.com/anza-xyz/agave/blob/085c6055c4ba6b2593927cef4392f32e3b43a888/runtime/src/snapshot_utils.rs#L1044-L1046

This code did not did not change between v2.0 and v2.1, which is why I was surprised to start hearing these reports. I posted a bit more information over in our discord here too: https://discord.com/channels/428295358100013066/838890116386521088/1352668469015089152

Some additional info that may be useful:

• The files being archived have not changed. Agave uses a strict write-all and write-once approach, and does not append/modify the files being snapshotted.
• These files are written in the same thread that does the archiving, so there aren't concurrency concerns e.g. if the files are actively still writing while archiving happens.
• All of the reports of this error (that I've seen) have been on XFS, except for one that was using tmpfs.
• The errors are non-deterministic and sporadic.
• All the machines I have access to are ext4, and have not had this error.
• I'm going to try getting a machine with XFS to try and reproduce
• I'm going to request the people that saw this error to turn off spare files and see if that helps. Unfortunately the absence of an error doesn't necessarily prove anything definitively, esp. given the (in)frequency of the error.

Please do let me know if there's any additional information that would be useful, and thanks in advance for any help/suggestions!

[brooksprumo]

Maybe this is related to #387?

[xzfc]

In short: I suspect it may be caused by xfs_fsr (defragmenter) running in the background. Also, is this the only possible error message reported?

Maybe this is related to #387?

No, #387 is about unit test input data.

Explanation

This error happen within a loop that detects holes in a sparse file. The error lseek(SEEK_DATA) did not advance means there was a hole, but now there is data. Illustrated:

• Step 1:

  // Before:  | DATA | HOLE | DATA |
  //          ^
  off = lseek(file, off, SEEK_HOLE);
  // After:   | DATA | HOLE | DATA |
  //                 ^

• Step 2a (normal flow):

  // Before:  | DATA | HOLE | DATA |
  //                 ^
  off = lseek(file, off, SEEK_DATA);
  // After:   | DATA | HOLE | DATA |
  //                        ^

• Step 2b (erroneous flow):

  // Before:  | DATA | DATA | HOLE |
  //                 ^
  off = lseek(file, off, SEEK_DATA);
  // ↑ lseek did not advance since `off` is already inside the data
  // After:   | DATA | DATA | HOLE |
  //                 ^

Normally, the state of Step 1's after is the same as Step 2's before.

So, the suspects are:

1. SEEK_HOLE did not seek to the hole for some reason (a kernel bug; unlikely).
2. Something updated file sparseness explicitly (e.g., running fallocate on a file).
   I consider this not an issue as you're supposed not to modify files being archived.
3. Something updated file sparseness implicitly (e.g., FS sync or a defragmenter running in the background).

So, digging the third suspect, I was able to reproduce it by running a xfs_fsr in the background during a load-test of find_sparse_entries_lseek(). It seems it is able to convert some holes into data.

Some debugging commands

• fallocate -d $FILE to dig holes in a file.
• fallocate -l $SIZE $FILE to convert holes into data. (not sure tho)
• xfs_fsr to run defragmenter continuously; or xfs_fsr $FILE to defragment a specific file.
• xfs_bmap $FILE or xfs_io -r -c "fiemap -v" $FILE or filefrag -v $FILE to see holes in a sparse file.
• To create an XFS within a file:

  dd if=/dev/zero of=xfs.img bs=1M count=1024
  mkfs.xfs xfs.img
  sudo mount xfs.img /mnt/xfs

Why tar-rs reports this as an error?

In the #375, I've added this check to make sure that this loop never runs forever (due to possible bugs in the kernel/filesystems; or implementations behave differently on other non-Linux systems). But I did not take into account that defragmenters running in the background can interfere with this logic, and it's outside of the application developer control. I'll come with a fix once I have time (should be within a week).

[brooksprumo]

Thanks for the reply! I'll see if I can get information about if xfs_fsr is running on the machines that saw the errors.

As a work-around, would setting .sparse(false) on the tar Builder work?

[xzfc]

Yes, it should.