tar: file changed as we read it (cron backup fix)
A nightly tar job exits 1 and prints tar: site/access.log: file changed as we read it even though nothing is broken. The archive is usually valid and the message informational — but how your cron wrapper reads that exit code decides whether you get a silent gap in your backups or an alert you will eventually mute. This post measures what the message means on GNU tar 1.35, what each "silencer" flag does to the exit status, and which fix removes the race instead of hiding it.
Prerequisites: a Linux host with GNU tar (these flags are GNU-specific; BusyBox tar and bsdtar differ), a cron or systemd timer you can edit, and shell access to the directory being archived. Everything quoted ran on Linux 6.17.0-1019-oracle, aarch64, GNU tar 1.35 — the tar transcript on 20 October 2026, the audit and wrapper runs on 21 October.
TL;DR — exit 1 means "one file moved", not "backup failed"
file changed as we read itis a warning: tar recorded the size when it opened the file, copied that many bytes, then re-stat'ed and found the size or mtime had moved. Exit status 1.- Exit 2 is the fatal one (“Error is not recoverable”): bad archive path, unreadable directory, I/O error.
--warning=no-file-changedhides the text and leaves exit 1 — evidence removed, monitoring still paging.--ignore-failed-readdoes the reverse: exit 0, warning kept.tar -cf - … | gzipwithoutpipefailreturns 0 while tar returned 1:rc=0withoutset -o pipefail,rc=1with it.copytruncateproduces a different message —File shrank by N bytes; padding with zeros, which--warning=no-file-changeddoes not silence.- The archive is not corrupt: it holds exactly the bytes tar saw, short only of the tail written after
fstat()— 46 bytes in the run below.
What tar is actually comparing
tar does not watch the file. For each regular file it calls fstat() on the open descriptor, writes a header with that size and mtime, copies exactly size bytes, then re-checks the descriptor. Size grew, shrank or mtime moved — one warning line per file, status 1.
That design is why the archive is still readable: tar never writes more bytes than the header promised, so it stops at the size the file had when the run reached it. A log that grew 46 bytes during the backup is not corrupt in the archive, it is short by 46 bytes, and the tail exists only on disk. The message is a consistency signal, not an integrity failure; conflating the two produces muted alerts or silent gaps, and which one you get depends on the wrapper.
| Exit status | Meaning (GNU tar) | What a backup job should do |
|---|---|---|
0 | All files archived unchanged. | Success. Verify, then rotate. |
1 | Some files differ: changed, shrank, or vanished during the run. | Keep the archive, record which files, decide per volume whether to fail. |
2 | Fatal: not recoverable. Measured: tar: Error is not recoverable: exiting now for an unwritable archive path. | Fail loudly. The archive is unusable. |
Reproduce it on demand, without waiting for a busy log
The hard part is timing: the file must move while tar is inside it, and on a quiet box the append lands after tar finished and you conclude it is fixed. Withhold back-pressure instead — tar streams into a pipe nobody reads, the buffer fills, tar blocks mid-file, and the append has to race.
Two details keep the harness reliable: stderr goes to a file, because a full stderr pipe would deadlock a parent busy reading stdout, and the read size is the real F_GETPIPE_SZ capacity — pipe capacity is configurable, and a guessed 65536 turns into a race again.
The harness is small enough to rebuild in an afternoon, and three details make it reliable. Redirect stderr to a file: a full stderr pipe would deadlock a parent busy reading stdout. Read exactly fcntl.fcntl(pipe, F_GETPIPE_SZ) bytes from the archive stream, since a guessed 65536 turns the race into a coin flip. Then mutate, drain the rest, wait for the child and write both halves to one file — tar re-stats the file when it resumes, and that is where the warning comes from.
Three mutations cover every message you will meet: append, truncate in place (what copytruncate does), and unlink (a rotation with create plus an impatient cleanup). One contiguous run produced the lines below, verbatim with exit codes and archive sizes.
Three mutations cover every message you will meet: append, truncate in place (what copytruncate does), and unlink (a rotation with create plus an impatient cleanup). One contiguous run produced the lines below, verbatim with exit codes and archive sizes.
[append] appended to 00-victim.log while tar was blocked
[append] pipe capacity: 65536 bytes, tar exit: 1, archive: 2109440 bytes
[append] stderr: tar: src/00-victim.log: file changed as we read it
[truncate] truncated 00-victim.log in place (copytruncate) while tar was blocked
[truncate] pipe capacity: 65536 bytes, tar exit: 1, archive: 2109440 bytes
[truncate] stderr: tar: src/00-victim.log: File shrank by 1968128 bytes; padding with zeros
[remove] unlinked 00-victim.log while tar was blocked
[remove] pipe capacity: 65536 bytes, tar exit: 1, archive: 2109440 bytes
[remove] stderr: tar: src: file changed as we read it
Read the third case twice: after an unlink tar names the directory (tar: src: file changed as we read it), not the file it lost — the path you were grepping for never appears. A rule matching the bare string file changed fires; a rule grepping a specific path does not.
The same run continued with the flag variants, in order — the part that contradicts most of the advice on page one:
[quiet] appended to 00-victim.log while tar was blocked
[quiet] pipe capacity: 65536 bytes, tar exit: 1, archive: 2109440 bytes
[quiet] stderr: (empty)
[quiet-shrank] truncated 00-victim.log in place (copytruncate) while tar was blocked
[quiet-shrank] pipe capacity: 65536 bytes, tar exit: 1, archive: 2109440 bytes
[quiet-shrank] stderr: tar: src/00-victim.log: File shrank by 1968128 bytes; padding with zeros
[ignore] appended to 00-victim.log while tar was blocked
[ignore] pipe capacity: 65536 bytes, tar exit: 0, archive: 2109440 bytes
[ignore] stderr: tar: src/00-victim.log: file changed as we read it
[both] appended to 00-victim.log while tar was blocked
[both] pipe capacity: 65536 bytes, tar exit: 0, archive: 2109440 bytes
[both] stderr: (empty)
So --warning=no-file-changed bought a cleaner log and nothing else: the job still exits 1 and monitoring still fires, with no explanation attached. Because file-shrank is a separate warning class in tar --warning=help, silencing file-changed leaves the copytruncate case intact — same message, same exit 1. Neither flag is a fix; both are policy.
Three messages, three causes, one exit code
All three mutations exit 1, so the status alone cannot tell you what happened — the message can, if you keep it.
| Message (measured) | Mutation | Real-world cause |
|---|---|---|
file changed as we read it | appended or rewritten in place | Live application log, database WAL, growing queue file. |
File shrank by N bytes; padding with zeros | truncated in place | logrotate with copytruncate — the classic. |
tar: src: file changed as we read it (the directory, not the file) | unlinked while open | Rotation with create plus a cleanup that deletes old segments. |
The shrink case silently degrades restores: tar pads the short read with zeros, the archive lists the pre-truncation size, and extraction hands you a log with NULs glued to the end. That file passes tar -tf and gzip -t, and is wrong. Run copytruncate on a backed-up directory and you have this case already.
Why the alert never fires: the pipeline eats the status
The second half of the problem is in the cron line. Most people write tar -czf backup.tar.gz /srv/site, and many pipe it into gzip, pigz or ssh — and a pipeline reports the status of the last command. Measured with the append landing mid-run:
no pipefail shell rc=0 tar stderr='tar: src/00-victim.log: file changed as we read it'
pipefail shell rc=1 tar stderr='tar: src/00-victim.log: file changed as we read it'
Both runs carried the same warning; only one told the scheduler — the whole difference between a monitoring gap and a signal, and it costs one line. The wrapper I run does six things, and none of them is exotic: set -Eeuo pipefail and LC_ALL=C; unset TAR_OPTIONS so an inherited flag cannot change the run; realpath -e on the source and a destination that is owned by the run user and not inside that source; a flock on a lock file in the destination; the archive written into a private mktemp -d staging directory and renamed into place only after gzip -t; and the stderr log read whatever tar returned, not only when it returned 1.
Each of those exists for a measured reason. Archiving an unresolved symlinked docroot stores the link and nothing else: one member, site -> ..., exit 0, no data. Resolved and archived with tar -C "$SOURCE" . the same tree yields ./, ./logs/, ./logs/access.log, ./index.html and rc 0, while a source that is a file or missing exits 2, and so does a destination inside the source. The lock exists because the timestamp in the archive name has one-second resolution: two overlapping runs would otherwise write the same paths, and the second now exits 75 with already running. Reading the log regardless of exit status closes the case the transcript above shows — exit 0 with warnings present. Exit 1 is accepted only when every stderr line is classified as changed, shrunk or removed and the volume has explicitly opted in; anything unclassified stays a failure, because an unknown line is exactly where a real I/O error would hide.
Did the archive survive? Prove it, do not assume
The claim to test is narrow: the archive holds exactly the bytes tar saw. Extracting the archive written during the blocked read gave 2,097,152 against 2,097,198 bytes live — a 46-byte tail, on disk, absent from the archive, with no corruption in the stream.
archived 00-victim.log: 2097152 bytes / live: 2097198 bytes (tail 46 bytes only on disk)
tar -df compares members against the filesystem and names discrepancies, but note what it compares: size and mtime, not content. Restoring a file to byte-identical content with a fresh mtime gave src/01-other.txt: Mod time differs and exit 1, while a same-size content change was invisible. That is not a content audit, so hash the members instead.
For an audit that survives a restore drill, walk the archive and the live tree, then hash the members — an archive that is missing a file passes every archive-only check ever written. The list of things to compare is longer than it looks, and each item was a finding in its own right when I built this: member type, plus mode, owner and group, because otherwise a setuid bit or a changed owner rides along; device numbers for character and block devices; the recorded symlink target and whether its resolved path stays under the root; and for hardlinks, that the target is a regular member on the same device and inode — a pair of links pointing at each other passes a name check, stores no file data, and cannot be extracted.
Names need normalising before any comparison, or a valid archive reports every file as missing: posixpath.normpath makes ./src/a and src/a the same entry, which matters the moment a wrapper archives with tar -C "$SOURCE" .. Only a literal . member is the root; absolute names, parent-relative names and anything else that normalises to . are rejected. Open the root itself with O_NOFOLLOW and resolve members one component at a time relative to that descriptor, so a swapped symlink cannot relocate the whole audit; open the leaf with O_NOFOLLOW|O_NONBLOCK, re-check device and inode with fstat() before trusting it, and read at most member.size + 1 bytes so a FIFO substituted between the two calls cannot hang the run. A truncated archive must end as a reported finding, not a traceback.
Run it against the parent of the archived path, or every member resolves a level too deep. On the blocked-read archive it reported size src/00-victim.log: archive 2097152, live 2097198 and exited 1 — the 46-byte tail, caught without tar saying a word. A same-size rewrite gave content src/a.txt: digest differs; a file replaced by a symlink gave type src/a.txt: archive 32768, live 40960; a re-pointed hardlink gave hardlink src/a.txt: archive -> src/h.txt, different inode; a live file the archive never recorded gave not archived: src/new.txt; and the same tree archived as tar -C src . returned 0. Point it at a tree wider than the archive and it names every live file tar never recorded — the gap no archive-only check can see.
Run the audit after the backup, and alert on the count of findings rather than the first line: one moved log is routine, a directory of them is worth a page.
Removing the race instead of filtering the message
Flags change what you hear; only one class of fix changes what happens. A snapshot is correctness, a stop-the-writer window is correctness with downtime, the rest is a smaller race.
- Snapshot the volume, archive the snapshot. LVM thin, ZFS, btrfs or a cloud block-device snapshot: nothing can move under tar because the object it reads is not being written. The only fix that scales to a database directory.
- Ask the writer to rotate, do not truncate. For nginx,
kill -USR1reopens the logs: a new inode, and the old file stops growing — theFile shrankcase disappears at source. - Quiet window.
systemctl stoparound the backup is honest and cheap, but it costs availability: fine for a config tree, not a live site. - Hardlink snapshots for files, not for consistency.
rsync --link-destgives space-efficient point-in-time directories — documented behaviour, and rsync is not installed on the host where this was verified, so I describe the flag rather than quote a run. A file rewritten mid-transfer can still be copied torn. - Dump, do not copy, for databases. A live Postgres data directory is the worst case; use the engine's consistent dump, as in the SQLite unable-to-open write-up.
Two rules keep a snapshot directory honest. Order the steps so latest never points at a directory whose dump failed halfway: stage per run, dump and restore-check there, then swap the link in one rename, so a failure leaves the previous snapshot current — silent to readers of latest, loud in the exit status. And give every run its own resources: a staging path and a throwaway database name that both carry the run id, created by a plain createdb that fails if the name exists, plus an EXIT trap that drops only that database. A fixed appdb_restore_check name and dropdb --if-exists is how a restore check deletes somebody's database.
The rotation half is a logrotate stanza with a trap in it. create plus a postrotate signal gives the application a new inode, so the file tar may be reading stops growing — but write the hook as if/fi, not [ -s pid ] && kill -USR1: the && form returns 1 whenever nginx is stopped, which logrotate reports as a failed postrotate script, and -s accepts any non-empty file as a PID. Validate the PID as numeric, confirm /proc/$pid/comm says nginx, and let a failed signal be the exit status of the hook. Then check the reopen by comparing inode numbers across a rotation — identical inodes mean the process still writes the old file.
What to change today
- Add
set -Eeuo pipefailto every cron backup script that pipes tar, plusLC_ALL=Cso the messages your greps depend on cannot drift, andunset TAR_OPTIONSso an inherited flag cannot change the run. - Separate exit 1 from exit 2; make exit 1 fail closed with a per-volume opt-in.
- Resolve the source and refuse to run when it is not a directory — a symlinked docroot archives as one symlink member and exits 0.
- Never ship
--warning=no-file-changedas a fix: it leaves exit 1 and removes the only explanation your logs had. - With
copytruncate, expectFile shrank by N bytes; padding with zeros; the flag will not hide it, and the padding survives into the restore. - Verify with a digest audit, and move the source to a snapshot for anything written continuously.
Sibling failures worth reading next: ENOSPC from the inotify watch limit is a wall of noise from a kernel quota, not a moving file, and E2BIG on execve belongs to the same "the command is fine, the interface is not" family.