TL;DR: Using cat file | while read runs the loop in a subshell, so any variable you set inside it disappears when the loop exits. Instead of the pipe, a redirect, using while read < file, runs the loop in the current shell and the variable exists after the loop. Another trick, process substitution (< <(command)), does the same when the producer is a command rather than a file.
1. The canonical form vs. the form that may fail
Let's say you have the file players.txt as follows, and you want to read every line and remember the last one.
alice
bob
carolThe form that works for this use case is:
while read -r name; do
last="$name"
done < players.txt
echo "$last" # carolThree things are happening here:
Using
< players.txtopens the file as the loop's standard input.The loop runs in the current shell.
The
lastvariable is set in the parent shell's memory and survives the loop.
The form to avoid is the one that adds a cat and a pipe, cat file | while read, as in the following code block:
cat players.txt | while read -r name; do
last="$name"
done
echo "$last" # emptyThe only change is the producer and the connector, but with the form that uses cat file | while read the effect is different: after the loop has run, the echo runs in a shell where last was never assigned.
Both forms have different mechanisms, and only one of them makes sure that the variable is present after the loop.
If you read the Iterate a list of items and do something: three shell idioms issue, you saw the first form presented as the recommended one, and the subshell concept briefly mentioned. Here, we are going to cover what "subshell" means and what to write when you need the variable to exist after the loop.
2. Why the loop runs in a subshell
When you write cat file | while read, the shell creates two processes connected by a pipe: the cat process on the left, the while loop on the right. That while loop is a subshell, i.e. a child process forked from the shell that owns the pipe. Anything you write to a variable inside that subshell lives in the subshell's memory, and as a consequence, when the loop ends and the subshell exits, that variable disappears.
But using the redirect, you avoid that behavior: The shell opens the file and hands it to the loop as stdin, and the loop runs in the same process. Since there is no fork (no subshell), variables set inside the loop live in the parent shell.
3. Two forms that work, depending on the producer
As we have just seen, the redirect form is the right answer when the producer is a file (with no need to use cat).
while read -r name; do last="$name"; done < players.txtBut when the producer is a command rather than a file, you need process substitution to feed the loop from that command. This is the < <(command) form, a syntax that expands the inner command and hands its stdout to the loop as a file descriptor; here the loop runs in the current shell, and there is no fork on the right-hand side of the <. For example, for the cat players.txt command, you would use:
while read -r name; do last="$name"; done < <(cat players.txt)Both forms give the same result: the loop's stdin is fed from a file descriptor, so the loop runs in the parent shell, so the variable is present after the loop.
4. When the pipe form is fine
If the loop's job is to produce output, for example write to a file, pipe to another command, or fill a buffer, the pipe form is correct and idiomatic. For example:
cat players.txt | while read -r name; do
echo "$name scored $((RANDOM % 100))"
done > scores.logHere the write happens inside the subshell and lands on disk. The loop's output is the result we want, not the loop's variables, and most pipelines work this way.
The trap is only when the loop's job is to communicate back to the calling shell.
5. Differences in behavior across versions
Some versions of Bash, such as 3.2, fork for the right-hand side of every pipe, so the loop is always a subshell. Bash 4+ offers the shopt -s lastpipe solution to opt into running the last command of a pipeline in the current shell, while zsh does this by default. Anyway, the redirect form works in any shell, so you can just stick to it.