Search This Blog

2015-02-17

Piped Redirects and Variables

When you try to perform a while loop with input from another command it is possible you will see unexpected results.

Stolen directly from http://mywiki.wooledge.org/BashFAQ/024


# Works only in ksh88/ksh93, or bash 4.2 with lastpipe enabled
# In other shells, this will print 0
linecount=0

printf '%s\n' foo bar |
while read -r line
do
linecount=$((linecount + 1))
done

echo "total number of lines: $linecount"


The reason for this potentially surprising behaviour, as described above, is that each SubShell introduces a new variable context and environment. The while loop above is executed in a new subshell with its own copy of the variable linecount created with the initial value of '0' taken from the parent shell. This copy then is used for counting. When the while loop is finished, the subshell copy is discarded, and the original variable linecount of the parent (whose value hasn't changed) is used in the echo command.

Different shells exhibit different behaviors in this situation:

BourneShell creates a subshell when the input or output of anything (loops, case etc..) but a simple command is redirected, either by using a pipeline or by a redirection operator ('<', '>').

BASH creates a new process only if the loop is part of a pipeline.

KornShell creates it only if the loop is part of a pipeline, but not if the loop is the last part of it. The read example above actually works in ksh88 and ksh93! (but not mksh)

POSIX specifies the bash behaviour, but as an extension allows any or all of the parts of the pipeline to run without a subshell (thus permitting the KornShell behaviour, as well).

There are a few ways to get around this:

If Input is a file


# POSIX
while read -r line; do linecount=$((linecount + 1)); done < file echo $linecount


Bash Only Process Substitution


# Bash
while read -r line
do
((linecount++))
done < <(grep PATH /etc/profile) echo "total number of lines: $linecount"


Store Output of while loop in variable


unset -v foo
foo=$(
echo nothing | {
while read line; do
foo=bar
done
echo $foo
}
)
echo foo2: $foo

# Example output:
# foo2: bar (ok)


Additional Methods and Explanations

http://mywiki.wooledge.org/BashFAQ/024
http://www.fvue.nl/wiki/Bash:_Piped_%60while-read%27_loop_starts_subshell
http://unix.stackexchange.com/questions/9954/why-is-my-variable-being-localized-in-one-while-read-loop-but-not-in-another
http://serverfault.com/questions/259339/bash-variable-loses-value-at-end-of-while-read-loop
http://stackoverflow.com/questions/19570413/bash-how-to-pipe-input-to-while-loop-and-perserve-variables-after-loop-ends

No comments:

Post a Comment