xargs pipe non empty stdin lines to command while preserving double quotes

If, as your tags suggest, you are running on linux, you have GNU xargs, which supports the -0 option. Then, you can pass in completely arbitrary text, including even newlines:

printf '%s\0' "foo" "'bar"' '"baz"' 'quux
with a newline' | xargs -0 foo

Removing empty lines could be accomplished with a simple grep in front. There is also xargs -r which says to not run the command if xargs receives empty input (this too is a GNU extension).

Your attempts are slightly problematic, though; you should pass the arguments as command-line arguments rather than have xargs interpolate them into the sh -c '... {} ...' string literally.

Slightly depending on your requirements, this could even work portably on other platforms:

xargs sh -c 'if [ $# -gt 0 ]; then echo "$@" | foo; fi' _

The _ is just a placeholder; the arguments to sh -c '...' are used to populate $0, $1, $2, etc and so we put in something, anything, to occupy the slot for $0.

GNU Parallel uses this internally:

perl -e 'if(sysread(STDIN,$buf,1)){open($fh,"|-",@ARGV)||die;syswrite($fh,$buf);if($read=sysread(STDIN,$buf,131071)){syswrite($fh,$buf);}while($read=sysread(STDIN,$buf,131072)){syswrite($fh,$buf);}close$fh;exit($?&127?128+($?&127):1+$?>>8)}' /usr/bin/bash -c 'wc -l'

If you only want a single line try:

seq 3 | parallel --pipe -N1 wc -c
echo "'foo'" | parallel --pipe -N1 --rrs "echo -n i have executed for \"'\";cat;echo \"'\""
echo '"foo"' | parallel --pipe -N1 --rrs "echo -n i have executed for \"'\";cat;echo \"'\""

I want to preserve double quotes from stdin, for that people suggest using -d '\n' but this causes foo to run on empty lines.

xargs performs quote processing by default unless you specify a delimiter via either -d/--delimiter or -0/--null. You must use one of these to avoid xargs removing the quotes you are trying to preserve.

What's more, supposing that you manage to pass the quoted input through xargs unchanged, the shell that xargs launches to run the command will perform its own quote removal, as well as parameter expansion, variable assingments, redirection processing, etc. You can observe the effects of that directly with this variation on your command:

$ xargs -d '\n' -I{} sh -c 'echo {} >>tmp.txt' 
bar
'bar'
$ cat tmp.txt
bar
bar
$

Note that the quotes are removed despite specifying a delimiter to xargs.

It's a bit hard to parse your exact requirements, but it sounds like you just want to filter empty lines out of the standard input to some command. sed can do that pretty easily:

foo() {
  while IFS= read -r line; do
    echo "I have executed for '$line'"
  done
}

$ sed '/\S/!d' | foo


bar

"bar"
A whole line with "quotes" and 'quotes' and metacharacters > < !
I have executed for 'bar'
I have executed for '"bar"'
I have executed for 'A whole line with "quotes" and 'quotes' and metacharacters > < !'
$

Explanation of the sed command: the regex /\S/ matches any non-whitespace character, anywhere on the line. The ! negates the match, and the d deletes lines matching the (negated) pattern -- that is any line that does not contain at least one non-whitespace character.

As you can see in the example run transcript, there is a difference in buffering between your example command and the effect of filtering with sed. It's unclear whether that's important to you.