Remove carriage return in Unix

Viewed 466097

What is the simplest way to remove all the carriage returns \r from a file in Unix?

21 Answers

I'm going to assume you mean carriage returns (CR, "\r", 0x0d) at the ends of lines rather than just blindly within a file (you may have them in the middle of strings for all I know). Using this test file with a CR at the end of the first line only:

$ cat infile
hello
goodbye

$ cat infile | od -c
0000000   h   e   l   l   o  \r  \n   g   o   o   d   b   y   e  \n
0000017

dos2unix is the way to go if it's installed on your system:

$ cat infile | dos2unix -U | od -c
0000000   h   e   l   l   o  \n   g   o   o   d   b   y   e  \n
0000016

If for some reason dos2unix is not available to you, then sed will do it:

$ cat infile | sed 's/\r$//' | od -c
0000000   h   e   l   l   o  \n   g   o   o   d   b   y   e  \n
0000016

If for some reason sed is not available to you, then ed will do it, in a complicated way:

$ echo ',s/\r\n/\n/
> w !cat
> Q' | ed infile 2>/dev/null | od -c
0000000   h   e   l   l   o  \n   g   o   o   d   b   y   e  \n
0000016

If you don't have any of those tools installed on your box, you've got bigger problems than trying to convert files :-)

Old School:

tr -d '\r' < filewithcarriagereturns > filewithoutcarriagereturns

There's a utility called dos2unix that exists on many systems, and can be easily installed on most.

sed -i s/\r// <filename> or somesuch; see man sed or the wealth of information available on the web regarding use of sed.

One thing to point out is the precise meaning of "carriage return" in the above; if you truly mean the single control character "carriage return", then the pattern above is correct. If you meant, more generally, CRLF (carriage return and a line feed, which is how line feeds are implemented under Windows), then you probably want to replace \r\n instead. Bare line feeds (newline) in Linux/Unix are \n.

Removing \r on any UNIX® system:

Most existing solutions in this question are GNU-specific, and wouldn't work on OS X or BSD; the solutions below should work on many more UNIX systems, and in any shell, from tcsh to sh, yet still work even on GNU/Linux, too.

Tested on OS X, OpenBSD and NetBSD in tcsh, and on Debian GNU/Linux in bash.


With sed:

In tcsh on an OS X, the following sed snippet could be used together with printf, as neither sed nor echo handle \r in the special way like the GNU does:

sed `printf 's/\r$//g'` input > output

With tr:

Another option is tr:

tr -d '\r' < input > output

Difference between sed and tr:

It would appear that tr preserves a lack of a trailing newline from the input file, whereas sed on OS X and NetBSD (but not on OpenBSD or GNU/Linux) inserts a trailing newline at the very end of the file even if the input is missing any trailing \r or \n at the very end of the file.


Testing:

Here's some sample testing that could be used to ensure this works on your system, using printf and hexdump -C; alternatively, od -c could also be used if your system is missing hexdump:

% printf 'a\r\nb\r\nc' | hexdump -C
00000000  61 0d 0a 62 0d 0a 63                              |a..b..c|
00000007
% printf 'a\r\nb\r\nc' | ( sed `printf 's/\r$//g'` /dev/stdin > /dev/stdout ) | hexdump -C
00000000  61 0a 62 0a 63 0a                                 |a.b.c.|
00000006
% printf 'a\r\nb\r\nc' | ( tr -d '\r' < /dev/stdin > /dev/stdout ) | hexdump -C
00000000  61 0a 62 0a 63                                    |a.b.c|
00000005
% 

cat input.csv | sed 's/\r/\n/g' > output.csv

worked for me

Using sed

sed $'s/\r//' infile > outfile

Using sed on Git Bash for Windows

sed '' infile > outfile

The first version uses ANSI-C quoting and may require escaping \ if the command runs from a script. The second version exploits the fact that sed reads the input file line by line by removing \r and \n characters. When writing lines to the output file, however, it only appends a \n character. A more general and cross-platform solution can be devised by simply modifying IFS

IFS=$'\r\n' # or IFS+=$'\r' if the lines do not contain whitespace
printf "%s\n" $(cat infile) > outfile
IFS=$' \t\n' # not necessary if IFS+=$'\r' is used

Warning: This solution performs filename expansion (*, ?, [...] and more if extglob is set). Use it only if you are sure that the file does not contain special characters or you want the expansion.
Warning: None of the solutions can handle \ in the input file.

Though it's a older post, recently I came across with same problem. As I had all the files to rename inside /tmp/blah_dir/ as each file in this directory had "/r" trailing character ( showing "?" at end of file), so doing it script way was only I could think of.

I wanted to save final file with same name (without trailing any character). With sed, problem was the output filename which I was needed to mention something else ( which I didn't want).

I tried other options as suggested here (not considered dos2unix because of some limitations) but didn't work.

I tried with "awk" finally which worked where I used "\r" as delimiter and taken the first part:

trick is:

echo ${filename}|awk -F"\r" '{print $1}'

Below script snippet I used ( where I had all file had "\r" as trailing character at path /tmp/blah_dir/) to fix my issue:

cd /tmp/blah_dir/
for i in `ls`
  do
    mv   $i     $(echo $i | awk -F"\r" '{print $1}')
done

Note: This example is not very exact though close to what I worked (Mentioning here just to give the better idea about what I did)

I made this shell-script to remove the \r character. It works in solaris and red-hat:

#!/bin/ksh

LOCALPATH=/Any_PATH

for File in `ls ${LOCALPATH}`
do
   ARCACT=${LOCALPATH}/${File}
   od -bc ${ARCACT}|sed -n 'p;n'|sed 's/015/012/g'|awk '{$1=""; print $0}'|sed 's/ /\\/g'|awk '{printf $0;}'>${ARCACT}.TMP
   printf "`cat ${ARCACT}.TMP`"|sed '/^$/d'>${ARCACT}
   rm ${ARCACT}.TMP
done

exit 0
Related