Remove '\r' Character from Files in Directory Recursively

user_5289570 0
Viewed 450

I write code on a Windows 10 machine, upload it to remote Linux machines where it's actually run. Typically using a IDE feature like Jetbrains upload or WinSCP. I also do all my version control remotely usually with the following workflow:

(in remote session)
 1. $ git clone git@github.com:myorg/myrepo.git
(in local)
 2. Download from remote: /myrepo -> C://User/Me/myrepo
 3. Edit some_file.py
 4. Upload to remote: C://User/Me/myrepo/some_file.py -> /myrepo/some_file.py
(in remote session)
 5. $ python some_file.py  # ERROR: something about bad chars or line endings with '\r'
 6. $ sed -i 's/\r//' some_file.py; python some_file.py  # WORKS!
 7. $ git add some_file.py; git commit -m "removed bad win char"

This error and my current method of resolution is rather annoying. I tried automating it with the following bash script I included in my $PATH at ~/mytools/remove_win_char.sh

#!/usr/bin/bash

find . -type f -exec sed -i 's/\r//g' {} \;

Unfortunately this has some unintended side effects in git repos: (i.e. this answer does not work)

$ remove_win_char.sh
$ git status
fatal: unknown index entry format 0x2f610000

I tried to fix by specifying only certain files in the script:

find . -name *.py -o -name *.sql -o -name *.sh -exec sed -i 's/\r//g' {} \;

Unfortunately this only seems to hit the .sh files.

Anyone know how to filter .py, .sql, and .sh files only with find? Or know a better way to remove these \r characters created locally by Windows?

linux windows git bash
1 Answers

Using find and sed may destroy your repository because they are not aware of the git repository, its internals and the way how git treats tracked files. You have to use git ls-files to produce the list of files it tracks as text files with CR/LF line endings and then process the files accordingly:

git ls-files --eol

It produces tabular output like

i/lf    w/lf    attr/                   .gitignore
i/crlf  w/crlf  attr/                   README.md
i/lf    w/lf    attr/                   env/install.sh

that can be filtered using awk (unfortunately, not sure if grep can handle fields) and cut, and then CR/LF-to-LF-fixed using dos2unix.

git -c core.quotepath=off ls-files --eol '*.py' '*.sql' '*.sh' \ # query git
    | awk '$1 ~ /^i\/crlf/' \                                    # filter only lines starting with i/crlf
    | cut -f2 \                                                  # filter files only (see why it is TAB-delimited https://git-scm.com/docs/git-ls-files#_output)
    | xargs -I{} dos2unix {}                                     # convert CR/LF to LF
user_122328700
Related