Removing carriage returns

When working with shell scripts, it's very easy to get tripped up by line returns and get output which looks something like this:

$ ./example.sh
-bash: ./example.sh: /bin/sh^M: bad interpreter: No such file or directory

The first time you run into this problem, it can be a little confusing. Running the script as an argument to bash will work as expected:

$ bash example.sh
Hello world

And displaying the contents of the file with cat won't show any obvious problems:

$ cat example.sh
#!/bin/sh
echo 'Hello world'

What's actually going on?

A brief history of new lines

Most modern computers use ASCII, or a derivative like Unicode. ASCII characters can be split into two groups, printable characters and control characters. Printable characters are characters which can be represented visually, such as alphanumeric characters or punctuation.

Control characters on the other hand are not normally displayed. Instead they provide instructions intended for the device displaying the text (e.g. a printer). Many of the control characters have their roots in actions you would perform on a manual typewriter. For example when moving to a new line on a typewriter, you have to move the carriage back to the starting position and then feed the paper so you don't overwrite the previous line. These movements map to carriage returns (\r) and line feeds (\n) in ASCII.

Unfortunately not all operating systems use the same sequence of control characters to represent a new line. Unix-like systems (Linux, OS X, FreeBSD etc) all use a single line feed (\n) to represent a new line. DOS and windows systems on the other hand use a carriage return followed by a line feed (\r\n).

Looking at the first line

When you execute a shell script, the first line contains a shebang, followed by the path to the interpreter which should be used. If you're using windows-style new lines you will have an extra carriage return before the line feed character.

$ hexdump -c example.sh
0000000   #   !   /   b   i   n   /   s   h  \r  \n   e   c   h   o
0000010   '   H   e   l   l   o       w   o   r   l   d   '  \r  \n
000001f

As a result bash will try to use /bin/sh\r as the interpreter and fail. You can easily tell if you're using windows-style new lines using the file command:

$ file example.sh
example.sh: POSIX shell script, ASCII text executable, with CRLF line terminators

Removing extra carriage returns

To fix this issue, any extra carriage returns before a line feed need to be removed. There is a command called dos2unix for doing this:

$ dos2unix example.sh
dos2unix: converting file example.sh to Unix format ...

If you don't have dos2unix installed you can use sed:

$ sed -i 's/\r$//' example.sh

If for some reason you don't have sed, you can use tr:

$ tr -d '\r' < example.sh > example_fixed.sh

After doing one of the above you should be able to run your script:

$ ./example.sh
Hello world

Converting with Vim

Alternatively editors like Vim can also be used to convert scripts. Typing the following in normal mode will set the new line style and rewrite the script without any carriage returns:

:set ff=unix
:w

Note: ff is short for fileformat and w is short for write.