SOLVED: Escaping strings in Bash

Update, 2011-06-13:

I raised a question on Stack Overflow, How can I escape an arbitrary string for use as a command line argument in Bash?. Several people contributed incomplete or incorrect answers but eventually I figured out the following.

To escape a string for use as a command line argument in Bash, simply put a backslash in front of every non-alphanumeric character. Do not wrap the string in single quotes or double quotes. Escape everything that is non-alphanumeric, including spaces, exclamation marks, dollar signs, ampersands, angle brackets, double quotes and single quotes.

In Perl, the code looks like this:

my @args = (
    # a list of arbitrary strings...
);

$cmd = join " ", map { escape($_); } @args;

sub escape {
    my $arg = shift;
    $arg =~ s/([^a-zA-Z0-9])/\\$1/g;
    return $arg;
}

In fact, you can leave some characters unescaped safely, such as underscores. But there seems to be no complete list of "dangerous" characters in Bash, and you can escape any character in Bash with impunity, so it is safest to just escape everything.

Original problem

This has been driving me nuts at work lately and I can't find anything about it using Google. Since this site seems to get a decent number of highly intelligent readers, I'm sure one of you will be able to answer this. This should also prove my "if you want to find the right answer to something, the quickest way is to put the wrong answer online and wait for someone to correct you" hypothesis.

In the Bash shell, how do you escape an arbitrary string for encapsulation in quotes?

Let's say my string was:

"That's impossible!" he said.

Note that this string includes double quotes, an apostrophe and an exclamation mark. Now, I want a function or procedure which will give me a new string so that I can do:

$ echo <the new string>
"That's impossible!" he said.

Here are the problems I get, though:

$ echo "That's impossible!" he said
bash: !": event not found

$ echo "That's impossible\!" he said
That's impossible\! he said

$ echo ""That's impossible!" he said"
>

$ echo "\"That's impossible!\" he said"
bash: !\": event not found

$ echo "\"That's impossible\!\" he said"
"That's impossible\!" he said

$ echo '"That's impossible!" he said'
bash: !": event not found

$ echo '"That\'s impossible!" he said'
bash: !": event not found

$ echo '"That\'
"That\

$ echo '\''
>

$ echo '''
>

$ echo "!"
bash: !: event not found

$ echo "\!"
\!

There's no way to put a literal single quote into a single-quoted string because a single-quoted string doesn't allow you to escape anything. The single quote will invariably terminate the string in progress and everything afterwards will be interpreted or whatever. But there's also no way to put a literal exclamation mark into a double-quote string because either it will get expanded into some previous event, or if you do backslash-escape it, the backslash gets printed as well! So if you have a string containing both an exclamation mark and a single quote - or arbitrary numbers of both - is there no way to escape it at all?

Is there something I'm missing here? I hope the solution isn't "install a different shell" or "radically modify the existing shell" because I interact with dozens of distinct machines which have this problem, as do dozens of other people and automated test processes.

Back to Blog
Back to Things Of Interest

Facebook Twitter Reddit Email Hacker News StumbleUpon

Discussion (25)

2010-06-22 13:23:43 by Sticky:

$ echo "\"That's impossible"'!'"\" he said."
"That's impossible!" he said.

Suitable?

2010-06-22 13:58:17 by aaroncrane:

You’re correct that there’s no way of embedding a single-quote in a single-quoted literal in Bourne shell. However, that’s not actually necessary, because you can concatenate multiple string-like tokens. So you can break out of single quotes, append a backslashed single-quote, and start again. It doesn’t end up pretty or easy to read, though:

echo '"That'\''s impossible!" he said.'

So, in summary, the way you embed a single-quote character into a single-quoted literal in Bourne shell is with '\''. Share and enjoy!

2010-06-22 14:45:56 by Sam:

I guess that's the closest I'm going to get. Alas, neither of those cases encapsulate the whole string in a single type of quote mark. In both cases I'd have to break out of the original quoted string, and possibly even quote another, separate string, before reopening the quotes for the rest. These techniques use what basically amounts to code injection in order to work properly! Seems like a pretty glaring oversight in the shell. It may work anyway, I'll let you know.

2010-06-22 15:03:22 by Pi:

I tried dollar-quotes ($'foo'):

bash-4.0$ echo $'"That\'s impossible\!" he said'
"That's impossible\!" he said

Unfortunately, that doesn't quite work for the bang. Workaround!
bash-4.0$ echo $'"That\'s impossible\x21" he said'
"That's impossible!" he said

2010-06-22 15:27:17 by Jim:

Can someone with zsh try this out and let us (all) know if it's better in any way?

(Once I start working on a bash shell beyond 15 lines, I switch to python)

2010-06-22 19:04:38 by Mike:

Well, this is all said and true (and I hate Bash for that). Still, if you're familiar with Python, you'll try what others have said before me: try concatenating multiple strings. There's no other way around Bash's crazy, evil parser. It may be heavy on the character side (I count 38 characters plus spaces in the first suggestion), but you can't put the whole thing in a single string.

2010-06-22 21:56:25 by TJSomething:

I never knew about dollar quotes, but a quick experiment showed that you only need to escape single quotes in dollar quotes.

$ echo $' !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

2010-06-22 23:19:26 by skztr:

I find it convenient that single-quotes in bash mean "absolutely no special characters" This makes it easy to escape arbitrary strings with only one rule: turn ' into '\''.

I do this with the following routine:
function esc(){
local escaped="'\''"
echo "${1//\'/$escaped}"
}

I'm not quite sure what you're asking for, though. The above turns an already-created variable into one which is safe to use in eval. If you're asking for a reliable way to allow a raw string to appear directly in code, that could be handled by a HERE document:

foo="$(cat <<'ENDOFVAR'
Anything you want goes here, even things like ! and " and '
ENDOFVAR
)"

I'm sure some bash purists can give a way to do that without the use of "cat" or the extra parens. I'd call what I've given "an ugly hack", but if I understand the question (and I'm pretty sure I don't), that'll do what you want.

As a last resort: the people in #bash on freenode are knowledgeable and generally helpful, if a little rough around the edges. If you can get past them acting like your question is the dumbest thing they've ever heard, there's usually a concise and well-written answer that comes right along with it.

2010-06-22 23:35:08 by Grayswx:

Just run through and replace the sensitive characters with their ascii escapes:

echo -e "\0042That\0047s Impossible\0041\0042 he said."
"That's Impossible!" he said.

2010-06-23 00:12:38 by pascal:

in ZSH it's just
echo "That's impossible\!"
(I don't know a lot about shells, but ZSH is just nice)

2010-06-27 11:29:42 by pozorvlak:

This doesn't really meet your spec, but it's easy enough to get your desired output by simply leaving off the outer quotes:

~$ echo \"That\'s impossible\!\" he said.
"That's impossible!" he said.

2010-06-27 11:51:58 by pozorvlak:

But really, this is the point where I tend to give up and use Perl.

2010-08-05 12:41:30 by Chris:

echo "\""That\'s impossible,\" he said.""

works for me and is completely enclosed in DOUBLE quotes, if that helps

2010-08-05 15:09:38 by Sam:

You omitted the exclamation mark. That prints

"That's impossible," he said.

instead of

"That's impossible!" he said.

2010-11-11 02:20:17 by npostavs:

You can use

set +o histexpand

to disable expansion of ! (it's also disabled by
default in non interactive shells)

2011-06-13 17:26:57 by dankuck:

Thank you! And I think I speak for us all.

2011-08-01 22:05:03 by zoon:

It's sad that the basics disappear slowly but more and more...
"Mastering Regular Expressions" - Every answer you're looking for could be found at http://oreilly.com/catalog/9781565922570
It's a pity; everything seems to dissapear...
with best regards

2011-08-13 10:45:15 by Nathan:

I was so fed up with this in bash. Every time I wanted to write a commit message, I would either not be allowed to use an exclamation mark, or I couldn't write words like couldn't and don't.

So I bound some keyboard shortcuts to make this possible. Now I use a 'read' prompt, and can type whatever I want!

bind "\"\C- \": \"git_prompt 'git commit'\n\""
bind "\"\C-a \": \"git_prompt 'git commit -a'\n\""
git_prompt(){ read -p "Commit message for '$1': " git_msg; echo $git_msg | $1 -F -; }

2013-02-28 10:19:38 by Kalin:

[admin@sf-escalbani-director ~]$ echo '"That'''s mine!" you idiot!'
"That's mine!" you idiot!

2013-02-28 10:26:39 by Kalin:

My last comment was parsed incorrectly.
A way to escape the single quote would be to put the whole string in sigle qoutes and escape quote(s) inside.

like:

'''
<single_quote><single_quote><single_quote>

2013-02-28 12:14:46 by Sam:

/:>echo '"That'''s mine!" you idiot!'
bash: !': event not found

2013-05-06 06:06:49 by Brian:

rather hilariously:

$echo "\"That's impossible"""\!"""\" he said."

gets the exact desired output and escapes with a single quotation...

"That's impossible!" he said.

2013-06-24 22:19:31 by spirit:

$ echo '"That'"'"'s impossible!" he said.'
"That's impossible!" he said.

$ echo $'"That\'s impossible!" he said.'
"That's impossible!" he said.

2013-07-10 22:23:14 by Stephen:

You could also use `printf "%q" $FOO` which handles shell escape chars quite handily.

e.g.
FOO='myTest3$t%^&*()"'
FOO=`printf "%q" $FOO`
# yeilds myTest3\$t%\^\&\*\(\)\"
echo $FOO

2013-09-26 00:44:28 by Chris:

THANK YOU! Discovering this was such a relief! Why is there no definitive list? With this solution who cares, brilliant, thanks