SOLVED: Escaping strings in Bash

Update, 2011-06-13:

I raised a question on Stack Overflow, How can I escape an arbitrary string for use as a command line argument in Bash?. Several people contributed incomplete or incorrect answers but eventually I figured out the following.

To escape a string for use as a command line argument in Bash, simply put a backslash in front of every non-alphanumeric character. Do not wrap the string in single quotes or double quotes. Escape everything that is non-alphanumeric, including spaces, exclamation marks, dollar signs, ampersands, angle brackets, double quotes and single quotes.

In Perl, the code looks like this:

my @args = (
    # a list of arbitrary strings...
);

$cmd = join " ", map { escape($_); } @args;

sub escape {
    my $arg = shift;
    $arg =~ s/([^a-zA-Z0-9])/\\$1/g;
    return $arg;
}

In fact, you can leave some characters unescaped safely, such as underscores. But there seems to be no complete list of "dangerous" characters in Bash, and you can escape any character in Bash with impunity, so it is safest to just escape everything.

Original problem

This has been driving me nuts at work lately and I can't find anything about it using Google. Since this site seems to get a decent number of highly intelligent readers, I'm sure one of you will be able to answer this. This should also prove my "if you want to find the right answer to something, the quickest way is to put the wrong answer online and wait for someone to correct you" hypothesis.

In the Bash shell, how do you escape an arbitrary string for encapsulation in quotes?

Let's say my string was:

"That's impossible!" he said.

Note that this string includes double quotes, an apostrophe and an exclamation mark. Now, I want a function or procedure which will give me a new string so that I can do:

$ echo <the new string>
"That's impossible!" he said.

Here are the problems I get, though:

$ echo "That's impossible!" he said
bash: !": event not found

$ echo "That's impossible\!" he said
That's impossible\! he said

$ echo ""That's impossible!" he said"
>

$ echo "\"That's impossible!\" he said"
bash: !\": event not found

$ echo "\"That's impossible\!\" he said"
"That's impossible\!" he said

$ echo '"That's impossible!" he said'
bash: !": event not found

$ echo '"That\'s impossible!" he said'
bash: !": event not found

$ echo '"That\'
"That\

$ echo '\''
>

$ echo '''
>

$ echo "!"
bash: !: event not found

$ echo "\!"
\!

There's no way to put a literal single quote into a single-quoted string because a single-quoted string doesn't allow you to escape anything. The single quote will invariably terminate the string in progress and everything afterwards will be interpreted or whatever. But there's also no way to put a literal exclamation mark into a double-quote string because either it will get expanded into some previous event, or if you do backslash-escape it, the backslash gets printed as well! So if you have a string containing both an exclamation mark and a single quote - or arbitrary numbers of both - is there no way to escape it at all?

Is there something I'm missing here? I hope the solution isn't "install a different shell" or "radically modify the existing shell" because I interact with dozens of distinct machines which have this problem, as do dozens of other people and automated test processes.

Discussion (48)

2010-06-22 13:23:43 by Sticky:

$ echo "\"That's impossible"'!'"\" he said." "That's impossible!" he said. Suitable?

2010-06-22 13:58:17 by aaroncrane:

You’re correct that there’s no way of embedding a single-quote in a single-quoted literal in Bourne shell. However, that’s not actually necessary, because you *can* concatenate multiple string-like tokens. So you can break out of single quotes, append a backslashed single-quote, and start again. It doesn’t end up pretty or easy to read, though: echo '"That'\''s impossible!" he said.' So, in summary, the way you embed a single-quote character into a single-quoted literal in Bourne shell is with **'\''**. Share and enjoy!

2010-06-22 14:45:56 by qntm:

I guess that's the closest I'm going to get. Alas, neither of those cases encapsulate the whole string in a single type of quote mark. In both cases I'd have to break out of the original quoted string, and possibly even quote another, separate string, before reopening the quotes for the rest. These techniques use what basically amounts to code injection in order to work properly! Seems like a pretty glaring oversight in the shell. It may work anyway, I'll let you know.

2010-06-22 15:03:22 by Pi:

I tried dollar-quotes ($'foo'): bash-4.0$ echo $'"That\'s impossible\!" he said' "That's impossible\!" he said Unfortunately, that doesn't quite work for the bang. Workaround! bash-4.0$ echo $'"That\'s impossible\x21" he said' "That's impossible!" he said

2010-06-22 15:27:17 by Jim:

Can someone with zsh try this out and let us (all) know if it's better in any way? (Once I start working on a bash shell beyond 15 lines, I switch to python)

2010-06-22 19:04:38 by Mike:

Well, this is all said and true (and I hate Bash for that). Still, if you're familiar with Python, you'll try what others have said before me: try concatenating multiple strings. There's no other way around Bash's crazy, evil parser. It may be heavy on the character side (I count 38 characters plus spaces in the first suggestion), but you can't put the whole thing in a single string.

2010-06-22 21:56:25 by TJSomething:

I never knew about dollar quotes, but a quick experiment showed that you only need to escape single quotes in dollar quotes. $ echo $' !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~' !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

2010-06-22 23:19:26 by skztr:

I find it convenient that single-quotes in bash mean "absolutely no special characters" This makes it easy to escape arbitrary strings with only one rule: turn ' into '\''. I do this with the following routine: function esc(){ local escaped="'\''" echo "${1//\'/$escaped}" } I'm not quite sure what you're asking for, though. The above turns an already-created variable into one which is safe to use in eval. If you're asking for a reliable way to allow a raw string to appear directly in code, that could be handled by a HERE document: foo="$(cat <<'ENDOFVAR' Anything you want goes here, even things like ! and " and ' ENDOFVAR )" I'm sure some bash purists can give a way to do that without the use of "cat" or the extra parens. I'd call what I've given "an ugly hack", but if I understand the question (and I'm pretty sure I don't), that'll do what you want. As a last resort: the people in #bash on freenode are knowledgeable and generally helpful, if a little rough around the edges. If you can get past them acting like your question is the dumbest thing they've ever heard, there's usually a concise and well-written answer that comes right along with it.

2010-06-22 23:35:08 by Grayswx:

Just run through and replace the sensitive characters with their ascii escapes: echo -e "\0042That\0047s Impossible\0041\0042 he said." "That's Impossible!" he said.

2010-06-23 00:12:38 by pascal:

in ZSH it's just echo "That's impossible\!" (I don't know a lot about shells, but ZSH is just nice)

2010-06-27 11:29:42 by pozorvlak:

This doesn't really meet your spec, but it's easy enough to get your desired output by simply leaving off the outer quotes: ~$ echo \"That\'s impossible\!\" he said. "That's impossible!" he said.

2010-06-27 11:51:58 by pozorvlak:

But really, this is the point where I tend to give up and use Perl.

2010-08-05 12:41:30 by Chris:

echo "\""That\'s impossible,\" he said."" works for me and is completely enclosed in DOUBLE quotes, if that helps

2010-08-05 15:09:38 by qntm:

You omitted the exclamation mark. That prints "That's impossible," he said. instead of "That's impossible!" he said.

2010-11-11 02:20:17 by npostavs:

You can use set +o histexpand to disable expansion of ! (it's also disabled by default in non interactive shells)

2011-06-13 17:26:57 by dankuck:

Thank you! And I think I speak for us all.

2011-08-01 22:05:03 by zoon:

It's sad that the basics disappear slowly but more and more... "Mastering Regular Expressions" - Every answer you're looking for could be found at http://oreilly.com/catalog/9781565922570 It's a pity; everything seems to dissapear... with best regards

2011-08-13 10:45:15 by Nathan:

I was so fed up with this in bash. Every time I wanted to write a commit message, I would either not be allowed to use an exclamation mark, or I couldn't write words like couldn't and don't. So I bound some keyboard shortcuts to make this possible. Now I use a 'read' prompt, and can type whatever I want! bind "\"\C- \": \"git_prompt 'git commit'\n\"" bind "\"\C-a \": \"git_prompt 'git commit -a'\n\"" git_prompt(){ read -p "Commit message for '$1': " git_msg; echo $git_msg | $1 -F -; }

2013-02-28 10:19:38 by Kalin:

[admin@sf-escalbani-director ~]$ echo '"That'''s mine!" you idiot!' "That's mine!" you idiot!

2013-02-28 10:26:39 by Kalin:

My last comment was parsed incorrectly. A way to escape the single quote would be to put the whole string in sigle qoutes and escape quote(s) inside. like: ''' <single_quote><single_quote><single_quote>

2013-02-28 12:14:46 by qntm:

/:>echo '"That'''s mine!" you idiot!' bash: !': event not found

2013-05-06 06:06:49 by Brian:

rather hilariously: $echo "\"That's impossible"""\!"""\" he said." gets the exact desired output and escapes with a single quotation... "That's impossible!" he said.

2013-06-24 22:19:31 by spirit:

$ echo '"That'"'"'s impossible!" he said.' "That's impossible!" he said. $ echo $'"That\'s impossible!" he said.' "That's impossible!" he said.

2013-07-10 22:23:14 by Stephen:

You could also use `printf "%q" $FOO` which handles shell escape chars quite handily. e.g. FOO='myTest3$t%^&*()"' FOO=`printf "%q" $FOO` # yeilds myTest3\$t%\^\&\*\(\)\" echo $FOO

2013-09-26 00:44:28 by Chris:

THANK YOU! Discovering this was such a relief! Why is there no definitive list? With this solution who cares, brilliant, thanks

2014-08-08 12:46:12 by pimgeek:

It seems [ echo \""That's impossible!"\" he said. ] will do the job.

2014-08-10 15:05:07 by squarebash:

Hi, The usual thing to do is: <pre> $ echo '"That'\''s impossible!" he said' "That's impossible!" he said </pre>

2015-08-21 06:39:09 by devey:

Thanks for the explanation and solutions. It helped.

2015-10-27 18:38:46 by Resuna:

Why are you trying to quote/escape arbitrary strings to feed to bash (or any other shell) in the first place? If you're dealing with a variable in the script, always quote it, and that will prevent expansion. If you're calling an external program, use execlp()/execvp() or something derived therefrom. If you're having this problem a lot, you've got a design flaw somewhere in the toolchain. See if you can fix that.

2015-10-27 19:50:16 by qntm:

As is stated very clearly above, I wanted to echo an exclamation point.

2015-10-27 21:30:24 by Resuna:

Um. "How can I escape an arbitrary string for use as a command line argument in Bash?" That's not "How can I echo an exclamation point." If\ you\'re\ typing\ the\ text\ in\ at\ the\ command\ line\ then\ putting\ backslash\ before\ every\ non\-alphabetic\ character\ is\ not\ the\ answer\ because\ it\'s\ a\ pain\ in\ the\ ass. "You quote the whole string. You play silly quoting games. It's a pain. There isn't a general answer because they borrowed the behavior of exclamation for command history from csh without thinking about it a lot. That's not the only way bash is screwed up." "But it's something you do ad-hoc, manually, and probably differently each time. And you cuss out Brian Fox and Bill Joy and the rest of the GNU and BSD people who didn't think this shit through." "But what you don't do is write a script to do it, because if you're writing a script to do it, you're doing it because you're doing something dangerous. The only time you're escaping the whole string is if you're passing it from another program." "And you don't want to do that. REALLY."

2015-10-27 22:06:26 by qntm:

Sorry, I guess I wasn't being clear enough on what the process was. I wanted to echo an exclamation point at the command line, and it was giving me some trouble, so I first wanted to answer that specific case. But I thought there might be other dangerous metacharacters in Bash, so I reckoned I might as well cut to the chase and find out the best general approach. Maybe a full list of those characters. This would let me be more confident in doing things like assigning values to environment variables without literal quotes disappearing, and writing short Perl scripts at the command line. I take your point that using a Perl script to generate command lines should never be necessary but regrettably it sometimes is, and I agree with you that it's unfortunate that nobody thought this through very clearly.

2015-10-27 23:24:48 by Resuna:

Probably the #1 source of webserver exploits is injection attacks. SQL seems to be more common, but passing user generated data to shell commands is a problem too. To defend against SQL injection attacks you use stored procedures. Somehow people don't seem to think about writing a script that can be passed a simple string, and using it like a stored procedure, passing the input to the script using exec(). If the script is rigorous about handling variable expansions, this is much safer than quoting and escaping. So... if you really think about it, you can probably eliminate most of the "sometimes necessary" uses of system().

2015-11-05 18:12:27 by Anon:

Off topic: Stored procedures can be vulnerable to SQL injection too. Use parameterized queries!

2015-11-08 02:38:47 by NoLongerFrustrated:

I've also been plagued with the issue until I spent today trying to figure out a method and I stumbled on the solution by accident using the "read" command. Before I give away the solution I want to dive a little bit into why I knew there must be solution because I noticed if reading in a file and set a variable it had properly added the escapes when unbalanced quotes or even an exclamation point existed. It's only through the BASH CLI where it would complain. This method does not work if you want to pass as an argument, but it does work interactively, including through the use of a function. It is actually quite simple. Put the command into the terminal and you will notice how the system is properly escaping where needed ~$ read -er; foo="$REPLY"; set | grep foo Now type in the phrase: "That's impossible!" he said. And, the output will be: foo='"That'\''s impossible!" he said.' To use this method to echo the same phrase, use the following: ~$ read -er; echo "$REPLY" There are many times where I need to paste a long path but cd will complain if the path name contains single quotes or exclamation points. This solution also works for cd. ~$ read -er; cd "$REPLY"

2016-05-12 11:13:03 by WowzerMcPowzer:

Oh my gods! You configured it out!

2016-09-17 09:50:04 by Areve:

you should do echo $'"That\'s impossible!" he said.'

2016-11-09 16:33:39 by jkl:

$ echo '"That'"'"'s impossible!" he said.' "That's impossible!" he said. $ echo '"That'"'"'s impossible!" he said.' | sed -E "s/'/'\"'\"'/g; s/^/'/; s/$/'/" | xargs "That's impossible!" he said. Funny that this thread is almost as old as the Obama administration, and yet no one has answered it in the simplest way possible. Bourne shell strings can be concatenated. Instead of mad escaping, the above simply puts everything except a single quote in single quotes. Single quotes embedded in the literal are wrapped in double quotes. The way to think about it: the shell just trips along, de-quoting until it hits an unquoted space (IFS). As long as it's inside a single-quoted string, no interpolation is done. If, on reaching the closing single quote it encounters a double quote, it continues interpreting the *same* string, but changes modes (to double-quote interpolation). You can continue that ssddss pattern indefinitely, and never need to use a backslash.

2017-01-18 12:44:54 by druuu:

@jkl echo '"That'"'"'s impossible!" he said.' Thats the best simple solution. Thank You

2017-03-17 06:09:01 by binaryphile:

Heredocs can be useful to avoid the quoting unpleasantness: $ read -r quote <<'EOS' >"That's impossible!" he said. >EOS $ echo "$quote" "That's impossible!" he said.

2017-03-21 16:48:40 by Viqsi:

For the record, none of this appears to work for multiline strings. (Yes, this did come up for me recently. Thankfully, it's not something I have to automate, or else I'd be in real trouble.)

2017-03-28 15:21:44 by Ingvar:

Viqsi: You can use $( ... ) to construct a multi-line single argument, as long as it's embedded in double-quotes. If it's not embedded in double-quotes, the shell will interpret newlines (and spaces) as IFS (essentially, as argument separators). This would be one example: echo "$(echo first; echo second; echo third and final line)"

2017-11-05 01:41:09 by Jpc:

Thanks. I have slightly modified the method for it to work in presence of newlines. I make a special case for \n and I replace it with the sequence "'\n'". My C++ implementation: for (char c; (c = *arg); ++arg) { if (c == '\n') { cmd.append("'\n'"); } else { bool alpha = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); bool num = (c >= '0' && c <= '9'); if (!alpha && !num) cmd.push_back('\\'); cmd.push_back(c); } }

2018-02-07 23:11:06 by nbercher:

That was surely the most delightful bash discussion I've ever read! Considering ! character, the issue only concerns shells with enabled history expansion (usually the interactive ones). Now, if one has to use random text as typed by random users from an interactive shell, the best solution to me seems to disable history expansion. Since I don't use a lot of *nix fashions, I expect =set +o histexpand= not to be supported by some older bash versions or maybe this is not POSIX or whatever. One of the most interesting solution to this problem I've seen here is the one from NoLongerFrustrated, involving the =read= command as stdin reader (doesn't this sound like... obvious from the Unix old ages?!). But I would do it shorter (-e being just for optional comfort and -r to allow \ as literal or not to support line breaks not being newline): $ read line "That's impossible!" he said. $ echo ${line} Another way of seeing the problem might be to just "literally" read stdin: $ cat /dev/stdin "That's impossible!" he said. "That's impossible!" he said. ^D # this is ctrl+d to end the reading of stdin

2018-07-18 09:45:06 by Foo:

Why don't you use double quotes? Wouldn't that allow you to not escape whitespace?

2018-10-22 13:30:33 by Phil:

This works for me: echo '"'That"'"s impossible"!"'"' he said "That's impossible!" he said

2019-11-05 11:42:27 by Mike Dixson:

printf "That's impossible"\!" he said" Works for me! i.e. exclaimation character outside of quotes but still escaped.

2022-02-06 23:11:52 by Jack L Hamilton:

The following worked for multiline: # Redirect to file $ cat << 'EOF' > /tmp/data.txt heredoc> "That's impossible!" she said heredoc> "That's impossible!" they said heredoc> EOF $ cat /tmp/data.txt "That's impossible!" she said "That's impossible!" they said # Output to screen and redirect to file: $ cat << 'EOF' | tee /tmp/data.txt pipe heredoc> "That's impossible!" he said pipe heredoc> "That's impossible!" she said pipe heredoc> "That's impossible!" they said pipe heredoc> EOF "That's impossible!" he said "That's impossible!" she said "That's impossible!" they said # SOURCE: https://linuxize.com/post/bash-heredoc/ # Refer to link for examples with indentation in code.

New comment by :

Plain text only. Line breaks become <br/>
The square root of minus one: