Issue with shell command used to wrap noisy python script and remove specific lines with sed [duplicate]

Question

I am using a python script that outputs many warnings/errors (going to stderr) based on perfectly normal bad test certificates. Based on several SO posts, I was able find a way to run the script and ignore select stdout, stderr lines but its cumbersome:

runThing 3>&1 1>&2 2>&3 3>&- | grep -r 's/Insecure/'
 OR 
runThing 3>&1 1>&2 2>&3 3>&- | sed 's/Insecure/g'

Both filter out lots of lines like:

 /Users/xxx/.blah/lib/python2.7/site-   packages/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html  InsecureRequestWarning)

The sed one has this error, though: sed: 1: "s/Insecure/g": unterminated substitute in regular expression

Its a lot to add at the end of a line that is used a lot (the real runThing has commands and parameters), so I am trying to make a command like:

runThingClean() { command runThing "$@" > /dev/null 3>&1 1>&2 2>&3 3>&- | sed 's/Insecure/g' & }

When I run this, it now fails to filter (and shows the error is sed is used):

 sed: 1: "s/Insecure/g": unterminated substitute in regular expression

Can someone help me fix this command?

Thanks in advance..

Answer 1

Don't do that at all: Just tell Python not to print the warnings in the first place, and then you have no need to filter them out.

python -W ignore yourprogram ...

Alternately, consider modifying your code calling the relevant components to suppress the warning at the call site, as given in the Python documentation:

import warnings
with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    urllib3.doTheThingThatCausedTheWarning(...)

---

Finally, if you really want to do it out-of-process as a filter, just use grep -v Insecure; sed is a larger hammer than is needed for this job.

sh -c 'printf "%s\n" "stderr: keep" "stderr: Insecure" >&2; echo "stdout"' \
  3>&1 1>&2 2>&3 3>&- | grep -v Insecure 3>&1 1>&2 2>&3 3>&-

What all this redirection is doing is swapping stdout and stderr twice -- once before the grep, and once after to restore them. Let's look at how this works:

• FD 1 (original stdout) is copied to FD 3 (to serve as backup)
• FD 2 (original stderr) is copied to FD 1 (so it can be filtered)
• FD 3 (the backup of original stdout) is copied to FD 2 (now stored in stderr)
• FD 3 is closed, as it's no longer needed with the swap complete.

Answer 2

The error means you didn't close the s in sed correctly. For s it should be s/regex/substitution/g

If you are trying to remove Insecure or substitute:

sed 's/Insecure/YOUR_SUBSTITUTION/g'

if you are trying to display only lines with Insecure:

sed -n '/Insecure/p'