Error when running curl inside script (No such file or directory)

Question

I have the following .sh file I pieced together off of things I found on the internet.
The goal is to read the 2nd line 2nd item in a CSV file and use that to send a delete commmand to elasticsearch.

#!/bin/sh
OLDIFS=$IFS
IFS=,


function quit {
        echo "Quitting Script"
        IFS=$OLDIFS
    exit 1
}
function fileExists {
    if [ ! -f "$1" ]
        then
            echo "File $1 does not exists"
                quit
        fi
echo "File Name:  $1"
}
function work {
        linesToSkip=1
        {
            for ((i=$linesToSkip;i--;)) ;do
                read
            done
                #Read 2nd item of the 2nd line of CSV file to get PROGRAMURL
            read INDEX PROGRAMURL JUNK
                echo "$PROGRAMURL"
                QUERY="curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ \"query\" : { \"match\" : { \"PROGRAMURL\" : "$PROGRAMURL" } } }'"
                $("$QUERY")
                #RESPONSE=`$QUERY`
                #echo $RESPONSE

        } < $1

}

fileExists $1
work $1
IFS=$OLDIFS

Everything works except the execution of the curl script. I've tried it with $(), backtics, exec, and I cannot get it to work.

The following is the error when I run bash -vx script.sh:

bash -vx ./deleteExisting.sh catalog.csv
#!/bin/sh
OLDIFS=$IFS
+ OLDIFS='  
'
IFS=,
+ IFS=,


function quit {
    echo "Quitting Script"
    IFS=$OLDIFS
    exit 1
}
function fileExists {
    if [ ! -f "$1" ]
    then
        echo "File $1 does not exists"
        quit
    fi  
echo "File Name:  $1"
}
function work {
    linesToSkip=1
    {
        for ((i=$linesToSkip;i--;)) ;do
            read
        done
        #Read 2nd item of the 2nd line of CSV file to get PROGRAMURL
        read INDEX PROGRAMURL JUNK
        echo "$PROGRAMURL"
        QUERY="curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ \"query\" : { \"match\" : { \"PROGRAMURL\" : "$PROGRAMURL" } } }'"
        $("$QUERY")
        #RESPONSE=`$QUERY`
        #echo $RESPONSE

    } < $1  

}

fileExists $1
+ fileExists catalog.csv
+ '[' '!' -f catalog.csv ']'
+ echo 'File Name:  catalog.csv'
File Name:  catalog.csv
work $1
+ work catalog.csv
+ linesToSkip=1
+ (( i=1 ))
+ (( i-- ))
+ read
+ (( 1 ))
+ (( i-- ))
+ read INDEX PROGRAMURL JUNK
+ echo '"http://www.website.com"'
"http://www.website.com"
+ QUERY='curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '\''{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }'\'''
"$QUERY")
"$QUERY"
++ 'curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '\''{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }'\'''
./deleteExisting.sh: line 29: curl -XDELETE http://127.0.0.1:9200/cj/_query  -d '{ "query" : { "match" : { "PROGRAMURL" : "http://www.website.com" } } }': No such file or directory
IFS=$OLDIFS 
+ IFS='     
'

Example CSV file would be

INDEX, PROGRAMURL, other, info
"1", "http://website.com", "other info", "information"

Do not try to form a command out of pieces of strings and variables. Use a function. — n. m. could be an AI
– n. m. could be an AI, Commented Mar 30, 2014 at 4:59
@n.m. even with using functions, at some point in shell scripting it often becomes necessary to have to glue things together... this can often be an indicator it might be time to consider another language such as python though — 6EQUJ5
– 6EQUJ5, Commented Mar 30, 2014 at 6:01
Write a function named query. Pass $PROGRAMURL as a parameter. — n. m. could be an AI
– n. m. could be an AI, Commented Mar 30, 2014 at 6:04
he still needs to know how to join what then becomes $1 to the rest of the argument -d inside that function... — 6EQUJ5
– 6EQUJ5, Commented Mar 30, 2014 at 6:43

6EQUJ5 · Accepted Answer · 2014-03-30 06:39:02Z

Running $("$QUERY") actually tries to run in a sub-shell, a command called <the expanded value of $QUERY>, hence your '(bunch of stuff) No such file or directory' error.

You probably want something like:

CURLURL="http://127.0.0.1:9200/cj/_query"
CURLDATA='{ "query" : { "match" : { "PROGRAMURL" : "'$PROGRAMURL'" } } }'
RESPONSE=`curl -XDELETE "$CURLURL"  -d "$DATA"`

The trick here is how you nest the single and double quotes. Its a bit hard to explain concisely, but here goes:

Shell variables do not expand inside an outer single quote.
But you dont need to escape a double quote inside a single quote
So how we get $PROGRAMURL to expand above is immediately concatenate it between a pair of closing-opening single quotes.
To subsequently use the variable CURLDATA it has to get passed to the curl command inside double quotes
A much simpler example:

VARIABLE1="Hello, world"
VARIABLE2='This is "$verbatim" stuff"'$VARIABLE1'" More stuff'

This produces a value of $VARIABLE2 of This is "$verbatim" stuff"Hello, World" More stuff

Other things to note:

The above will put the entire stdout of the curl command into the variable RESPONSE
You will need to check $? afterwards to see if curl actually managed to talk to the host, etc. - it will be 0 if everything went OK
You may wish to turn off the progress bar

Skipping lines:

A better way to skip lines is to use:

tail -n +$(( $linesToSkip +1 ))

I updated the script with what you had said, and I'm having the same issue again, I think I'm doing it correctly.. Would you mind looking at it? stackoverflow.com/questions/22759076/… Thanks

Collectives™ on Stack Overflow

Error when running curl inside script (No such file or directory)

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related