2

Is there any reason why a curl command won't work with variables?

Here is my set up - I am not using real values as I can not expose client data and I have no test environment.

You will notice that I am using two curl commands, yes I know it can be combined into one command, but I haven't been able to make that work so I am splitting then up unto two separate commands to test with.

Here I set up the paths for inbound folder on my system:

[root@acme logs]# act="roadrunner"
[root@acme logs]# content="xml"
[root@acme logs]# inbndfldr='/ftp/'"$act"'/inbound'
[root@acme logs]# destinationFolder="$inbndfldr"/"$content"/
[root@acme logs]# logFolder="$inbndfldr"/logs

Here I set up the paths for outbound folder on the remote system:

[root@acme logs]# targetHost="SFTP://welcometotheroad.net"
[root@acme logs]# THFolderName="OUTBOUND"
[root@acme logs]# THFolder='/'"$THFolderName"
[root@acme logs]# THSftpAddress=$targetHost$THFolder
[root@acme logs]# THDeleteSftpAddress=$targetHost$THFolder'/'

Here I set up authentication on the remote system:

[root@acme logs]# username=mrRoadRunner
[root@acme logs]# password=Beep-8eep!
[root@acme logs]# sessionAuthentication="$username":"$password"

Here I set the file reference:

[root@acme logs]# fileref="openroad2019.xml"

Here I verify that the file exists on the remote server:

[root@acme logs]# THFiles=$(curl -sSk -u $sessionAuthentication $THSftpAddress | grep $fileref | awk '{print $9}')
[root@acme logs]# echo $THFiles
[root@acme logs]# openroad2019.xml

Here I begin my loop:

[root@acme logs]# for targetFile in $THFiles
> do
>   dateTimeStamp=`date +%Y%m%d%H%M%S`
>   traceLog="$logFolder"/"$targetFile"trace"$dateTimeStamp".log
>   testLog="$logFolder"/test"$dateTimeStamp".log
>   deltraceLog="$logFolder"/del"$targetFile"trace"$dateTimeStamp".log
>   deltestLog="$logFolder"/deltest"$dateTimeStamp".log
>   deleteString='"-rm '"$THFolderName"'/'"$targetFile"'"'
>   masterGetStr=`echo curl \-k \-\-trace "$traceLog" \-\-user "$sessionAuthentication" \-O "$THSftpAddress"\/"$targetFile" \-\-ftp\-ssl \&\>\>"$testLog" 2\>\&1`
>   masterDeleteStr=`echo curl \-sSk \-\-trace $deltraceLog \-u $sessionAuthentication $THDeleteSftpAddress \-Q$deleteString  \&\>\> $deltestLog 2\>\&1`
>   echo
>   echo \-\-\-\-\-
>   echo "$masterGetStr"
>   echo \-\-
>   echo "$masterDeleteStr"
>   echo \-\-\-\-\-
>   echo
>   curl -k --trace "$traceLog" --user "$sessionAuthentication" -O "$THSftpAddress"/"$targetFile" --ftp-ssl &>> "$testLog" 2>&1
>   curl -sSk --trace $deltraceLog -u $sessionAuthentication $THDeleteSftpAddress -Q$deleteString  &>> $deltestLog 2>&1
> done

Here is the output:

-----
curl -k --trace /ftp/roadrunner/inbound/logs/openroad2019.xmltrace20190628180349.log --user mrRoadRunner:Beep-8eep! -O SFTP://welcometotheroad.net/OUTBOUND/openroad2019.xml --ftp-ssl &>>/ftp/roadrunner/inbound/logs/test20190628180349.log 2>&1
--
curl -sSk --trace /ftp/roadrunner/inbound/logs/delopenroad2019.xmltrace20190628180349.log -u mrRoadRunner:Beep-8eep! SFTP://welcometotheroad.net/OUTBOUND/ -Q"-rm OUTBOUND/openroad2019.xml" &>> /ftp/roadrunner/inbound/logs/deltest20190628180349.log 2>&1
-----

The first curl command works just fine, drops the file right where I need it.

The second curl command is supposed to delete the file on the remote system and it is not working. Here I view the logs on that:

::::::::::::::
delopenroad2019.xmltrace20190628180349.log
::::::::::::::
== Info:   Trying 0.0.0.0...
== Info: TCP_NODELAY set
== Info: Connected to welcometotheroad.net (0.0.0.0) port 22 (#0)
== Info: SSH MD5 fingerprint: Th3re1$4WarpDr1v3N$+4r$#1P!
== Info: SSH authentication methods available: publickey,password
== Info: Using SSH private key file '/root/.ssh/id_rsa'
== Info: SSH public key authentication failed: Username/PublicKey combination invalid
== Info: Initialized password authentication
== Info: Authentication complete
== Info: Sending quote commands
== Info: Syntax error in SFTP command. Supply parameter(s)!
== Info: Connection #0 to host welcometotheroad.net left intact
== Info: getaddrinfo(3) failed for OUTBOUND:80
== Info: Couldn't resolve host 'OUTBOUND'
== Info: Closing connection 1
::::::::::::::
deltest20190628180349.log
::::::::::::::
curl: (21) Syntax error in SFTP command. Supply parameter(s)!
curl: (6) Couldn't resolve host 'OUTBOUND'
::::::::::::::
test20190628180349.log
::::::::::::::
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4290  100  4290    0     0   3248      0  0:00:01  0:00:01 --:--:--  3252
100  4290  100  4290    0     0   3247      0  0:00:01  0:00:01 --:--:--  3247
[root@acme logs]#

Yet, when I type in the command manually - it works and deletes the file off the remote system. 

curl -sSk --trace /ftp/roadrunner/inbound/logs/delopenroad2019.xmltrace20190628180349.log -u mrRoadRunner:Beep-8eep! SFTP://welcometotheroad.net/OUTBOUND/ -Q"-rm OUTBOUND/openroad2019.xml" &>> /ftp/roadrunner/inbound/logs/deltest20190628180349.log 2>&1

[root@rh1 logs]# more *20190628180546*.log

::::::::::::::
openroad2019.xmltrace20190628180546.log
::::::::::::::
Info:   Trying 0.0.0.0...
== Info: TCP_NODELAY set
== Info: Connected to welcometotheroad.net  (0.0.0.0) port 22 (#0)
== Info: SSH MD5 fingerprint: Th3re1$4WarpDr1v3N$+4r$#1P!
== Info: SSH authentication methods available: publickey,password
== Info: Using SSH private key file '/root/.ssh/id_rsa'
== Info: SSH public key authentication failed: Username/PublicKey combination invalid
== Info: Initialized password authentication
== Info: Authentication complete
<= Recv data, 2000 bytes (0x7d0)
0000: ef bb bf 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e ...<?xml version
...
... (details removed)
...
0110: 6e 73 74 72 75 63 74 69 6f 6e 73 48 65 61 64 65 nstructionsHeade
0120: 72 3e                                           r>
== Info: Connection #0 to host welcometotheroad.net  left intact

::::::::::::::
del_openroad2019.xmltrace20190628180546.log
::::::::::::::
== Info:   Trying 0.0.0.0...
== Info: TCP_NODELAY set
== Info: Connected to welcometotheroad.net  (0.0.0.0) port 22 (#0)
== Info: SSH MD5 fingerprint: Th3re1$4WarpDr1v3N$+4r$#1P!
== Info: SSH authentication methods available: publickey,password
== Info: Using SSH private key file '/root/.ssh/id_rsa'
== Info: SSH public key authentication failed: Username/PublicKey combination invalid
== Info: Initialized password authentication
== Info: Authentication complete
=> Send data, 55 bytes (0x37)
0000: 64 72 77 78 72 2d 78 2d 2d 2d 20 32 20 66 74 70 drwxr-x--- 2 ftp
0010: 61 64 6d 69 6e 20 66 74 70 61 64 6d 69 6e 20 20 admin ftpadmin  
...
... (details removed)
...
0050: 30 30 2d 32 36 37 36 39 36 37 33 5f 44 49 5f 4c 00-26769673_DI_L
0060: 45 30 2e 58 4d 4c 0a                            E0.XML.
== Info: Sending quote commands
== Info: Connection #0 to host welcometotheroad.net  left intact

::::::::::::::
delopenroad2019.xmltrace20190628180546.log
::::::::::::::
== Info:   Trying 0.0.0.0...
== Info: TCP_NODELAY set
== Info: Connected to welcometotheroad.net  (0.0.0.0) port 22 (#0)
== Info: SSH MD5 fingerprint: Th3re1$4WarpDr1v3N$+4r$#1P!
== Info: SSH authentication methods available: publickey,password
== Info: Using SSH private key file '/root/.ssh/id_rsa'
== Info: SSH public key authentication failed: Username/PublicKey combination invalid
== Info: Initialized password authentication
== Info: Authentication complete
== Info: Sending quote commands
== Info: Syntax error in SFTP command. Supply parameter(s)!
== Info: Connection #0 to host welcometotheroad.net  left intact
== Info: getaddrinfo(3) failed for OUTBOUND:80
== Info: Couldn't resolve host 'OUTBOUND'
== Info: Closing connection 1

::::::::::::::
del_test20190628180546.log
::::::::::::::
drwxr-x--- 2 ftpadmin ftpadmin        0 Jun 28 23:38 .
drwxr-x--- 2 ftpadmin ftpadmin        0 Apr 26 2019 ..
-rwxr-x--- 1 ftpadmin ftpadmin     7780 Jun 27 14:36 NextFile20190620172738.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2440 Jun 27 14:36 NextFile20190620172740.XML
-rwxr-x--- 1 ftpadmin ftpadmin     7980 Jun 27 14:36 NextFile20190624111531.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2441 Jun 27 14:36 NextFile20190624111533.XML
-rwxr-x--- 1 ftpadmin ftpadmin     7858 Jun 27 14:36 NextFile20190626165728.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2500 Jun 27 14:36 NextFile20190626165729.XML
-rwxr-x--- 1 ftpadmin ftpadmin    15424 Jun 27 14:36 NextFile20190627091426.XML
-rwxr-x--- 1 ftpadmin ftpadmin     4104 Jun 27 14:37 NextFile20190627091427.XML
-rwxr-x--- 1 ftpadmin ftpadmin    10417 Jun 27 14:37 NextFile20190627092525.XML
-rwxr-x--- 1 ftpadmin ftpadmin     3071 Jun 27 14:37 NextFile20190627092527.XML
-rwxr-x--- 1 ftpadmin ftpadmin    15594 Jun 27 14:37 NextFile20190627093345.XML
-rwxr-x--- 1 ftpadmin ftpadmin     4279 Jun 27 14:37 NextFile20190627093347.XML
-rwxr-x--- 1 ftpadmin ftpadmin    17237 Jun 27 14:37 NextFile20190627094528.XML
-rwxr-x--- 1 ftpadmin ftpadmin     4290 Jun 27 14:37 NextFile20190627094530.XML
-rwxr-x--- 1 ftpadmin ftpadmin    15608 Jun 27 14:37 NextFile20190627100222.XML
-rwxr-x--- 1 ftpadmin ftpadmin     4288 Jun 27 14:37 NextFile20190627100224.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2455 Jun 27 14:37 NextFile20190627102026.XML
-rwxr-x--- 1 ftpadmin ftpadmin     7858 Jun 27 15:18 NextFile20190627111647.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2500 Jun 27 15:18 NextFile20190627111648.XML
-rwxr-x--- 1 ftpadmin ftpadmin     7858 Jun 27 18:56 NextFile20190627145327.XML
-rwxr-x--- 1 ftpadmin ftpadmin     2500 Jun 27 18:56 NextFile20190627145328.XML
-rwxr-x--- 1 ftpadmin ftpadmin    12797 Jun 27 20:08 NextFile20190627160344.XML
-rwxr-x--- 1 ftpadmin ftpadmin     3609 Jun 27 20:08 NextFile20190627160345.XML

My question is "Why won't it work in the loop?"

Improve this question
3
  • First, which shell are you using? Is this bash? I would add a shebang and run this through shellcheck. Also, be sure to export your variables. Commented Jun 29, 2019 at 12:02
  • Why don't you quote your variables in the second curl command like you do in the first? Commented Jun 29, 2019 at 14:00
  • this message: == Info: Couldn't resolve host 'OUTBOUND', suggest that your THDeleteSftpAddress variable is expanding just to OUTBOUND instead of SFTP://welcometotheroad.net/OUTBOUND/... In addition to what was said in the others comments, check your variable names: you are mixing case all the time: maybe you have one or more names wrong. And, just as a piece of advice, I think that something like this, is kinda pushing the limits of what can be easily an cleanly done with a shell script: if it's an option, consider a python script, using the requests module. Commented Jun 29, 2019 at 20:51

1 Answer 1

Reset to default
1

Don't (try to) put quotes in variables.

Shell command parsing proceeds in a definite, fixed sequence, and variable substitution (formally parameter expansion) is done (well!) after commandline quoting is processed. When you do

 deleteString='"-rm '"$THFolderName"'/'"$targetFile"'"'
 ...
 curl ... -Q$deleteString ...

it puts the value "-rm OUTBOUND/openroad2019.xml" in the variable, and when that value is substituted it is split at whitespace (or in general IFS) and passes two arguments to curl (in addition to the others) as follows: 

 ...
 -Q"-rm
 OUTBOUND/openroad2019.xml"
 ...

The first of these confuses the remote server and produces the error

 Syntax error in SFTP command. Supply parameter(s)!

and the second is treated by curl as a second URL to access, causing it complain correctly that OUTBOUND is not a valid hostname.

Instead quote the space into the variable value, and double-quote the substitution to preserve that space, but don't put any kind of quotemarks in the value:

 deleteString="-rm $THFolderName/$targetfile" 
 ...
 curl ... -Q"$deleteString" ...
 # or if you prefer
 curl ... "-Q$deleteString" ...
 # (-Q are not special to shell so it doesn't matter if they're quoted)

 # or more simply drop the extra variable and just
 curl ... -Q"-rm $THFolderName/$targetfile" ...

and it didn't have anything to do with the for loop only with the variable.

FYI your complicated and clunky echo commands, which apparently tried to 'reconstruct' what the shell would do, didn't actually do that and probably wasted your time by misleading you. If you want to know how the shell is expanding and executing (a) command(s), set option x and it will tell you. For a specific command (or here two) do set -x before and set +x after. For scripts you can debug a whole script just by running with sh -x script args... without editting.

Improve this answer
1
  • @dave-thompson-085 you are fantastic, thank you sir. I am planning to go python with this in the future, but for now I needed to go this route because I don't know python, yet. I did just as you suggested and it worked beautifully. I didn't know about the set option x but i will look that up asap. Thank you so much for the time you spent to look this over and give me you input, suggestions and your solution. Commented Jul 1, 2019 at 16:38

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.