How to avoid losing my escape characters when reading in variables from a file with bash

Question

I have a lab ntlm-extract.ntds file which has usernames and hashes in the format: domain\username:integer:hash:hash2

For example: somedomain.local\jcricket:5201:0020cfaecd41954fb9c9da8c61ccacd7:0020cfaecd41954fb9c9da8c61ccacd7

I'm comparing the hashes in the LINE[3]/hash2 column with hashes in the NTLM HIBP database, then I'd like to print out usernames that have matches, but the domain\username keeps losing the \ whatever I try, and I'm not sure if it's on the read or write that it loses it.

The script I have so far is:

#!/usr/bin/bash

while read line
do
    IFS=':' read -ra "LINE" <<< ${line}
    HASH=${LINE[3]}
    HASH=${HASH^^}
    printf "Checking for %s\n" $HASH
    found=(`grep "$HASH" "./pwned-passwords-ntlm-ordered-by-hash-v7.txt"`)
    if [ -n $found ]; then
        printf "Match on username %s\n" "${LINE[0]}"
    fi
done < "ntlm-extract.ntds"

Following the recommendations the final working script ended up being:

#!/usr/bin/bash

numoflines=(`wc -l ntlm-extract.ntds`)
numcomp=0
while IFS= read -r line; do
    IFS=: read -ra hashline <<< "${line}"
    passhash="${hashline[3]}"
    printf "Checking for %s\n" $passhash
    printf "Line %d of %d\n" $numcomp $numoflines
    numcomp=$((numcomp+1))
    found=''
    found=(`grep -m 1 -i "$passhash" "./pwned-passwords-ntlm-ordered-by-hash-v7.txt"`)
    wait 
    if [ -z "$found" ]; then
        continue
    else
        printf "found return value is %s\n" "$found"
        printf "%s\n" "${hashline[0]}" >> ./hibp-busted.txt
    fi
done < "ntlm-extract.ntds"

Not sure if the "wait" is necessary though

flerb
– flerb

2021-06-06 01:29:20 +00:00
Commented Jun 6, 2021 at 1:29 — flerb
– flerb, Commented Jun 6, 2021 at 1:29

steeldriver · Accepted Answer · 2021-06-05 23:47:16Z

3

You need -r on the outer read, not just on the inner read -a. You should also quote "${line}" and (probably) want IFS= unless you explicitly want to strip leading whitespace:

while IFS= read -r line; do 
  IFS=: read -ra LINE <<< "${line}"; printf '%s\n' "${LINE[@]}"
done < ntlm-extract.ntds
somedomain.local\jcricket
5201
0020cfaecd41954fb9c9da8c61ccacd7
0020cfaecd41954fb9c9da8c61ccacd7

I'd also suggest changing the name of variable LINE to something that is not all uppercase.

answered Jun 5, 2021 at 23:47

steeldriver

83.8k12 gold badges124 silver badges175 bronze badges

Hmm... now I'm second guessing myself about quoting of the herestring variable; When is double-quoting necessary? doesn't appear to mention this case but var='foo * bar'; read -ra arr <<< $var; printf '%s\n' "${arr[@]}" does not seem to expand the *

steeldriver
– steeldriver

2021-06-05 23:59:53 +00:00
Commented Jun 5, 2021 at 23:59

Add a comment |

Stack Exchange Network

How to avoid losing my escape characters when reading in variables from a file with bash

1 Answer 1

You must log in to answer this question.

Linked

Hot Network Questions

How to avoid losing my escape characters when reading in variables from a file with bash

1 Answer 1

You must log in to answer this question.

Linked

Related

Hot Network Questions