Skip first 3 bytes of a file

Instead of using cat you can use tail as such:

tail -c +4 FILE

This will print out the entire file except for the first 3 bytes. Consult man tail for more information.

If one has Python on their system, one can use small python script to take advantage of seek() function to start reading at the nth byte like so:

#!/usr/bin/env python3
import sys
with open(sys.argv[1],'rb') as fd:
    fd.seek(int(sys.argv[2]))
    for line in fd:
        print(line.decode().strip())

And usage would be like so:

$ ./skip_bytes.py input.txt 3

Note that byte count starts at 0 (thus first byte is actually index 0), thus by specifying 3 we're effectively positioning the reading to start at 3+1=4th byte

I needed to recently do something similar. I was helping with a field support issue and needed to let a technician see real time plots as they were making changes. The data is in a binary log that grows throughout the day. I have software that can parse and plot the data from logs, but it is currently not real time. What I did was capture the size of the log before I started processing the data, then went into a loop that would process the data and each pass create a new file with the bytes of the file that had not yet been processed.

#!/usr/bin/env bash

# I named this little script hackjob.sh
# The purpose of this is to process an input file and load the results into
# a database. The file is constantly being update, so this runs in a loop
# and every pass it creates a new temp file with bytes that have not yet been
# processed.  It runs about 15 seconds behind real time so it's
# pseudo real time.  This will eventually be replaced by a real time
# queue based version, but this does work and surprisingly well actually.

set -x

# Current data in YYYYMMDD fomat
DATE=`date +%Y%m%d`

INPUT_PATH=/path/to/my/data
IFILE1=${INPUT_PATH}/${DATE}_my_input_file.dat

OUTPUT_PATH=/tmp
OFILE1=${OUTPUT_PATH}/${DATE}_my_input_file.dat

# Capture the size of the original file
SIZE1=`ls -l ${IFILE1} | awk '{print $5}'`

# Copy the original file to /tmp
cp ${IFILE1} ${OFILE1}

while :
do
    sleep 5

    # process_my_data.py ${OFILE1}
    rm ${OFILE1}
    # Copy IFILE1 to OFILE1 minus skipping the amount of data already processed
    dd skip=${SIZE1} bs=1 if=${IFILE1} of=${OFILE1}
    # Update the size of the input file
    SIZE1=`ls -l ${IFILE1} | awk '{print $5}'`

    echo

    DATE=`date +%Y%m%d`

done