Your question states that the solution from x -> y should be the same as the solution y -> x, i.e. we're only interested in defining the points on the path, not in any ordering of those points. If that's true, then simply find out which path has the smaller x (or y) and designate that as the origin.
origin = (3,3)
dest = (1,3)
origin, dest = sorted([origin, dest])
path = {(i,j) for i in range(origin[0], dest[0]+1) for j in range(origin[1], dest[1]+1)}
# note that this is now a set comprehension, since it doesn't make any sense
# to use a list of unique hashable items whose order is irrelevant
of course, this solves any obstructionless 2-D pathfinding. If you know that only one direction is changing, then only look in that direction.
origin, dest = sorted((origin, dest))
if origin[0] == dest[0]: # y is changing
path = {(origin[0], j) for j in range(origin[1], dest[1]+1)}
else: # x is changing
path = {(i, origin[1]) for i in range(origin[0], dest[0]+1)}