I don't fully follow what's going from your answer, but a couple of comments:
- You should use contiguous memory. At the very least, at the outer level (for the x-y coordinates) you should have a single vector, not a vector of vectors. A vector is basically a pointer to a contiguous block of memory. A vector of vectors is a contiguous block of pointers, to contiguous blocks of memory, i.e. the data itself is not contiguous. Contiguity is incredibly important to speed due to factors like cache missing and pre-fetching.
- Linked lists are slow. The working of modern processors tends to give vectors a very large practical advantage. Linked lists are especially terrible for small types, for several reasons. First off, if the types are small the extra space costs of the pointers is huge. Second, when the types are small and in contiguous storage, the processor can read them very efficiently because processors generally read from memory an entire cache line at a time, which is 64 bytes. If your data type is just a double, that means that 8 doubles are read simultaneously by the processor. When you use linked lists you give this up.
The bottom line: you want a loop over your data to read the data precisely sequentially from memory in one contiguous chunk. This is the model that will give the best performance.