Client Side Prediction Doesn't Work with Fluctuating Latency

Ive been working on a prototype for client side prediction in multiplayer games, Ive managed to make it work but for some reason the prediction breaks when the latency fluctuates.

For example if I have a constant 200ms latency, client side prediction works just fine, but it starts doing weird things when latency fluctuates, first I tought it may have to do with packages arriving out of order (packets have an ID) but discarding duplicates / out of order packets didnt really change anything.

The purple point represents the position received from the server and the yellow one the predicted position based on the one received + all non processed inputs.

Client Moving with constant 400ms latency:

Client Moving with only 200ms latency (but fluctuates +-25ms):

Code for the client movement (prediction, called every frame):

private void Predict(float delta)
{
    instruction_count++;
    byte instruction = PlayerMovement.ReadInput();
    network.SendClientMovementInstructions(instruction_count, instruction);

    instructions.Add(new Instruction{tick = instruction_count, data = instruction});
    
    Position += PlayerMovement.Movement(instruction, delta);
}

Code for the reconciliation (this is called when a package arrives from the server):

private void Reconciliate(int tick, Vector2 received_position)
{
    Vector2 prediction = received_position;

    foreach(var entry in instructions)
        if (entry.tick >= tick)
            prediction += PlayerMovement.Movement(entry.data, 1.0f / 60.0f);

    instructions.RemoveAll(i => (i.tick < tick));

    //i dont use it here but if this goes past certain threshold we snap back the client to were it should be.
    float deviation = (prediction - Position).Length();

}