considering it now seems as if Peter is the first Observer

I don`t know. It`s a possibility and Windmark smiled a little seeing him but I don`t see how Peter starting the Observers would work. And if he did, why did they genuinely try to erase him from the timeline after Season 3?

My understanding is that the Observers always would have come to invade, unrelated to Walter`s actions at the beginning of the show.