What happens is this:
They get separated when they are really young (about 10-12 years old, I think). They keep in touch for sometime, but over a period, their correspondence stops. The guy is so much in love with her that even after a long time, he can't get over her. She, on the other hand, falls for another guy and forgets him.
Towards the end of the movie, it is clear that even years later, he still remembers and misses her. She on the other hand has forgotten him entirely such that when they cross at a rail-crossing, she sees her and is stunned, while she doesn't even notice or recognize him and simply walks away.

1st part: Tohno and Akari are probably just entering Middle school and are really close to each other. 1st Akari moves to another city and they start keeping in touch via letters. Then Tohno has to move to a really distant town and doing so causes their correspondence to drop. What's shown in the movie is the final train trip he takes before moving to that other town which is remotely connected.
2nd Part: He is in that town he had moved to and slowly gotten into habit of just writing messages to himself. Sumida (I think) is a girl in his school who likes him and want sto go out with him. However, she realises that he has some distant personality where he doesn't get close to anyone. She thinks its about someone or his ambitions (I don't really remember). There Tohno has not forgotten Akari and wants to move towards a good future.
3rd part: I think Tohno dates Mizuno for a year and then moves out and away from her as he just can't get over Akari and keeps thinking about her. So he does not return any of her messages. Akari on the other hand has lost track of Tohno and is getting married to some guy. Still, apparently she remembered him after finding a letter she was supposed to mail to him. But she moves on...

As for why they didn't look while passing is because both were rembering each other and the past.