1st part: Tohno and Akari are probably just entering Middle school and are really close to each other. 1st Akari moves to another city and they start keeping in touch via letters. Then Tohno has to move to a really distant town and doing so causes their correspondence to drop. What's shown in the movie is the final train trip he takes before moving to that other town which is remotely connected.
2nd Part: He is in that town he had moved to and slowly gotten into habit of just writing messages to himself. Sumida (I think) is a girl in his school who likes him and want sto go out with him. However, she realises that he has some distant personality where he doesn't get close to anyone. She thinks its about someone or his ambitions (I don't really remember). There Tohno has not forgotten Akari and wants to move towards a good future.
3rd part: I think Tohno dates Mizuno for a year and then moves out and away from her as he just can't get over Akari and keeps thinking about her. So he does not return any of her messages. Akari on the other hand has lost track of Tohno and is getting married to some guy. Still, apparently she remembered him after finding a letter she was supposed to mail to him. But she moves on...

As for why they didn't look while passing is because both were rembering each other and the past.