The poem
Since we agreed to let the road between us Fall to disuse, And bricked our gates up, planted trees to screen us, And turned all time's eroding agents loose, Silence, and space, and strangers - our neglect Has not had much effect.
Leaves drift unswept, perhaps; grass creeps unmown; No other change. So clear it stands, so little overgrown, Walking that way tonight would not seem strange, And still would be followed. A little longer, And time would be the stronger,
Drafting a world where no such road will run From you to me; To watch that world come up like a cold sun, Rewarding others, is my liberty. Not to prevent it is my will's fulfillment. Willing it, my ailment.

You miss the whole ******* point of the poem by misquoting it:
'Walking that way tonight would not seem strange,
And still would be followed.'
The word is ALLOWED not FOLLOWED!
Time will still allow him to go to her house but his will prevents him.
And again I quote you:
'A little longer,
And time would be the stronger,'
It's time WILL, not WOULD.
Future tense not past tense ffs!
Love and topography are both eroded by time and eventually sever all links.
This is a brilliant poem, too good to be misquoted.