One of the Problems with Time Travel

Seriously, how would you know that you’re not ending up like inside a wall or something?