In our experience we distinguish between past and future in the sense that the content of our mental states is temporally directed (we call this the psychological arrow of time). For example, we distinguish between memories of past events and expectations for future events, even if the two events stripped off their temporal index are identical. If one accepts the prevalent idea in cognitive sciences and neuroscience that our mental states are determined by our brain states (and processes), then the question arises: what precisely in the physics of our brain is responsible for the psychological arrow of time? This question is reinforced by the fact that our best fundamental physical theories are temporally direction-less: they are invariant under the reversal of the direction of time. One of the mainstream answers to this question is that our psychological arrow is fixed by the temporal direction of the Second Law of thermodynamics, where this direction itself is determined by the so-called Past Hypothesis of low entropy of the universe. In this paper we argue that this explanation is wrong headed, because given contemporary physics, the Past Hypothesis is neither necessary nor sufficient for accounting for the psychological arrow of time. We propose, instead, two conditions that together are both necessary and sufficient: one is that the psychological arrow is fixed by the same brain states that determine our mental states; and the other is that these brain states exhibit an asymmetry which is itself non-temporal. We argue that it is the breaking of this non-temporal local in-brain symmetry that determines the psychological arrow, and that this asymmetry need not have anything to do with entropy or the Second Law of thermodynamics. Our proposal has radical implications: For example, it entails that the psychological arrow of time built into our brains is fundamental, while the direction of time in which entropy increases according to the Second Law of thermodynamics is fixed by it, rather than vice versa.