Obese-insulin resistance following chronic high-fat diet consumption led to cognitive decline through several mechanisms. Moreover, sex hormone deprivation, including estrogen and testosterone, could be a causative factor in inducing cognitive decline. However, comparative studies on the effects of hormone-deprivation on the brain are still lacking. Adult Wistar rats from both genders were conducted sham operations or orchiectomies/ovariectomies and given a normal diet or high-fat diet for 4, 8, and 12 weeks. Blood was collected to determine the metabolic parameters. At the end of the experiments, rats were decapitated and their brains were collected to determine brain mitochondrial function, brain oxidative stress, hippocampal plasticity, insulin-induced long-term depression, dendritic spine density, and cognition. We found that male and female rats fed a high-fat diet developed obese-insulin resistance by week 8 and brain defects via elevated brain oxidative stress, brain mitochondrial dysfunction, impaired insulin-induced long-term depression, hippocampal dysplasticity, reduced dendritic spine density, and cognitive decline by week 12. In normal diet-fed rats, estrogen-deprivation, not testosterone-deprivation, induced obese-insulin resistance, oxidative stress, brain mitochondrial dysfunction, impaired insulin-induced long-term depression, hippocampal dysplasticity, and reduced dendritic spine density. In high-fat-diet-fed rats, estrogen deprivation, not testosterone-deprivation, accelerated and aggravated obese-insulin resistance and brain defects at week 8. In conclusion, estrogen deprivation aggravates brain dysfunction more than testosterone deprivation through increased oxidative stress, brain mitochondrial dysfunction, impaired insulin-induced long-term depression, and dendritic spine reduction. These findings may explain clinical reports which show more severe cognitive decline in aging females than males with obese-insulin resistance.