Dirty hats or those that trap sweat and oil can lead to clogged pores, dandruff or fungal infections, explains Vaughn. These can cause scarring conditions on the scalp that can lead to hair loss.