Reference: http://www.alfawzan.af.org.sa/ar/node/16869
Question: [What is your opinion regarding] those who go for prayer wearing dirty clothes?
Response: Their prayer is correct; [so long as] their clothing is clean [from all impurities] – even if it is dirty. However, it is best they go for prayer in a good state of preparedness and [clean] clothing. That is because [the prayer] is a great act of worship, and a great sight with the [collective] gathering of the muslims, so let them [honour it and] prepare themselves for it with clean clothing which suitably covers them.