PS: I also would do some shots in the dark, e.g. move or delete ~./cache. Add a new user and check if Xfce is ok for the new account. Did the issue start after an upgrade? If so, which packages where upgraded?