Fix mixed use of `fmt.Errorf` and `xerrors.Errorf`

[mafredri]

We should use unified error handling in the project. For simplicity we should just use fmt.Errorf.

---

          nit: I see there is a mix of `fmt.Errorf` and `xerrors.Errorf`. Should we unify them?

Originally posted by @mtojek in #359 (comment)