If it does this when you take your hands off the bars, then it's the way your front tyre is worn. I've had the same thing happen on past bikes of mine, and it was always because the front tyre was worn.
If it does this while you still have your hands on the bars, then you have other problems.