The sight base pins are tapered pins that only drive out one way. There is a small end that you need to hit to drive them out. They come out left to right (when the barrel is pointed away from you). They are very tight and need to be hit pretty hard with a large flat punch to start them. Then you use a 1/8th inch punch to drive them the rest of the way out.
Then with a wooden or plastic mallet the front sight base can be knocked free of the barrel.