If I recall correctly, the distributor is held on with one bolt - the one that allows you to turn it to adjust the timing. I remove that one bolt and just pull it out as I recall. You'll want to make sure you look at the rotor and note where it's pointing so it can be re-installed in the correct position for ignition timing but other than that, it comes right out.
This is the bolt which needs to be removed
This picture shows the distributor in a 1951 six cylinder. I believe the four cylinder has this bolt on the bottom side of the distibutor as opposed to the top but it looks the same.