I had this happen on my Henry J. Drove it for two years like that and decided I had to do something. I don't know what it's like to do a clutch on the Kaiser but doing it on the J is major surgery so when I got it all apart, I replaced everything - disc, pressure plate, throwout bearing and had the flywheel resurfaced. It's amazing now - works so smooth and shifts so nice, like a new car. As joe mentioned, it may go away for you and I hope it does. But if it doesn't and you're going to go through the effort of taking it down to get to at the clutch, you may as well do the whole job. You won't regret it.
