So I think I may have figured out what the problem was. I remembered I had this old book on how to grow bonsai trees and thought it might have some useful information since how different can trees in pots be.
The symptoms for root rot fit pretty much exactly. Sudden loss of vigor and leaves falling off. As suggested by the book, I transplanted the few I had left into pots with a 50/50 mix of peat moss and pea gravel for freer draining soil.
Some lost their remaining leaves while I was transplanting so I'm not getting my hopes up, but maybe I can still have a handful to plant. It's too bad though, I had over a hundred at one point.