After prematurely claiming victory, I am reluctant to say that this is the final fix, but it is pretty stable.
There seems to be 3 things needed fixing in order for me to reach this point, all are related to power.
I needed enough Voltage, this is the least of my issues though. At just under 5V things seem to be less stable. It was better for my voltage to be a hair above 5V.
The next issue is Current, this was an issue, I think the regulator was fine, but I think the USB port did not provide enough current. When I used a wall wort, it was a little better.
I think the not so optimal voltage and current may have lead to the issue that I think is the root of all this, DC quality.
I did add some Tantalum caps and changed out my linear voltage regulator.
But I think the real deal was soldering a 10uF cap.
It has been stable, still, relying on USB to power this thing is not going to work. Results are pretty iffy.
Had the last meetup of the year. Pretty good group, I inadvertently said it was a presentation, it was more of a workshop.