Hi, I try to chime in with some info to clear things up:
Threaded rod: TR8*8/4-start means: TR=standardized trapezoidal screw; the first 8 is the outer diameter of the rod; the second 8 is the lead, 8mm per revolution; 4 start means there are really 4 threads on the rod, each with 2mm offset (pitch) to the previous. 4-start helps with max load and smoothness, and is important to match rod and nut, but it does not influence the calculation of lead. Your $3tr should really be 8.
Step angle: According to the parts list, both the Nema17 and Nema23 motors are 200 steps per revolution (1.8 deg step angle). Set your $3sa=1.8 . Because your $3tr was also wrong, it evened out to the same distance (when tinyg tried to move the Z axis 8 mm, it calculated that it needs 0.5 motor revolutions = 200 steps with the old settings; with the new settings, it uses 1 motor revolution = 200 steps). But this does not explain the squirrelyness.
X/Y travel: 60mm per revolution is correct according to the parts list (20 tooth pulley * 3mm belt pitch).
I ran your gcode circle files, both of them run "fine". I noticed some things in the files:
1. You placed 4 tabs on the inner circle, but not on the outer circle. For the tabs, the machine has to decelerate x and y, lift Z up a bit, cut the tab, drop z down, accelerate x and y again.
2. The gcode assumes that Z=0 is at the top of the material (very common). That means that positive Z coordinates are in the air, negative ones are cutting in the material. If you set up the 0 coordinate at the top of the work bed (not the material), it would explain why the machine tries to cut into the workbed.
Velocity: The feeds in your files and parameters seem to be quite fast (in fact, I think 5000 mm/min feedrate would be too fast for all except *very* soft materials). I cannot see the 50mm/min and slow speed to move the gantry. Did you do this test with a different file or different settings?
Limit/homing switches are a good thing, but I don't think they would change anything about your problem.
Edit: Nearly forgot, another common issue with Z going to deep is when the Z axis actually goes too short when commanded to go up (caused by sticking/binding Z nut, over temperature of driver or motor, too fast max speed $zvm, slipping pulley/coupling, slack belt). I don't think this is your problem, but you can try to send the Z axis up and down manually with "G0 Z35" and "G0 Z0". If it does not stop at the same point every time, investigate further in this direction.
Btw, thank you for reminding me to download sketchUcam 1.2 :)