Milling aluminum??

[astroracer]

I used a speed controller on my router when I made the step plates for my truck. Did a few test cuts to dial in the rpms and it worked very well.
I am surprised your drill press didn't drop out the chuck due to vibration. A mill uses a collet that is held in with a long bolt through the spindle. Most drill presses rely only on the taper and pressure from drilling to keep the chuck in the collet. I think you dodged a bullet with this one...
Mark