Closed Loop Computer Vision Enabled CNC

Hello everyone!

I am very inexperienced in this field and I am attempting to develop a project. The basic premise is that I want to deposit material from a cnc machine fitted with an extruder on its end effector onto a free standing object. As the object is free standing, it may occasionally inadvertently move causing the predetermined toolpath (original g-code) to deposit material in the incorrect position. As a remedy, I am working to add cameras which can track the object being operated on and any movement will be tracked and it will automatically adjust the toolpath so that it does correctly print regardless of toolpath. I currently use mach4 to operate the machine and as such, I am struggling to determine how I can in real time, adjust the inputted g-code. I came across linuxcnc and QtPyVCP which from my own understanding, enables custom applications. Can anyone give me a sense of direction as to whether something like this can be implemented on this application? Thanks!

May be the 'external offsets' feature could be of interest. This will let you superimpose offsets to any axis through hal pins (ie outside of GCode) while the program is executed. LinuxCNC plasma applications use this to regulate the torch height on the sheet metal.
For a short demonstration:

This is very interesting. Would I need additional controllers or components to enable this external offset feature or could I run this directly from LinuxCNC? Has this type of feature been explored on multiple axes simultaneously? Thanks for the response!

I would look at opencv.. (I think it is still a thing) I goofed around with it a few years ago with linuxcnc.. I can see if I can find the python code...

No need for additional controllers but you would likely need to write a custom hal component to get your machine vision results into hal.

There is a simulation configuration 'configs/sim/axis/external_offsets/eoffsets' with external offsets for X,Y,Z axes (this will also run on a non-real time installation):

 

Regarding OpenCV there is also Dmitry's Pick and Place machine, uses OpenCV (python) for parts positioning and angular alignment.


his config files can be found here:
github.com/yur7aev/linuxpnp/tree/master/configs/pnp

Hello, I have been using opencv to perform just regular tracking but I have no idea how to implement the opencv tracking algorithms in a cnc machine. Aciera has given some awesome ideas for doing the closed-loop part, how does your implementation work?

Hello again, what is the difference between using hal and using Dmitry's pick and place machine? I have been briefly reviewing Dmitry's honestly incredible machine and it is frankly very complicated for me given my inexperience although I am hoping to obtain the necessary experience for my own project.

Might be something here worth looking at

Hello,

Thank you so much for all the help. I just wanted to confirm that applying external offsets through a custom HAL with the data obtained from a visual tracking algorithm doesn't require any additional hardware, i.e., external controllers. The only reasons I want to be 100% sure is that I currently run my cnc using mach4 and it has been made clear to me that mach4 would require an additional controller to perform what I am looking to achieve. I reached out to a motion controller company and the cost for customizing their motion controllers was very expensive. So, before I transition to LinuxCNC and overcoming the learning curve, I wanted to confirm with you if my project could all be achieved purely with software on linux? Thanks