Sure it’s much much cheaper than training, but importantly those companies are not recouping anything with interference because it is still more expensive than what they are selling it for.
They are double bankrupting themselves.
At work we run interference for a research project with an open weights model in the public cloud another part of my company provides and we pay around 25$ a day for a VM with a single L40s. It’s both slow - despite not even serving concurrent users - and kind of bad in its outputs.
No, it is cheap and efficient. It is relative, and the comparison is to model training. But yeah, its not free
Sure it’s much much cheaper than training, but importantly those companies are not recouping anything with interference because it is still more expensive than what they are selling it for.
They are double bankrupting themselves.
At work we run interference for a research project with an open weights model in the public cloud another part of my company provides and we pay around 25$ a day for a VM with a single L40s. It’s both slow - despite not even serving concurrent users - and kind of bad in its outputs.