Yeah, unfortunately it seems this can’t be converted to a llama.cpp compatible format yet, and that’s pretty big a tradeoff right now. Not surprising with how new it is, but we’ll have to wait to combine it with other improvements. Pretty exciting for the future though.
Update: I actually couldn’t get this to run even on HuggingFace Transformers. I made a bug report, but basically I’m getting some torch incompatibilities with flash-attn. Maybe this is a known issue for more experienced folks, but I couldn’t solve it.
They said they’re working on Orthus for Qwen 3.5. It’ll be amazing!
Yeah, unfortunately it seems this can’t be converted to a llama.cpp compatible format yet, and that’s pretty big a tradeoff right now. Not surprising with how new it is, but we’ll have to wait to combine it with other improvements. Pretty exciting for the future though.
Update: I actually couldn’t get this to run even on HuggingFace Transformers. I made a bug report, but basically I’m getting some torch incompatibilities with flash-attn. Maybe this is a known issue for more experienced folks, but I couldn’t solve it.