Can't connect to oMLX inference server

[SpaceMonkeyMtn415]

Howdy,

Running oMLX (https://omlx.ai/), which is by far the best inference server I have seen so far, but I'm having problems connecting to it via the available smart chat/smart plugin config options for chat models. I have tried modifying ollama, lm_studio, and LiteLLM configs without success. It's pretty easy to access:

http://127.0.0.1:8000/v1
http://127.0.0.1:8000

plus an API key.... but I'm not having any luck. I will get a successful connection, but I don't get anything back from the server. I can curl it just fine.

thanks!

[mimetic]

Similar question: I want to use MLX embedding using a local LLM. There used to be a way to specify a local OPENAI standard URL, which would allow me to use mlx-serve-embeddings.

I now have tried oLMX, also, and if I try to use the Ollama connection for embedding, it doesn't get the list of models available (because, I guess, you're using Ollama specific endpoints).

Really, how can I use local MLX models with Smart Connections?

Please help. (Yes, I have PRO version)

[brianpetro]

The Pro plugins have the option to configure both Ollama and LM Studio for the embedding model. The Ollama integration also allows specifying a custom host.

[SpaceMonkeyMtn415]

I found the fix.

Configure Chat Model using the following:

• Create a new Chat Model using LiteLLM
• Name it "oMLX"
• Use the OpenAI URL to access oMLX (http://IP address:8000/v1)
• Add your API key if you are using one.
• If you hit "test", it will appear to have failed. (it will show success, but errors on the parameters)
• THERE WILL BE NO MODELS LOADING IN THE MODEL DROP DOWN ON FIRST TRY.
• Now, close the editing dialog and re-open, now you will see the models loading into the model drop down.

[brianpetro]

@SpaceMonkeyMtn415 thanks for following up about this.

And thanks for letting me know about the issue with the models not appearing at first. I'll see what I can do 🌴

[SpaceMonkeyMtn415]

One small fix also, the base URL field is too short which truncates the string... would suggest expanding a bit to avoid the visual truncation.

[mimetic]

I got it (for Chat)! The fix is to do what you say, above... but the LiteLLM base URL must be the "OpenAI API" version, NOT the general URL!
OpenAI API version --> http://127.0.0.1:1234/v1
General URL --> http://127.0.0.1:1234 (FAILS)

However, the embedding doesn't give access to LiteLLM — basically an OpenAI server — so we still cannot use MLX embedding via oMLX.

[mimetic]

Any chance you will allow us to use MLX embedding models? They are much faster!