unable to add PDF to context

[bschwand]

Hello,

I am running my own local llama-server instance with a multimodal model, verified working with its simple webUI (which allows uploading images and PDFs). However the simple webUI does ask to allow to convert PDF into images, and it does.

I have gptel 0.9.9.4 installed, as well as the emacs pdf-tools.

In emacs, when I add a pdf file to the context and query the llm, I get the following error :

Querying llama-cpp...
llama-cpp error: ((HTTP/1.1 500 Internal Server Error) server_error) Invalid url format: data:application/pdf;base64

llama-server logs the following:

srv operator(): got exception: {"error":{"code":500,"message":"Invalid url format: data:application/pdf;base64","type":"server_error"}}
srv log_server_r: done request: POST /v1/chat/completions 192.168.6.251 500

I read that gptel converts PDFs to images and sends them to the backend as base64 encoded, however it looks here like it is sending the whole unconverted file

and here is my gptel configuration:

;; Llama.cpp offers an OpenAI compatible API
;; configure my default gptel backend
(setq gptel-track-media 't)
(setq
 gptel-model   'local-llama
 gptel-backend (gptel-make-openai "llama-cpp"          ;Any name
                   :stream t                           ;Stream responses
                   :protocol "http"
                   :host "files:8080"     ;Llama.cpp server location
                   :models              ;Any names, doesn't matter for Llama
                    '((local-llama
                      :description "my own local llama-server instance"
                      :capabilities (media tool-use json)
                      :mime-types ("image/jpeg" "image/png" "image/gif" "image/webp" "application/pdf" "text/plain" "text/csv" "text/html")))))

I am not sure if this is a bug due to a different setup or a misconfiguraton on my part.
If I do not include "application/pdf" in the mime-types, adding a pdf to the context just results in an "Unsupported binary format" error from gptel/llama
If I do include the pdf mime type, I can add the file but obviously it does not get converted and makes an error.

What am I missing here ?
I am wondering if it would be related to this issue :
#929

Any help much appreciated !

[karthink]

IIUC the model can only parse images, not PDFs. WebUI is converting PDFs to images before feeding it to the model. gptel is not. So you'll have to write the step of converting PDFs to images yourself. I can help integrate it into gptel-send.

This does not seem related to #929.

[bschwand]

ah yes I guess I did not think much of the use case and was just wrapped learning about the conversion within Emacs :-)
Maybe the pdf can be opened in a temp buffer that is hidden and use it to do the conversion. I'll check what can be done and how.

[karthink]

Suppose pdf-to-images accepts a file path and returns a directory containing the images:

(pdf-to-images pdf-path) ; => "/tmp/some-name/" containing "some-name-%d.png" files

Then you can advise gptel-add-file to automatically add images to the context instead of the original PDF file when required:

(define-advice gptel-add-file (:filter-args (path) pdf->images)
  (if-let* ((gptel--model-capable-p 'media)
            (mime (mailcap-file-name-to-mime-type path))
            ;; Check if PATH is a pdf, and if the model supports PNG but not PDF
            ((and (equal mime "application/pdf")
                  (not (gptel--model-mime-capable-p mime))
                  (gptel--model-mime-capable-p "image/png"))))
      (pdf-to-images path)
    path))

You'll also need some way to delete the directory of images when you're done with the gptel sesssion. When you're done really depends on your use case, so I can't provide an airtight solution for that. You probably don't want to be deleting the image directory after every call to the LLM, in the likely event that you're sending it repeatedly in several back and forths with the LLM.

[bschwand]

sounds good, but yes I am not clear on handling the deletion of the temporary files. I'd think removing them as soon as they are submitted is cleaner, even if they end up being re-created later.

Alternatively, it seems just as logical to create those files in the platform-specific TEMP directory and let the system clean up as usual. On MacOS that is after 3 days, I suppose other systems do it after reboot or on some schedule.

Well this is what I have so far

(defun pdf-to-images-path (file-path)
  "Export selected PDF file pages as PNG images."
  (interactive "fSelect a PDF file: ")
  (let* ((pdf-buf (find-file-noselect file-path))
         (pdf-name (file-name-base file-path))
         (output-dir (make-temp-file (concat pdf-name "-images") t))
         (total-pages (pdf-info-number-of-pages pdf-buf)))
    (set-buffer pdf-buf)
    (cl-loop for page from 1 to total-pages do
             (let ((image (pdf-view-create-page page)))
               (with-temp-buffer
                 (insert (plist-get (cdr image) :data))
                 (write-region (point-min) (point-max)
                               (format "%s/page-%03d.png" output-dir page)))))
    (message "PDF pages exported as images in %s" output-dir)
    (kill-buffer pdf-buf)
    output-dir)
  )

which works on its own but there seems to be an issue with the parameter returned for the function you mentioned above as I get an Error
Wrong type argument : stringp ("some pdf file path")

[bschwand]

alright, I got it working. This is everything I had to add to my init.el to get be able to add pdf files directly to the context:

;; pfd-tools
(use-package pdf-tools
  :ensure t)
(require 'pdf-tools)
(pdf-tools-install :no-query)  ; Standard activation command
;(pdf-loader-install) ; On demand loading, leads to faster startup time
(require 'pdf-info)
(require 'pdf-util)

;; gptel
(use-package gptel
  :ensure t)
(require 'gptel)

;; Llama.cpp offers an OpenAI compatible API
;; configure my default gptel backend
(setq gptel-track-media 't) ; Ensure media mode is on
(setq
 gptel-model   'local-llama
 gptel-backend (gptel-make-openai "llama-cpp"          ;Any name
                   :stream t                           ;Stream responses
                   :protocol "http"
                   :host "llama-host:8080"     ;Llama.cpp server location
                   :models              ;Any names, doesn't matter for Llama
                    '((local-llama
                      :description "my own local llama-server instance"
                      :capabilities (media tool-use json url)
                      :mime-types ("image/jpeg" "image/png" "image/gif" "image/webp" "text/plain" "text/csv" "text/html")))))

;; "application/pdf"
;; llama does not support PDF files, only images. this does the conversion
(defun pdf-to-images-path (file-path)
  "Export selected PDF file pages as PNG images."
  (interactive "fSelect a PDF file: ")
  (let* ((pdf-buf (find-file-noselect file-path))
         (pdf-name (file-name-base file-path))
         (output-dir (make-temp-file (concat pdf-name "-images") t))
         (total-pages (pdf-info-number-of-pages pdf-buf)))
    (set-buffer pdf-buf)
    (cl-loop for page from 1 to total-pages do
             (let ((image (pdf-view-create-page page)))
               (with-temp-buffer
                 (insert (plist-get (cdr image) :data))
                 (write-region (point-min) (point-max)
                               (format "%s/page-%03d.png" output-dir page)))))
    (message "PDF pages exported as images in %s" output-dir)
    (kill-buffer pdf-buf)
    output-dir)
  )

;; tell gptel about our function above so it does the conversion of PDF into PNG and use them
(define-advice gptel-add-file (:filter-args (path) )
  (if-let* ((gptel--model-capable-p 'media)
            (mime (mailcap-file-name-to-mime-type (car path)))
            ;; Check if PATH is a pdf, and if the model supports PNG but not PDF
            ((and (equal mime "application/pdf")
                  (not (gptel--model-mime-capable-p mime))
                  (gptel--model-mime-capable-p "image/png"))))
      (list (pdf-to-images-path (car path)))
    path))

;; gptel agents
(use-package gptel-agent
  :ensure t)
(require 'gptel-agent)

Thanks for all the help !
Feel free to incorporate in gptel or in the documentation/FAQ/README as you see fit if you like.