Segmentation also of the Compound ID

[HiteSit]

Dear Development team,

Locking trought your code i've noticed that there is not an option to segment not only the structures but also the relevant ID that is often present in many patent (more or less with the same style, attached an example). I'm imagining a protocol that segment also the ID and than a simple OCR (pytesseract) or more complex OCR (maybe something based on DL) could recognise the number ID and associate it to the structure.
I'm aware of the fact that not in all the patent the ID is present in a constant position (for example sometimes is at 12ptx another times is at 6ptx from the recognised structures. Or sometimes is horizontally and centrated other times is not centrated). But again I can imagine some sort of sample script in which the user input some parameters until is not satisfied of the segmentation.

Before that I start to see if I can do it by myself there is a specific reason why such feature was not implmented and/or what could be the challenges.

Thanks much and terrific work

[Kohulan]

Hi @HiteSit ,

Thank you for your interest in our work! We're continually working to improve the project, though we do face some limitations.

We’ve already been considering this idea and are currently focusing on journal articles, rather than patents. However, if you have a solution in mind and would like to contribute, we’d be more than happy to review a pull request.

Kind regards,
Kohulan

[HiteSit]

Yep I do have.
I created a fork working on it.

[Kohulan]

Hi @HiteSit ,

Thank you that would be a great addition!

Kind regards,
Kohulan

[MKCarter]

Hi @HiteSit - Did you ever get the patent ID extraction working?

Thanks,
Mike

[HiteSit]

Hi @MKCarter
Well yeah more or less. My approach is heavily rule-based (the patents are too different from each other and the training set needed to do real automation is probably pure nightmare). So pretty much the user must select things like: "Tell me the page numbers in which there are the structures", "Tell me the page numbers in which there are the IC50".
Then I've manually trained a simple computer vision algorithm (I've used YOLO, maybe the same that you used for the Segmentation) that should be able to "grasp" the ID associated with the structure (see above) and connect it with the ID present in the IC50 table (converted using MistralAI).

Again, more or less it works, but it is required a lot of validation (for example using the DataAPI of Reaxys or SciHub).
If you are interested we can discuss further (riccardo.fusco@upol.cz).

[Kohulan]

Hi @HiteSit ,

Do you have any numbers on how fast it is? Do you think you could contribute that to the project?

Kind regards,
Kohulan

[HiteSit]

So @Kohulan ,
to be fast is fast (not enough general thought). Now I don't have a number on top of my mind but I can give you an idea.

Imagine a Patent Page in which there are -say- 6 structures. The code takes the time of Decimer Segmentation + Decimer Converter (or whatever, the one that convert the structure to smiles) + an additional inference on the computer vision algorithm (that is as fast as Decimer Segmentation -is needed for extract the ID, see above-) + -here it comes the variable part- the time needed by MistraAI to convert the "picture" of a table
"""csv
Structure_ID, Activity
145,0.45
"""
into markdown. Then on top of it there is a bit of Bayesian Algorithm (say 20% on the Decimer (total) Segmentation Time).

Regarding the contribution that's something that we can talk about. I have anyway the intention to publish it, but again the most challenging part is the validation.

[Kohulan]

Hi @HiteSit ,

Thanks a lot for the detailed breakdown, which helps clarify things.

Just a quick note: DECIMER is an acronym, not a standalone name. The specific model you're referring to for converting images to SMILES is called the DECIMER Image Transformer.

As you mentioned, the overall speed will largely depend on how efficiently the fine-tuned vision model handles the various components.

If you’re considering contributing in the future, you'd be more than welcome, and collaboration is always appreciated!

Kind regards,
Kohulan

[jelymewang]

Dear Development team,

Locking trought your code i've noticed that there is not an option to segment not only the structures but also the relevant ID that is often present in many patent (more or less with the same style, attached an example). I'm imagining a protocol that segment also the ID and than a simple OCR (pytesseract) or more complex OCR (maybe something based on DL) could recognise the number ID and associate it to the structure. I'm aware of the fact that not in all the patent the ID is present in a constant position (for example sometimes is at 12ptx another times is at 6ptx from the recognised structures. Or sometimes is horizontally and centrated other times is not centrated). But again I can imagine some sort of sample script in which the user input some parameters until is not satisfied of the segmentation.

Before that I start to see if I can do it by myself there is a specific reason why such feature was not implmented and/or what could be the challenges.

Thanks much and terrific work

vision llm can do it

[Kohulan]

@jelymewang ,

If you have a solution, you could make a pull request.

[jelymewang]

           highlight_img = original_page.copy() #pdf_page
            
           
            if mask.shape[:2] != original_page.shape[:2]:
                
                mask_resized = cv2.resize(mask, (original_page.shape[1], original_page.shape[0]))  #segment_mask
            else:
                mask_resized = mask
            
        
            if len(mask_resized.shape) == 2:
                
                highlight_img[mask_resized > 0] = [0, 0, 255
            else:
       
                highlight_img[mask_resized[:, :, 0] > 0] = [0, 0, 255]  

Then, feed this to the vision large model and have it return the serial number. This can be deployed as a streamlined process that use llm api.
I know too little about your project to make any modifications. sorry

[Kohulan]

Hi @jelymewang ,

Thank you. The main issue in this case will be that the Vision LLM could be extremely slow, depending on which model works better. I will look into this.

[jelymewang]

Hi @jelymewang ,

Thank you. The main issue in this case will be that the Vision LLM could be extremely slow, depending on which model works better. I will look into this.

You can submit in batches. For example, you can batch upload the segmented highlights from 5 pages to the llm.