Reverse lookup

[sinisaos]

Related to #378. Prototype for reverse foreign key lookup. Any suggestions and help are welcome. If I missed the point or it's not a good api design feel free to close PR.
Usage of reverse FK lookup:

from piccolo.table import Table
from piccolo.columns import Varchar, ForeignKey
from piccolo.columns.reference import LazyTableReference
from piccolo.columns.reverse_lookup import ReverseLookup

class Manager(Table):
    name = Varchar()
    artists = ReverseLookup(
        LazyTableReference("Artist", module_path=__name__), reverse_fk="manager"
    )

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

rows = Manager.select(
        Manager.name, Manager.artists() # or as list Manager.artists(Artist.name, as_list=True)
    ).run_sync()

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 90.07634% with 13 lines in your changes missing coverage. Please review.

Project coverage is 92.54%. Comparing base (db57d6e) to head (bda0815).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
piccolo/columns/reverse_lookup.py	89.47%	8 Missing ⚠️
piccolo/query/methods/select.py	90.19%	5 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
+ Coverage   88.42%   92.54%   +4.12%     
==========================================
  Files         117      118       +1     
  Lines        8653     8772     +119     
==========================================
+ Hits         7651     8118     +467     
+ Misses       1002      654     -348     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dantownsend]

Thanks for this! It looks very promising.

[sinisaos]

Thanks. I used a lot of code (with some modifications) from M2M fields. I tried to use the best names of variables and methods. The main obstacle is that the user would have to enter reverse_lookup_name. I will try to make that optional (if table has only one FK default index value for foreign_key_columns will be 0), so that the user has to specify reverse_lookup_name only if the table has multiple FK so ReverseLookup knows which column to use. If you have any ideas, any help is welcome.
UPDATE:
I removed the reverse_lookup_name and added a table parameter to the ReverseLookupSelect class. With that, Piccolo can identify which table to use for reverse lookup if the table has multiple FKs. I think that is a better option.

[powellnorma]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

Also I like how in peewee one can just set the backref parameter on the ForeignKeyField, see here
Could we have a similar API in piccolo?

[powellnorma]

It looks like we compare column names with table names here. They don't have to match. I think instead of

Manager.artists(table=Manager)

it would be better to be able to specify:

Manager.artists(column="manager")

Since the column that is used for the reverse lookup doesn't change(?), I think it would be best if we could also specify it in the ReverseLookup constructor.

[sinisaos]

@powellnorma Thanks for your interest and review. It's been a while since I wrote this code, but if I remeber it has to be a table because we refer to the foreign key of parent table later in the code. We need this function to find the foreign key column index if there are multiple foreign keys per single table.

[dantownsend]

Just wanted to note that there is interest in this functionality. Is there anything specific holding this back (from being merged)?

I haven't had chance to review it yet - but I agree that it's a feature we need.

[powellnorma]

I think instead of _meta.name it should be _meta.table._meta.tablename. Otherwise reverse_lookup_fk_table_name is a column name

[powellnorma]

I think we should use AS "{original_column_name}s". The table must give us that name. See m2m.py:

piccolo/piccolo/columns/m2m.py

Line 93 in 14012e5

) AS "{m2m_relationship_name}"

[sinisaos]

@dantownsend It would be great if you could find the time to review this. @powellnorma This code works and passes all tests. If you think that things should be changed, it would be best after Daniel's review. Thank you both.

[powellnorma]

Why do we need this JOIN here? Isn't the following enough?

        reverse_select = f"""
            "{reverse_lookup_table_name}"
            WHERE "{reverse_lookup_table_name}"."{reverse_lookup_fk}"
                = "{reverse_lookup_fk_table_name}"."{reverse_lookup_pk}"
        """

[powellnorma]

This code works and passes all tests

I haven't looked, but I guess the tests always use a ForeignKey named the same as the referenced Table. As e.g. here:

class Manager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

the sections I pointed lead to errors / wrong queries

[powellnorma]

When I query an M2M field of the referenced table, I get:

AttributeError: type object 'M2MSelect' has no attribute 'value_type'

[powellnorma]

I have fixed the parts that I reviewed on my branch reverse_lookup, see: https://github.com/powellnorma/piccolo/commits/reverse_lookup

However querying an M2M field (of the referenced table) seems to be rather slow at the moment. When I set self.serialisation_safe = False in the case of the above mentioned AttributeError, it seems that all m2m joining keys are being used as arguments to a second query. This is overall (in my case) slower than using a for loop in python (which is located in a transaction) instead of the ReverseLookup.
@dantownsend what could we do to make this (querying m2m of reverse-referenced table) faster?

[sinisaos]

But if this is not true, as e.g. here:

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

the sections I pointed lead to errors / wrong queries

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist. Shouldn't it be like this

class MainManager(Table):
    name = Varchar()

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

I want to try your branch. In my reverse lookup attempt I use LazyTableReference like this

class Manager(Table):
    name = Varchar()
    bands = ReverseLookup(
        LazyTableReference("Band", module_path=__name__),
    )


class Band(Table):
    name = Varchar()
    manager = ForeignKey(Manager)

# query for all managers bands
query = await Manager.select(
        Manager.name, Manager.bands(Band.name, as_list=True, table=Manager)
    )

Can you show me how to write a table schema and an example query because right now I get TypeError: __init__() missing 1 required positional argument: 'reverse_fk'. Thanks in advance

[powellnorma]

I'm sorry, but I don't understand. How can we refer to a table that doesn't exist.

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

Can you show me how to write a table schema and an example query because right now I get TypeError: init() missing 1 required positional argument: 'reverse_fk'.

Like this:

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

[sinisaos]

The example I gave was wrong, I corrected it. What I meant was a situation in which the column name (of the foreign key) has a different name than the table that it refers to

You are right, it doesn't work if the column is different from the table name. I tried to use foreign_key_columns_index so that if the table has multiple fk columns we know the index of those columns, but it obviously doesn't work well.

class MainManager(Table):
    name = Varchar()
    artists = ReverseLookup(LazyTableReference("Artist", module_path=__name__), "manager")

class Artist(Table):
    name = Varchar()
    manager = ForeignKey(MainManager)

# select:
await MainManager.select(MainManager.artists(Artist.name))

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

[powellnorma]

This also doesn't work. Raise ValueError: _name isn't defined - the Table Metaclass should set it. on line 148 name property.

Hm, for me that example works. I (already) made changes so that _name should (indeed) be set from the Table Metaclass here.
Have you made sure that you checked out my branch correctly? Maybe only parts of my changes got applied?

[sinisaos]

Sorry, that was my mistake. I tried everything from scratch because I probably messed something up and you are right, the tests from my PR pass with your code.
The only thing I've noticed is that if the column name is plural the tests fail, but singular name pass. I tried to strip column name if it ends with s and it works in Sqlite but not in Postgres.
I have to go now and will be gone for 2 days, but you should continue with this because your version is much better and easier to understand and it works. I'll be happy to delete my PR so you can make a new one. I'll try to help as much as I can.

[powellnorma]

@sinisaos I created a PR on your fork/branch. If you accept it, my changes should be included in this PR.

The only thing I've noticed is that if the column name is plural the tests fail

Do you have an example that demonstrates this problem? How can I reproduce this? Thank you

[sam2x]

Hello, thank you for this lib, I plan to use it for an important sized project, and this feature is mandatory to me as it brings many benefits.
However, I am still confused, there is no update since several years. Is this feature planned on short time ? what are current issues/status ? I can't find any references in the master branch, if you could help updating us or provide an alternative ? Imho I can't imagine orm modern lib without a populated backref.

OK, seems like a lot of people want this. Getting Piccolo v1 out has taken most of my attention. Will take another look at this once I've released Python 3.12 support. Bear with me, I know this PR has been around for a while 🙏

Python 3.13 has been supported now.

Best regards

[sinisaos]

@sam2x I don't know if that makes sense to you, but you can try using the unofficial branch (until everything is officially supported). The link contains an explanation and how to use it. Hope that helps.

[sam2x]

Hey, any updates on this feature ? I still use the PR, but would be great to have in the master piccolo orm. As far i can see the PR is still doing great job in terms of integration over all this year piccolo's updates.
@dantownsend any vision on this ?

Best

[sam2x]

@dantownsend Please it has been 3years now. Can you share your feedbacks on this or if uncomfortable the reasons this patch isn't yet applied in the master branch ? Thank you

[sinisaos]

@sam2x I'm glad you find reverse lookup useful. I've created a branch that I update regularly (to keep it in sync with the master branch). This branch is not official and I know is not perfect, but it does contain some important features that are not in the main branch.
The branch includes: reverse lookup, composite constraints, composite indexes, self-referencing M2M columns etc. You can install and use that with pip install git+https://github.com/sinisaos/piccolo@dev#egg=piccolo