Future of Pyblish?

[munkybutt]

Hey - first off it is necessary to say thanks for pyblish, it is a great tool that I have gotten a lot of use out of professionally and personally!

I am wanting to contribute a bunch of changes/updates but I notice that this repo has slowed down a lot in the past year.
I know @mottosso you are super busy with ragdoll and I wonder if it would be worth considering passing the responsibility on to others so as to keep pyblish from dying, considering it has such far reaching use and relevance still?

[tokejepsen]

What kind of updates and changes do you have in mind?

I think there is little development happening because the project has fulfilled the needs atm.

[munkybutt]

That is a fair observation, it works well but improvements can be made ( as is evidenced by the number of outstanding PRs ).

Small changes:

Python is moving ( arguably has moved ) towards a more statically typed language with the introduction of type annotations and the various static analysis offerings in pretty much every text editor/IDE. Under the hood, python is still dynamic, but the benefits of type annotations and the static way of writing the code on large projects far outweigh the usefulness of dynamic types:

• Add type annotations for all the benefits of static analysis.

The giant folder of plugins is a nightmare:

• Add ability to organise plugins into sub-folders.

Py3 updates ( Py27 has been a dead language for almost 3 years now )

• Update to use pathlib
• There are a number of QoL updates, such as replacing depreciated function calls.

Large changes:

These larger changes could be introduced in a backwards compatible way OR a new major version of Pyblish can be released.

The Pyblish philosophy of little/no interdependence between plugins works for small scale projects, but once the projects grow in size and complexity the plugins become dependent on the data provided by previous plugins. This can become a nightmare! It can be improved by embracing the fact that it is not used as originally intended:

• Create a PluginRunner class that can contain plugin calls as methods on child classes.
• Create a PluginData class that allows a formalised structure to be defined for the expected data before any plugins actually run. This can help identify issues with missing attributes using the aforementioned static analysis and also provide much easier understanding and onboarding to anyone new to a codebase.

Due to the way Pyblish imports it's plugins, you cannot debug any Pyblish plugins directly. The current work around is to move the plugin code out of the plugin file into a normal package/module and call it in the plugin.py file. Testing something outside of it's intended context is not ideal. There are also extra tools being written to try and allow for debugging.

• This should be possible without moving code out of a plugin.py file or extra tooling being written.

Changing plugin order becomes a massive chore with a large number of plugin files:

• Provide an easier way to modify plugin order other than tweaking order values on the plugins themselves.

Defining what is validated is done differently, yet with a similar idea everywhere:

• Add native and consistent methods for defining what data is to be validated in the Pyblish core code base.

These are just issues I have come across in my time with Pyblish, I am sure there will be a lot more from others ( again look at the outstanding PRs )
I would potentially argue that others wanting to suggest changes look at the outstanding PRs and come to the same conclusion I have, that it will take ages to get a PR included if at all so probably not worth doing it.

[mottosso]

Heya @munkybutt, Pyblish may not be moving very fast, but this is a feature not a bug. At this stage, any change is a risk to anyone who's built their pipeline on top of it.

With this in mind, you are most welcome to submit changes and updates. Some of them may be incorporated quickly and at little risk - especially cosmetic ones and clear cases of bugs. Others can take longer, especially if they involve changes to how plug-ins are discovered and processed.

I know @mottosso you are super busy with ragdoll and I wonder if it would be worth considering passing the responsibility on to others so as to keep pyblish from dying, considering it has such far reaching use and relevance still?

Many of the projects already have varying maintainers - like @tokejepsen here for example - and it has worked well so far. If you find a particular project interesting - such as the Lite UI or the Nuke integration - then there's certainly room for responsibility to be had.

If you let us know what kind of responsibility you are interested in then I'm sure we could find a good fit!

Small Changes: Large Changes:

Was just about to post when you answered my question. Will return with more comments once I find another minute to reply. 😅

[munkybutt]

oh and to clarify - I am not asking to have the responsibility, especially the sole responsibility - but just asking if it is worth opening it up to a wider group so that PRs can be processed quicker.
Oh - and yeah I will start with my PRs today.

[tokejepsen]

Py3 updates ( Py27 has been a dead language for almost 3 years now )

Not sure its entirely safe yet to assume Py27 is dead for the consumers of Pyblish. VFX software industry has been especially slow at moving to Py3 so I think we can assume some pipelines are still reliant on Py27 support.

The Pyblish philosophy of little/no interdependence between plugins works for small scale projects, but once the projects grow in size and complexity the plugins become dependent on the data provided by previous plugins. This can become a nightmare!

I agree the simplistic nature of a numerical ordering of plugins can become very tedious on larger amounts of plugins. I do think however that pipelines can atm make their own class to inherit from, so I'm not sure what benefits we would gain from defining plugin classes in this project?
The cons would be less flexible Pyblish for people to use in their pipeline (?)

This should be possible without moving code out of a plugin.py file or extra tooling being written.

Could you maybe give an example of how you envisage this?
Currently I see that you could test the code within plugins, but it'll require a bit of boilerplate code to get started.

Provide an easier way to modify plugin order other than tweaking order values on the plugins themselves.

Very much agree on this. I have in the past experimented with "chains" of plugins, where you inherit the ordering between plugins to generate chains. There is probably a better way, but some kind of dependency graph would be great to incapsulate plugins that work together to produce an output.

Add native and consistent methods for defining what data is to be validated in the Pyblish core code base.

Could you elaborate on this?
I think the power of Pyblish atm is its flexibility so anyone can adapt it to their pipeline needs while still building on a solid foundation.

[munkybutt]

• fat fingers - actual post inc...

[munkybutt]

Thanks for the replies both, I appreciate the discussion.

Not sure its entirely safe yet to assume Py27 is dead for the consumers of Pyblish. VFX software industry has been especially slow at moving to Py3 so I think we can assume some pipelines are still reliant on Py27 support.

I understand there is still a lot of dependence of Py27 in VFX, however it is officially a dead language since 1 Jan 2020 and is considered a security risk. The python community as a whole needs to move forward ( and mostly has ). I would respectfully suggest that maybe the slowness of VFX to move on is in part due to tools like Pyblish continuing to support it over moving forward with Py3. Also - there are other industries that use Pyblish and we have moved on from Py27 ( Games 👋🏼 )

Idealistic discussions aside, my suggested changes would not be to drop support for Py27, but rather update the code base to be Py37+ that supports Py27 rather than Py27 that is roughly compatible with Py37+. Thus making Pyblish forward facing, rather than backwards looking.

I agree the simplistic nature of a numerical ordering of plugins can become very tedious on larger amounts of plugins. I do think however that pipelines can atm make their own class to inherit from, so I'm not sure what benefits we would gain from defining plugin classes in this project?
The cons would be less flexible Pyblish for people to use in their pipeline (?)

I am not sure how this would change flexibility? but rather provide a clearer structure that is consistent across projects/teams/companies as well as much easier to maintain on a more complex code base.
The implication that this should be on each individual team to implement kind of defeats the point of any OS tooling surely? It is also something that can be implemented whilst keeping the current method in place for those that prefer it.

Could you maybe give an example of how you envisage this?
Currently I see that you could test the code within plugins, but it'll require a bit of boilerplate code to get started.

Changing how plugins are imported to use the normal import mechanism pathlib.import_module - even if it is just for Py3. The move to providing structured plugin runner and data classes can also help

Very much agree on this. I have in the past experimented with "chains" of plugins, where you inherit the ordering between plugins to generate chains. There is probably a better way, but some kind of dependency graph would be great to incapsulate plugins that work together to produce an output.

Yeah that would be a good approach, historically I have solved this by allowing the GUI to edit the expected order per family with drag and drop and have the Pyblish backend manage the actual order value when plugins are discovered. Either way, some way of improving this would make working with Pyblish much more efficient.

Could you elaborate on this?
I think the power of Pyblish atm is its flexibility so anyone can adapt it to their pipeline needs while still building on a solid foundation.

I have worked with Pyblish on 3 different projects and each one of them defines the data to be validated in a similar yet different enough way to be incompatible between each other. Having a standardised method of this means plugins can be more easily shared and the method doesn't need to be re-implemented over and over again which is a big time saver and reduces the barriers to entry for new projects.
Ideally it will be written in an extendable way and there is also no reason to use the shipped method if another is preferred, as you said Pyblish is flexible and customisable. But these things I would say these come more from it's being open source.

[mottosso]

I would respectfully suggest that maybe the slowness of VFX to move on is in part due to tools like Pyblish continuing to support it over moving forward with Py3.

This made me chuckle. 😂 Pyblish has supported Python 3 since its first release in 2013, long before any DCCs had caught up.

[munkybutt]

This made me chuckle. 😂 Pyblish has supported Python 3 since its first release in 2013, long before any DCCs had caught up.

I didn't mean the disrespectfully 😄 but I do still have a point, however unintentionally preachy/condescending it came across as. Supporting Py3 is different from being written with Py3 in mind and my point is more that avoiding adding new Py3 focused changes because Py27 is still being supported is not ideal - currently opening Pyblish in Py37 Maya spams the console with depreciation warnings for example.

[mottosso]

Supporting Py3 is different from being written with Py3 in mind and my point is more that avoiding adding new Py3 focused changes because Py27 is still being supported is not ideal

You've lost me if what you are looking for is some political greater-cause of the world moving on from Python 2; that's not interesting here.

currently opening Pyblish in Py37 Maya spams the console with depreciation warnings for example.

This on the other hand is solvable and something suited for discussion and repair. How does that sound?

[munkybutt]

You've lost me if what you are looking for is some political greater-cause of the world moving on from Python 2; that's not interesting here.

That is not my motivation - using the tools that make my life easier is, and in the case of Py2 vs Py3, Py3 wins out which I think is understandable? Anyways I really don't want Py3 vs Py2 to derail my initial intent ☮️
All I wanted to do was try and gauge the direction of Pyblish before I started putting together larger PRs and having them rejected as they didn't fit the direction the project wanted to go in.

The console warnings are easily resolved with a six.PY2 check, importing inspect.getfullargspec for Py3 instead of inspect.getargspec for Py2, which I will add a PR for.

[mottosso]

Ok, great!

Add type annotations for all the benefits of static analysis.

I've seen type annotations in the form of comments that also work with Python 2. Those could be an option, even though I've personally not found any use for them. I'd leave it to the community to decide the usefulness of it.

Add ability to organise plugins into sub-folders.

This is already possible; you'd register any and all folders you have in mind.

register("folder1")
register("folder2")
etc()

Do you mean for the discover() mechanic to traverse folders within folders too? What benefit do you see here?

Update to use pathlib
There are a number of QoL updates, such as replacing depreciated function calls.

So long as Python 2.7 remains an option, there are no DCCs currently using 2.6 so that would be an option. People still use Maya 2018 and similarly old DCCs, so it's important to facilitate those until they are entirely left behind. Which in fairness isn't far from now, I expect 1-2 years from now Python 2 can safely be forgotten.

Create a PluginRunner class, Create a PluginData class

These and your other ideas sound interesting and are very welcome!

[munkybutt]

I've seen type annotations in the form of comments that also work with Python 2. Those could be an option, even though I've personally not found any use for them. I'd leave it to the community to decide the usefulness of it.

Oh man - type annotations are a game changer, especially when it comes to getting to grips with a new code base.
It is the difference between this (without):

And this (with):

Which then provides static analysis on your code in your IDE like a statically typed compiled language would:

As well as accurate auto-completion and allowing "goto" navigation with symbols and easier refactoring of variable names across a code base - and they will be Py2 compatible 😄
Once you start type annotating code, you wonder how you ever managed without it in the past as it takes such a massive mental load off.

Do you mean for the discover() mechanic to traverse folders within folders too? What benefit do you see here?

Yes but in register_plugin_path rather than discover - the benefit is this way of organising plugins is supported out of the box and doesn't need to be implemented by the end user. The current implementation, in my experience developing with Pyblish, is it encourages all plugins to be dumped in a single folder which is painful to work with past 15-20 plugin files. It could also be a way of namespacing plugin imports which could help with making plugins debuggable in a standard way (along with modifying the import mechanism - which is yet to be tested).

So long as Python 2.7 remains an option, there are no DCCs currently using 2.6 so that would be an option. People still use Maya 2018 and similarly old DCCs, so it's important to facilitate those until they are entirely left behind. Which in fairness isn't far from now, I expect 1-2 years from now Python 2 can safely be forgotten.

pathlib2 is a backport to 27 of pathlib, which can be vendored for backwards compatibility.
pathlib makes working with paths a dream compared to os.path

These and your other ideas sound interesting and are very welcome!

Great! - they are larger changes that will need more discussion once I get through the smaller PRs.

[mottosso]

Once you start type annotating code, you wonder how you ever managed without it in the past as it takes such a massive mental load off.

As a counterpoint, this is the same reason I don't use Gmail's auto-complete for sentences. It takes such a massive mental load off that I lose the ability to think. Like replacing your 25kg dumbbell with a 25g bucket of popcorn. 😅

That said, happy for it to be incorporated. Everyone should have a choice.

The current implementation, in my experience developing with Pyblish, is it encourages all plugins to be dumped in a single folder which is painful to work with past 15-20 plugin files

Mm, I can see and have seen this, it's a solid point. We'd need to maintain backwards compatibility, so it'd need to be an opt-in. E.g. Via en environment variable PYBLISH_RECURSIVE_DISCOVERY=1

[munkybutt]

That said, happy for it to be incorporated. Everyone should have a choice.

Yeah the nice part about the Py2 annotations are they are less intrusive in the code, so for the purists out there they basically don't exist unless you have all the features enabled in your IDE

Mm, I can see and have seen this, it's a solid point. We'd need to maintain backwards compatibility, so it'd need to be an opt-in. E.g. Via en environment variable PYBLISH_RECURSIVE_DISCOVERY=1

Makes sense and will incorporate it into my PR