cargo: Allow filtering packages by name

[aleb]

Cargo.lock might contain unwanted packages.

[gasinvein]

Can you please explain when and why it could be useful?
Cargo.lock contains complete list of the project's dependencies, and omitting any of them is expected to result in a broken build. So, why someone would want that is definitely not obvious.

[aleb]

For example out of https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs we want only package "gst-plugin-gtk4", nothing else.

gst-plugins-rs $ ../flatpak-builder-tools/cargo/flatpak-cargo-generator.py Cargo.lock -o x.json
gst-plugins-rs $ ../flatpak-builder-tools/cargo/flatpak-cargo-generator.py Cargo.lock -p gst-plugin-gtk4 -o z.json
gst-plugins-rs $ ls -l *.json
-rw-r--r-- 1 aleb users 248779 12. Okt 21:43 x.json
-rw-r--r-- 1 aleb users  62082 12. Okt 22:57 z.json

[gasinvein]

I don't see a lockfile in the project you've linked, where does the Cargo.lock come from?

[aleb]

https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

If you’re building a non-end product, such as a rust library that other rust packages will depend on, put Cargo.lock in your .gitignore.

[aleb]

You can cargo fetch to create the Cargo.lock file when it's missing.

[gasinvein]

I still fail to understand what you're trying to achieve and what could be the reason for excluding cargo dependencies from a lockfile.

Just to be clear, to avoid possible misunderstandings: the flatpak-cargo-generator.py script doesn't build anything, neither cause anything to be installed. It only pre-fetches cargo packages cache (via flatpak-builder), very similar to what cargo vendor does (and largely replicating its behaviour). What to build and what not to is still up to Cargo.

[aleb]

For example out of https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs we want only package "gst-plugin-gtk4", nothing else.

gst-plugins-rs $ cargo build --package 2>&1 | grep gst | sort
    gst-plugin-audiofx
    gst-plugin-aws
    gst-plugin-cdg
    gst-plugin-claxon
    gst-plugin-closedcaption
    gst-plugin-csound
    gst-plugin-dav1d
    gst-plugin-fallbackswitch
    gst-plugin-ffv1
    gst-plugin-file
    gst-plugin-flavors
    gst-plugin-fmp4
    gst-plugin-gif
    gst-plugin-gtk4
    gst-plugin-hlssink3
    gst-plugin-hsv
    gst-plugin-json
    gst-plugin-lewton
    gst-plugin-onvif
    gst-plugin-raptorq
    gst-plugin-rav1e
    gst-plugin-regex
    gst-plugin-reqwest
    gst-plugin-rspng
    gst-plugin-rtpav1
    gst-plugin-sodium
    gst-plugin-spotify
    gst-plugin-textahead
    gst-plugin-textwrap
    gst-plugin-threadshare
    gst-plugin-togglerecord
    gst-plugin-tracers
    gst-plugin-tutorial
    gst-plugin-uriplaylistbin
    gst-plugin-version-helper
    gst-plugin-videofx
    gst-plugin-webp
    gst-plugin-webrtchttp

[gasinvein]

From a workspace project like this, you select package(s) to build via cargo. Again, flatpak-cargo-generator.py doesn't control what is being built in any way. You can generate sources for the whole gst-plugins-rs project, build still build only desired components.

[aleb]

Without the filtering we'd have to download a lot of stuff that's not needed, which is wasteful.

[gasinvein]

cargo vendor doesn't consider "redundant" packages and downloads all the project dependencies, why should we do otherwise?

[aleb]

Because we can! :)

cargo-vendor "Vendor all dependencies for a project locally". Why does it not allow to vendor only the deps of a single package? That I don't know.

As I said, gst-plugins-rs contains multiple libraries but we only need [to build] only one of them, not all. That's why we download only the dependencies for the corresponding package, not for all the packages in the project.

[gasinvein]

Suggested change

	for dep in packages_dependencies(deps, all_packages):
	yield dep
	yield from packages_dependencies(deps, all_packages)

[gasinvein]

Better do parsing earlier in the flow, ideally in main(), and pass packages here as a list of strings.

[gasinvein]

The all_packages dict feels out of place here, since it's only used within packages_dependencies() scope. Better pass the whole cargo_lock to packages_dependencies(), and there get whatever information is needed from it.

[gasinvein]

Nitpick: please make unrelated changes in separate commits.

[gasinvein]

This help text is a bit misleading, since we don't control what Cargo builds. Maybe "...list of package name to fetch" would be better?

[bbhtt]

Do you plan to get back to this? I don't really like the idea having an out of band filtering here.

gst-plugins-rs contains multiple libraries

The upstream recommended way is to use the individual crates from crates.io https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1765

The cargo.lock inside the crates has dependencies for that only, afair.

wget https://static.crates.io/crates/gst-plugin-audiofx/gst-plugin-audiofx-0.13.4.crate
tar xf gst-plugin-audiofx-0.13.4.crate
cd gst-plugin-audiofx-0.13.4
python flatpak-cargo-generator.py Cargo.lock