Feature Request: Implement missing ops from backends

[am17an]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The list of supported operation in ggml per backend is maintained at https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md

How to contribute to make the list more complete:

1. Pick an operation which is not implemented in a backend (pick ones marked with ❌), pick only one per PR
2. Implement the operation
3. Test your implementation with test-backend-ops -o <op>
4. If it passes test for the op, create a PR.
5. Mention performance tests in the PR if they exist (via test-backend-ops perf -o <op>)

Notes:

1. The ops suffixed with _BACK are related to training which is WIP and does not require immediate attention
2. Do not link this issue in your PR, which will lead to closing this issue when your PR is merged. Instead, just mention it. Example PRs will be linked below
3. This list can be temporarily out of sync, you can check support by first running test-backend-ops -o <op> on master to confirm
4. Please do not ask to be assigned a particular task, make the PR first if you think it will be useful

Motivation

Feature parity between backends is always desirable and it's easy to test the implementation, making it an ideal good first issue.

Possible Implementation

No response

[jeemzz147]

I would like to contribute to this effort. May I ask whether it is okay to work on operators marked as [🟡 Partially supported] in the ops.md table?

[am17an]

I would like to contribute to this effort. May I ask whether it is okay to work on operators marked as [🟡 Partially supported] in the ops.md table?

Yes that would also work. The reason it's preferable to work on completely unsupported ops is because sometimes it's not possible to add full support to an op on a backend (e.g. some quantisation schemes in CUDA) and it might not be clear as a newcomer why that's the case. But if you make an operation support more complete, that would always be welcome!

[jeemzz147]

I've verified that PAD_REFLECT_1D is currently unsupported on CUDA.
I would like to work on the PAD_REFLECT_1D op for the CUDA. Please let me know if there are any considerations or edge cases I should keep in mind. Thanks!

[CISC]

I've verified that PAD_REFLECT_1D is currently unsupported on CUDA.

It's already in progress in #14659

[jeemzz147]

since PAD_REFLECT_1D is already in progress (#14659), I’d like to work on the SET op for the CUDA backend instead. Thanks!

[nimit25]

Can I pick up CONV_2D for CUDA please? thank you

Do I need to create a separate issue for that?