add functions to wallet API

[SNeedlewoods]

Edit: Should be ready to review now.

---

Do not merge, this is still in development and just PRd for easier discussion.

This is an attempt to make the wallet API "feature complete" and it's part of this proposal.

The approach was to search for every wallet2 function in simplewallet and wallet_rpc_server, if it is used in either of those, but is not implemented in the API I considered it missing and it was added in this PR.
I believe wallet2_api.h now contains all missing functions, but some are not implemented yet. I'm waiting on feedback on those, if you want to have a look, things I need help with are marked as QUESTION, you can also comment on things marked TODO (those may become QUESTIONs if I'm unable(/unsure how) to solve them) or anything else.

Also gathered this collection of (mostly minor) issues found during API work.

[rbrunner7]

As we can see from the table above, there are quite some things that wallet2 provides that are missing from current Wallet API. The additions will therefore be quite substantial.

I think we should not just rush headlong into adding those many things. After all, if we are successful, this API will be very important, will live for a long time, and will be used by many people. If the new resulting interface is not well designed, it will haunt us for years to come.

Problem is that there are too many ways how you could design it, it's not "naturally clear" what types and methods there have to be. My feeling is that you could do almost everything in a hundred different ways.

To contribute something useful in these quite difficult circumstances I try to come up with some general rules to follow when defining the API that will hopefully guide us towards a good solution.

[rbrunner7]

Make it complete

This rule is pretty self-evident and clear, but I repeat it here nevertheless in the sake of completeness.

Goal is to provide a viable successor to the "API" that the public things in wallet2.h provide and eventually be able to completely remove that header file from the codebase. Each and every user of it, e.g. the CLI wallet and the wallet RPC server, will have to switch and use the Wallet API exclusivelly.

Because of this, we have to make sure that the Wallet API is feature complete. We must not lose any wallet functionality that is needed and in actual use.

[rbrunner7]

Make it small

We should try, while respecting the first rule of completeness of course, to make the API small. The smaller it is, the faster will new devs be able to read into it, the easier it will be to find the right methods to use, the less work to document it, etc.

Where do we have potential to reduce the number of methods? One way I see is this:

Do not include methods from wallet2.h that would be trivial to implement by clients using other methods in the interface themselves with only a few lines of code. I would speak of essential methods, that being methods that do provide something that is not available through any other method and that clients can't build themselves because of this. So, include only such essential methods.

I have mentioned an example of such method in this earlier issue about wallet related methods: There is a variant of wallet2::get_payments that filters for payment ID. This method is not "essential" in the sense that I propose here because clients can easily use the general get_payments method and search themselves.

Of course use common sense: It may be essential that already the code of the Wallet API filters or sorts something because otherwise too much data must get transferred, or doing things in the client would be much, much slower.

[rbrunner7]

Be careful about terminology

A good API uses a consistent terminology when naming things, and uses terms that are "expressive", hard to misunderstand and hard to confuse.

IMHO quite a number of things in wallet2 have sub-optimal names, maybe because not enough importance was given to terminology when originally naming things. Prime example for me is the mixup of the terms transaction, payment and transfer. I detailed some examples in this already mentioned issue.

This motivated me to propose to ban the term transfer outright exactly because it's so easy to misunderstand and easy to confuse. I detailed my proposal here.

Of course many things in the Wallet API already have the names they have, and we must let them stand and are not completely free how to name the new things we add. Still I see quite a degree of freedom left.

[rbrunner7]

Limit the influence of existing clients

We have 2 main clients of wallet2.h in the existing codebase: The CLI wallet and the wallet RPC server. According to the plan we will have to migrate them to the Wallet API.

Of course, striclty only seen from the point of view of that migration, the more similar the Wallet API is to the "old" wallet2.h, the less work it will take. The question arises: It this important, and is this something that should guide us when designing the new API?

I propose "No" as answer. I think having a good API is much more important than trying to optimize towards migration work on the CLI wallet and the wallet RPC server being small. That migration happens once and is then over for all times, whereas the Wallet API will go indefinitely into the future and (hopefully) shape tons of Monero related software that together trump any migration work.

In detail that means we don't try to give each method of wallet2.h a more or less one-to-one counterpart in the parts we add to the Wallet API. We mostly design the API independently, following the other rules that I propose here, and then migrate.

[rbrunner7]

Keep the style consistent

I write this down more for the sake of completeness, like the very first rule Make it complete:

There is a clash in naming style between the Wallet API and most of the rest of the Monero codebase: The first uses casing to form names like TransactionInfo and getUnlockedBalance while the rest uses "snake casing", like so: device_derivation_path_option, unconfirmed_transfer_details.

For the things we add we basically have two choices: We keep the naming style consistent with the things that are already there in the Wallet API, or we use the snake casing that is much more widely used in the codebase. Both choices are well possible, and both have advantages as well as disadvantages. (Renaming existing things as a third option and a way to achieve overal naming style consistency is probably not on the table.)

We dicussed this in a Seraphis wallet workgroup meeting, and it seemed that more people voted for keeping the style consistent. I myself was also in that camp: To continue with the "non-standard" naming style is IMHO the lesser evil than mixing styles within a single API.

[rbrunner7]

Now my opinion about one the more important design questions:

wallet2.h has a number of separate methods to get various types of payments / transactions, e.g. get_payments (incoming confirmed payments), get_unconfirmed_payments (incoming unconfirmed payments), get_payments_out (outgoing confirmed transactions), and get_unconfirmed_payments_out (outgoing unconfirmed transactions).

As I explained here, neither the Wallet API, nor the RPC interface, nor Woodser's wallet API have such separate calls. They basically work with a single call that get payments / transactions of all types, using a struct that is "encompassing" i.e. is able to store all details of all these types. I hope I got that right, the Wallet API call for this seems to be history.

According to the rules Make it small and Limit the influence of existing clients I propose to continue with this one call and not create direct equivalents of these methods in the Wallet API.

Probably the type TransactionInfo needs some extensions to be able to give back every last bit of info that all those wallet2 methods provide, but I see this as a quite natural extension of the interface.

[rbrunner7]

Can you give some possible arguments for pushing this down to the account level (if I understand you correctly)? I don't see any right now, but you have probably give this much more thought than I :)

[SNeedlewoods]

Quick update:
There is no need to move createOneOffSubaddress to the account level. Since this is an exotic function, maybe we should add a note/warning that this function should only be used in special cases?
(moneromoo: "It was intended to help people who generated a subaddress far into the random index values.")
e.g. something like:

* brief: createOneOffSubaddress - Create a new account or subaddress for given index, without adding it to known accounts and subaddresses.
*                                 Use `SubaddressAccount::addRow()` or `Subaddress::addRow()` instead, to make sure the wallet adds accounts/subaddresses consecutively and keeps track of accounts and subaddresses.

WalletImpl::addSubaddress & WalletImpl::addSubaddressAccount also do not refresh the known accounts/subaddresses, if that's not for a good reason (that I don't see) I would either add refresh at the end of those functions, or at least give the same warning as proposed above.
Seems the GUI only uses the addRow functions (see here) not the addSubaddress* ones (see here).

[rbrunner7]

Here we probably enter a whole region of things that we won't need in this form if my proposal to make a single encompassing "get payments / transactions" methods really flies.

[rbrunner7]

Can go. We don't have the possibility to choose ringsize anymore for years, and well, soon we won't have any rings at all. Maximum that I see here is an info "ring size" in the new WalletState.

[tobtoht]

Would require something like this: tobtoht@894adf0. Can PR.

[rbrunner7]

I am not sure yet what possibilities of "pool info update" the Wallet API has to offer, or should offer, but I am pretty sure that this method in this form is on too low a level of abstraction, like the following processPoolState as well.

[SNeedlewoods]

This is too complex for me right now to fully dive into, maybe someone with more experience in this area can comment?

[jeffro256]

I try to follow the general principle of YAGNI, especially when making APIs. Don't add it unless someone asks for it. A use case for processing abstractions of the pool state might pop up in the future at some point, but it also might not. Until then, these functions are internal business logic and shouldn't be exposed to the API.

[SNeedlewoods]

If we don't add them to the API, how would the wallet-rpc & wallet-cli call this function, if the goal is to get rid of all the wallet2 function calls? Sorry if I'm missing something.

[rbrunner7]

I checked, and it's used indeed, so it's not a case of YAGNI like @jeffro256 suspected, and after I saw that @moneromooo-monero fixed an information leak a few years back when using remote daemons where somehow this method and the next for processing the pool state were involved, I think it's best to just add those to the interface and keep them "public".

[jeffro256]

I think that the function update_pool_state, with process_pool_state are both leaky abstractions as written. Both times they are used outside of wallet2, they are called like so:

std::vector<std::tuple<cryptonote::transaction, crypto::hash, bool>> process_txs;
m_wallet->update_pool_state(process_txs);
if (!process_txs.empty())
  m_wallet->process_pool_state(process_txs);

This sequence of statements basically means "rescan the pool" and IMO should be abstracted into a method as such. Calling update_pool_state without an immediate corresponding call to process_pool_state will almost always be a mistake, as this will update the fetching marker without actually scanning the transactions and getting up-to-date information about your balance. So I'd add a method to the Wallet interface called something like refreshPoolOnly which calls the above statements, and drop both updatePoolState and processPoolState from the interface.

[rbrunner7]

I wonder whether it would make sense to group all the multisig related methods together in the source, and not e.g. put non-multisig and multisig versions of methods that do similar things together. Thus, "write normal tx to file" and "write multisig transaction to file" would not be next to each other, but the first in the group of "normal" methods and the second in the multisig method group.

My argument: Many wallet apps will probably continue to not offer multisig, and their devs may welcome that all those multisig methods do not "clutter" the API for them.

There also have been thoughts that ideally, "multisig wallet" would be a completely separate wallet type, with its own API, and maybe even with the code in a separate repository. That multisig stuff is so highly special, so different, and so complicated ...

I also ask myself wether we can't make some of such methods "smart" anyway so they decide themselves what there is to do, based on internal knowledge whether a transaction is multisig or not.

[SNeedlewoods]

In regards to grouping multisig stuff together, that's already done in another PR (can be seen e.g. here or here). If I remember correctly for this PR I was asked to add everything new in one place, if the "organize functions" get merged this can get based onto that (actually there is still another one for in between in the pipeline I didn't PR yet, the "add comments" found here).

A separate multisig wallet really sounds ideal, not sure how many clients currently use the multisig functionality from the API, the GUI afaik doesn't. But that's out of scope for now I'd say!?

I'll look into if I can come up with such "smart" methods.

[rbrunner7]

Seems to me, if we "go with the pattern", we have a method somewhere that gives back all existing accounts with info, using an account_info struct. The tag would be part of that struct, and thus this call here not needed because not "essential".

Or maybe a call giving back all subaddresses, with info about them, and info how they group into accounts - and any tags of course.

Methods like the following setAccountTag stay of course even if we use this pattern whenever we can, because changing things is something fundamentally different and altogether impossible if there is no method, after all.

[SNeedlewoods]

I think that tags would fit into SubaddressAccountRow, but returning the "i-th account's tag" is just one part of this function, it also returns all tag descriptions (used by wallet-cli and wallet-rpc). So we probably need at least a getAccountTagDescriptions() method anyways, or alternatively we could store the tag description in SubaddressAccountRow, too.

[rbrunner7]

Went through the code, checked and made some comments. Looks pretty good already IMHO, could not detect any "true" code bugs.

[rbrunner7]

I wonder whether we should directly get the enotes from the wallet, like it happens in getEnoteDetails, and search through those, with:

m_wallet->get_transfers(tc);

It takes quite some effort to convert from "wallet2 style info" to "Wallet API style info" with getEnoteDetails, but most of the info is not needed anyway here to find by key image.

Or maybe not, with the argument "no premature optimization" ...

[SNeedlewoods]

This method is currently only used for the freeze/thaw/isFrozen(key_image) methods.
We can probably get rid of this and use wallet2::get_transfer_details(ki) instead, which is still inefficient to find an enote by key image, but at least we don't have to convert from wallet2 to Wallet API style.

For a more efficient approach we have freeze/thaw by pubkey, which uses this method to get an enote by pubkey:

monero/src/wallet/wallet2.cpp

Lines 2083 to 2088 in a04b70b

	size_t wallet2::get_output_index(const crypto::public_key &pk) const
	{
	auto search = m_pub_keys.find(pk);
	CHECK_AND_ASSERT_THROW_MES(search != m_pub_keys.end(), "Public key not found in owned outputs");
	return search->second;
	}

thanks to @tobtoht's recommendation

[rbrunner7]

Looking at the diff resulting from your last force-push yesterday "from a04b70b to f28a6be" I see a lot of files changed that seem to have nothing to do with your PR, and a lot of files given as deleted. Any idea what happened here?

(I was trying to see what changes resulted from my review comments, and saw them, but as mentioned a lot more changes, that's why I am confused ...)

[SNeedlewoods]

First I just looked at the commit links you provided:
First one says

15 files changed +1917 -69 lines changed

Second one says

15 files changed +1913 -69 lines changed

Which seems right to me.
But then I clicked on compare and saw

Showing 65 changed files with 228 additions and 34,414 deletions

Which is definitely messed up, no idea why though. Maybe happened on rebase. Will look into it later.

[iamamyth]

Your commit looks fine. The second view is the pertinent one (and, more generally, the first and second views are the right methodology to see the relevant diff). Comparing a pre-rebase commit to a rebased one will generate all the noise the rebase aims to eliminate, that's why you see a giant diff (for example, if you rebase onto a more recent version of the main branch, force push, and compare with your earlier commit, it'll show all the new development work on the main branch, now incorporated via rebase, in the comparison).

[rbrunner7]

After the reason for the large diff became clear, and after I found your changes that resulted from my review comments, I can approve this PR.

[SNeedlewoods]

The last two force pushes are supposed to be:

1. rebase onto master
2. the first round of working on vtnerds review from here

Seeing the ci failure and considering some other small things I noticed, there will be another force push soon(-ish).

I answered the review comments there, but I'd like to bring some extra attention to the following:

1. encrypting keys
   • unattended flag remove cached password from Wallet API (master) #10232 (comment)
   • encryptKeys() decryptKeys() remove cached password from Wallet API (master) #10232 (comment)
   • WalletKeysDecryptGuard remove cached password from Wallet API (master) #10232 (comment)
   • onGetPassword() callback remove cached password from Wallet API (master) #10232 (comment)
2. "manual" rescan blockchain
   • hashEnotes() remove cached password from Wallet API (master) #10232 (comment)
   • finishRescanBcKeepKeyImages() remove cached password from Wallet API (master) #10232 (comment)

---

Edit: unrelated to vtnerds review, I have one question and a note

• Not super important, but I just realized thatenum DeviceType in struct DeviceState is redundant, we already have enum Device in struct Wallet which does the same src. There is also this getter function, the name does imply what the function is doing, but IMO the return type and the comment could be misleading (implying the function returns some kind of Device object instead of an enum).
  There are many options, maybe someone can help make a decision here.
  • Q: Is it worth to replace old enum Device with new enum class DeviceType? Or which of the options a), b), c) or d) do you prefer?
    • a) replace enum Device with enum class DeviceType
      • pro:
        • makes struct DeviceState more complete
        • we can change enum to enum class
        • DeviceType is a more descriptive name compared to Device
      • con:
        • breaking change, from Device::Device_<"Software"|"Ledger"|"Tezor"> to DeviceState::DeviceType::DeviceType_<"Software"|"Ledger"|"Tezor">
        • raises question: What should we do with old method Device Wallet::getDeviceType() src now? If we allow breaking changes to clean things up then I would propose to remove it and let clients call m_wallet_impl->getDeviceState().device_type
    • b) keep both
      • pro:
        • doesn't break the API
      • con:
        • very messy, please not this
    • c) remove new enum class DeviceType and keep old enum Device
      • pro:
        • doesn't break the API
        • least changes in code
      • con:
        • less clean
    • d) enter your own suggestion here
• removed the NOTE + outcommented lines here, in regards to m_multisig_k and m_multisig_info in EnoteDetails, because AFAICT they're not needed

[iamamyth]

With respect to Wallet::Device, I see two issues:

1. How confusing is the existing enum's name? I don't like it, but I could live with it, mainly because namespacing provides disambiguation. I don't think enum class vs enum merits a breaking change, to me that distinction seems like noise (in other words: make it a class iff you change the name).

2. There's another function which uses Device: queryWalletDevice. Therefore, I would favor moving DeviceState::DeviceType to Wallet (effectively replacing Device with DeviceType), as I think it past and present usage suggest it's a Wallet-level type, yet somewhat poorly named. And, if you're breaking compatibility, you may as well drop getDeviceType. If there's good reason to keep compatibility here, that reasoning can inform how, or whether, to roll out a switch from Device to class DeviceType.

[SNeedlewoods]

Thanks for the feedback, I missed queryWalletDevice() (originated in this PR), AFAIK it's neither used by monero-wallet-[gui|cli|rpc], but it's used by xmrwallet/monerujo. With this in mind I'd prefer to keep old enum Device, getDeviceType() , queryWalletDevice() and just remove new enum class DeviceType, with the argument that this results in least code changes and no breaking changes. I also agree with your statement in regards to the name:

I don't like it, but I could live with it