Simplify and generalize host control of machine

[diegonehab]

Context

Controlling a machine engaged with rollups involves reading and writing to a bunch of different machine CSRs. The introduction of the send_cmio_response simplified, but it is still complicated.

run now returns the break_reason, which simplifies it further. But now there are unnecessary redundancies that may cause confusion.

Possible solutions

At the moment, iflags has fields PRV and the X, Y, and H flags.

PRV really is something internal that the host should never mess with (the current machine privilege level).

Let's promote iflags.PRV to its own full CSR iprv. This will simplify the state-access implementations, since they won't have to do field manipulation there anymore.

X, Y, and H, on the other hand, are things the host needs to look at and, in the case of Y, change.

There is never a case in which more than one of these flags is set. They are always set via HTIF from the inside.

In the case of X and Y, the host will also need to look into htif.tohost. This is because htif.tohost contains the reason for the yield and the amount of data written to tx_buffer.

X is set when the machine returns from an automatic yield. Let's remove X altogether, since the machine run already returns this as a break reason and X is cleared automatically.

Y when it returns from a manual yield.

H when it is permanently halted.

Let's relocate Y and H to htif.tohost. With some reorganization there, we can make this happen.
(This will also simplify HTIF implementation, since it won't need to change the iflags register anymore.)
Let's rename Y to YM to make the distinction obvious. It's not a generic yield flag, but rather a Manual Yield flag.

Perhaps we can be smart and use the device+cmd fields together as "the flag", with a few changes to make them uniquely identify the halt and the manual yields.

There already are many WARL CSRs that prevent certain bits from being changed.
htif.tohost would be one of these. If H is set, it would remain set forever. I think we can even use a write to htif.fromhost to clear YM, saving the need to modify htif.tohost when returning from a manual yield.

[diegonehab]

After some thought, here is a possibility.

A machine is halted if tohost has dev=HTIF_DEV_HALT, cmd=HTIF_HALT_CMD_HALT, and (data & 1).
A machine is yielded manually if tohost has dev = HTIF_DEV_YIELD and cmd = HTIF_YIELD_CMD_MANUAL.

We change the part of the interpret loop that checks for fixed-point yield/halt to the following:

tohost = read_tohost();
if (halted(tohost)) { // dev=HTIF_DEV_HALT, cmd=HTIF_HALT_CMD_HALT, (data & 1) 
    return break_reason::halted;
}
if (yielded(tohost)) { // dev = HTIF_DEV_YIELD, cmd = HTIF_YIELD_CMD_MANUAL
    formhost = read_fromhost();
    if (!yielded(fromhost)) { // unless host wrote a response to this htif-yield command...
        return break_reason::yielded_manually;
    } 
    // here we know the host responded, so we clear tohost and the machine is not yielded anymore
    write_tohost(0);
}

We change the HTIF protocol to be as follows:

From the inside, to use HTIF, guest code writes dev+cmd+data to tohost. HTIF device itself then clears fromhost. If device is halt or yield, the run() returns. From the outside, host can check tohost to see what is up. To respond to a yield, host copies dev+cmd to fromhost, but changes the data as desired and resumes the machine. If device was yield and fromhost has the right combination of dev+cmd, the machine clears tohost. From the inside, guest code reads the response in fromhost.

We also change write_tohost() to guard against the removal of a halted combination of dev+cmd even from the outside.

[diegonehab]

Thinking ahead even more, when we have "multi-machines", there will be a situation in which one machine is servicing a GIO request from another machine. The concrete situation is this.

A machine wants to use some external sequencer. It really only wanted to receive inputs addressed to it on that sequencer, one after the other. Something specific like a GIO with (domain, id) = (sequencer domain, application address there). It would then be woken up with the next input sent to its address on that sequencer.

This is great for Sunodo. It would keep the machine sleeping, maybe even stored to disk, until some service it is running detects a new input for that application. It would then wake the machine up, feed it the input, and so on.

Unfortunately, from the point of view of Dave, this might be impossible or hard or too complicated to do. Can Dave, on L1, figure out what the new input is for that application on the sequencer? So here is what Dave would prefer.

The machine itself has to do the filtering. So it would do something generic a GIO (domain, id) = (sequencer next block, null). With the next block, it would do a bunch of GIO calls and "dehash" its way through the sequencer data-structures searching for its next input, if any.

The complicated logic would be inside the machine, so Dave can be a lot simpler.

The problem for sunodo is that it would have to wake the Cartesi Machine up for every sequencer block. Mostly, this would be a waste of time, because there would be new input for it. Even if there was an input, it would be a lot less efficient. First, the routing-by-dehashing is awkward at best, and is running inside the emulator. Second, if run outside, it would not only be faster, but could be done only once for all applications managed by Sunodo.

We could make both Dave and Sunodo happy if we had multi-machines.

Basically, we a main sub-machine, which would be doing the specific (sequencer domain, application address there) requests. It would have a domain sub-machine that would service requests from that domain. To do so, it would do the generic (sequencer next block, null) request followed by the "dehashing" crap. Let's call this "Dave's mode". When running the multi-machine this way, the only requests leaking out are the low-level ones.

Now let's deal with "Sunodo's mode". The key is that, once the domain sub-machine is done running, the only trace it leaves behind is the response to the generic call. What this means is that, if the host can trap the call made by the main machine, it can skip running the sub-machine and service the request itself. So the only requests leaking out are the high-level ones.

To make this work, we need a few things.

1. I think we need a single external cmio device, and one internal cmio device for each sub-machine. The emulator, knowing which sub-machine made a request, would be able to move the request data between the appropriate internal cmio device and the external one.
2. A sub-machine must be able to yield saying something like "I am done serving a request from this domain and I'm ready for the next"
3. Its state must be reverted afterwards
4. The emulator must be able to go back to whichever machine made the request, so probably needs a "sub-machine call stack".
5. The machine config should say which domains are served by each submachine
6. The maschine runtime config should say which submachines should be leaked out instead of being sent to sub-machines.

There are details missing, of course, but I think this would work.

[diegonehab]

Hey @stskeeps, we are thinking of following this idea (or some refinement of it) on the release after the next one. Do you have reservations? @GCdePaula? @vfusco @mpolitzer

[mpolitzer]

What you described seem to be the current behavior from the guest POV on yield[1].
https://github.com/cartesi/opensbi/blob/opensbi-1.3.1-ctsi-y/platform/cartesi/htif.c#L106

Is that so? would there be any required changes in: opensbi, linux or libcmt?

Seems close to the current workflow on the host side. I don't se an issue there.

[diegonehab]

I think we can do further than what I suggested above. Right now, we have DEV/CMD/(REASON/DATA). This seems to be very overkill.

We should perhaps have CMD(16)/DATA(48) and be done with it? With new CMD getting the current DEV/CMD/REASON. Or some other simpler division.

Also remove the LSB requirement from HALT, which is also overkill. If you issue a HALT, you are halted. :)

[GCdePaula]

I'm down for simplification, I don't mind updating our code. I'm trying to understand the changes.

---

Would this remove the possibility of querying "what was the latest break reason"? Like, it seems it's currently possible to look only at the current state of the machine and know what was the reason the machine broke last.

---

We should perhaps have CMD(16)/DATA(48) and be done with it? With new CMD getting the current DEV/CMD/REASON. Or some other simpler division.

What would be the possible values for CMD in this proposal? In the case of tohost, would it be representable (abstractly) by something like this?

type ToHost =
    | Automatic { reason: AutomaticReason }
    | Manual { reason: ManualReason }

type AutomaticReason =
    | Progress { mille_progress: u32 }
    | TxOutput { data_size: u48 }
    | TxReport { data_size: u48 }

type ManualReason {
    | RxAccepted { data_size: u48 } // will always be 32?
    | RxRejected
    | TxException { data_size: u48 }
    | GIO { domain: u16, data_size: u48 }

And the fromhost would be something like:

type FromHost
    | Advance  { data_size: u48 }
    | Inspect { data_size: u48 }

---

Why do we need both tohost and fromhost? Can't we use the same register?

[GCdePaula]

Also, do we need both halt and yield?

[diegonehab]

The break reason would still be visible in tohost/mcycle.
There is a chance we can simplify this even further.
The important part is the external view, not the internal.
The difference between HALT and YIELD and even PUTCHAR is still the same: it's a different command that you are sending to HTIF. You don't want to mix "I am ok, send me more input" with "There is nothing you can do, I am done." or "Output some character."