[HLSL][DirectX] Validation error seen when compiling code using D3DCOLORtoUBYTE4

[spall]

Seeing validation error when compiling this code:

StructuredBuffer<float4> In0 : register(t0);

RWStructuredBuffer<uint4> Out0 : register(u1);


[numthreads(1,1,1)]
void main() {

Out0[0] = D3DCOLORtoUBYTE4(In0[0]);
Out0[1] = D3DCOLORtoUBYTE4(float4(0, 11.11, -50.5, 100));
}

https://godbolt.org/z/oEsfTrTMr

Function: main: error: Assignment of undefined values to UAV.
# | note: at 'call void @dx.op.rawBufferStore.i32(i32 140, %dx.types.Handle %1, i32 1, i32 0, i32 undef, i32 2833, i32 0, i32 25500, i8 15, i32 4)' in block 'entry' of function 'main'.
# | Validation failed.

[farzonl]

This isn't a bug in D3DCOLORtoUBYTE4. What is going on here is we are implcitly casting float4(0, 11.11, -50.5, 100) to a uint4 which is fine for all elements except for -50.5. negative float to uint is undefined behavior hence the poison which on dxil disassembly is an undef triggering the validator.

DXCs Behavior was to maintain the signedness hence why we see -12877 for DXC. I do beleive this is also incorrect behavior.

call void @dx.op.rawBufferStore.i32(i32 140, %dx.types.Handle %Out0_UAV_structbuf, i32 1, i32 0, i32 -12877, i32 2833, i32 0, i32 25500, i8 15, i32 4)

The current Behavior is

call void @dx.op.rawBufferStore.i32(i32 140, %dx.types.Handle %1, i32 1, i32 0, i32 poison, i32 2833, i32 0, i32 25500, i8 15, i32 4)

We can cause this same validation error without D3DCOLORtoUBYTE4 by doing:

StructuredBuffer<float4> In0 : register(t0);
RWStructuredBuffer<uint4> Out0 : register(u1);
[numthreads(1,1,1)]
void main() {
    Out0[1] = float4(0, 11.11, -50.5, 100);
}

https://godbolt.org/z/WvfdMszeq

But if we do it with DXC we get the error
error: Assignment of undefined values to UAV.DXC (trunk) #49237
https://godbolt.org/z/4oWb81Kv8

So it seems like the right behavior is that DXC should have trated the float4 passed to D3DCOLORtoUBYTE4 that was undefined as the whole vector beign undefined. and propgated up the assignment error.

[farzonl]

Other intreresting findings this also doesn't compile in DXC

RWStructuredBuffer<uint4> Out0 : register(u1);

uint4 blah(uint4 a) {
    return a;
}

[numthreads(1,1,1)]
void main() {
Out0[1] = blah(float4(0, 11.11, -50.5, 100));
}

I think the fix we need in clang is
error: Assignment of undefined values to UAV.

[llvm-beanz]

For the specific case of D3DCOLORtoUBYTE4 the bug is that Clang implemented this as unsigned integers when they should be signed. See DXC's implementation:

https://github.com/microsoft/DirectXShaderCompiler/blob/main/utils/hct/gen_intrin_main.txt#L88

The "UBYTE" in the name is misleading because it makes you think it is an unsigned output, but it isn't.