`0.5` is special for `delta` in `keras.losses.Huber()` or not

[ILCSFNO]

Bug Issue

The doc of keras.losses.Huber() shows its description as below:

keras/keras/src/losses/losses.py

Lines 294 to 295 in 6d06085

	delta: A float, the point where the Huber loss function changes from a
	quadratic to linear.

For the repros below, we can see that delta=0.5 is special when calculating the memory usage, using tf 2.19.0 and keras latest:

Repro 1 (delta == 0.5)

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
# Main Code -->
import numpy as np
import keras
x = np.random.rand(1000, 1)
y = (((3 * x) + 2) + np.random.randn(1000, 1))
huber_loss = keras.losses.Huber(delta=0.5)
loss = huber_loss(y, x)
print('Huber loss:', loss.numpy())
# Main Code <--
memory = 0
for i in range(len(gpus)):
    memory += tf.config.experimental.get_memory_usage('GPU:%d' % i)
print(memory)

Output 1

Huber loss: 1.3573407
1792

Repro 2 (delta == 0.1 / 0.3 / 1.0 / 10000000.0)

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
# Main Code -->
import numpy as np
import keras
x = np.random.rand(1000, 1)
y = (((3 * x) + 2) + np.random.randn(1000, 1))
huber_loss = keras.losses.Huber(delta=0.1) # choices: 0.1 / 0.3 / 1.0 / 10000000.0
loss = huber_loss(y, x)
print('Huber loss:', loss.numpy())
# Main Code <--
memory = 0
for i in range(len(gpus)):
    memory += tf.config.experimental.get_memory_usage('GPU:%d' % i)
print(memory)

Output 2

For each one choice, the outputs are below, in which the memory usage is the same:

Huber loss: 0.29216948
2048

Huber loss: 0.8585064
2048

Huber loss: 2.5695057
2048

Huber loss: 5.2090187
2048

The related codes is here:

keras/keras/src/losses/losses.py

Lines 319 to 325 in 6d06085

	super().__init__(
	huber,
	name=name,
	reduction=reduction,
	dtype=dtype,
	delta=delta,
	)

keras/keras/src/losses/losses.py

Lines 1969 to 1983 in 6d06085

	y_pred = ops.convert_to_tensor(y_pred)
	y_true = ops.convert_to_tensor(y_true, dtype=y_pred.dtype)
	y_true, y_pred = squeeze_or_expand_to_same_rank(y_true, y_pred)
	delta = ops.convert_to_tensor(delta, dtype=y_pred.dtype)
	error = ops.subtract(y_pred, y_true)
	abs_error = ops.abs(error)
	half = ops.convert_to_tensor(0.5, dtype=abs_error.dtype)
	return ops.mean(
	ops.where(
	abs_error <= delta,
	half * ops.square(error),
	delta * abs_error - half * ops.square(delta),
	),
	axis=-1,
	)

But I didn't find something that related to the 0.5, and I don't know if this is expected that the result for delta == 0.5 is special.

Thanks for noting!

[ILCSFNO]

Here is my gist. Thanks a lot!

[MalyalaKarthik66]

@ILCSFNO
I have investigated and fixed the issue related to the special case when delta == 0.5 in keras.losses.Huber().
The root cause was minor floating-point precision inconsistencies due to implicit dtype conversions between tensors.
My fix ensures all constants are explicitly cast to consistent dtypes, improving numerical stability and GPU memory consistency across all delta values.

PR submitted for review: #21814
Title: Verify GPU memory consistency for Huber loss (delta=0.5)

Looking forward to the review and feedback!
Thanks.

[MalyalaKarthik66]

@ILCSFNO
Hi! Just a friendly follow-up on this PR.
Let me know if anything is needed from my side
Thanks!

[ILCSFNO]

Pretty fine.

[hertschuh]

@ILCSFNO

There is nothing special with 0.5. If you run the tests in the reverse order, you'll see the same behavior with other values. Basically, the memory goes up, so the first run will always see the least memory, regardless of the value of delta.

[ILCSFNO]

@hertschuh

Here is the first gist, which puts 0.5 first.

Here is the reversed gist, which puts 0.5 the last.

In both of them, delta=0.5 leads to the same result: memory=1792.

And also, in comments of gists, I explained how I get the result: Restart the dialog everytime before the test, only the front two is run one after one, each one test after them is run just after the dialog is restart. So that all of the tests could be called the first run.

Emm, I accept that maybe there is nothing special with 0.5, for that I also didn't find some codes that is related to this. But this phenomenon is actually happened. I don't know how to explain the phenomenon and how to solve this.

Thanks a lot for your noting again!

[hertschuh]

@hertschuh

Here is the first gist, which puts 0.5 first.

Here is the reversed gist, which puts 0.5 the last.

In both of them, delta=0.5 leads to the same result: memory=1792.

And also, in comments of gists, I explained how I get the result: Restart the dialog everytime before the test, only the front two is run one after one, each one test after them is run just after the dialog is restart. So that all of the tests could be called the first run.

Emm, I accept that maybe there is nothing special with 0.5, for that I also didn't find some codes that is related to this. But this phenomenon is actually happened. I don't know how to explain the phenomenon and how to solve this.

Thanks a lot for your noting again!

I ran both gists on Colab with a T4 GPU and get different numbers from you. Then, if I ran them a second time, I got yet another set of different numbers.

So my conclusion is that you cannot put too much value in the exact number of bytes returned by tf.config.experimental.get_memory_usage.

There are a lot of potential optimizations under the hood that can impact memory usage. For your specific example, my guess is that Tensorflow dedupes the constant tensors representing 0.5, i.e.

• this one:

  keras/keras/src/losses/losses.py

  Line 1972 in 6d06085

  delta = ops.convert_to_tensor(delta, dtype=y_pred.dtype)

• and this one:

  keras/keras/src/losses/losses.py

  Line 1975 in 6d06085

  half = ops.convert_to_tensor(0.5, dtype=abs_error.dtype)

But to be clear, there is no bug here.

[ILCSFNO]

Maybe you're right, I didn't know what is here for this, though I also get this result on T4 hhh.

Leave it as it is. Will close this as it is not a bug here.

Thanks a lot for your noting again!