Get live data from HTTP server with active aggregation

[shikousa]

The HTTP server seems not to provide realtime readings, but only the aggregated values.

This delivers exactly the same, but should be different, as my aggregation time is 600s.
http://localhost/middleware.php/data.json?options=raw&from=now&uuid[]=1111-2222-3333
http://localhost:8081/1111-2222-3333

Is there a way to get the live data over HTTP? The push server is working, as I can see the live data in the middleware.

[r00t-]

the server serves data from localbuffer:
https://github.com/volkszaehler/vzlogger/blob/master/src/local.cpp#L53

which is populated with data from just after aggregation:

vzlogger/src/threads.cpp

Line 166 in df03cf6

add_ch_to_localbuffer(*(*ch)); // add this ch data to the local buffer

i'm not sure how much changes would be needed to serve non-aggregated data.
(besides obvious things like requiring a larger buffer.)

[shikousa]

During aggregation it seems that both Push Server and MQTT Data is updated.
Is MQTT really updated during aggregation, as there is another update after aggregation?

vzlogger/src/threads.cpp

Lines 136 to 147 in df03cf6

	// provide data to push data server:
	if (pushDataList) {
	const std::string uuid = (*ch)->uuid();
	pushDataList->add(uuid, rds[i].time_ms(), rds[i].value());
	print(log_finest, "added to uuid %s", "push", uuid.c_str());
	}
	#ifdef ENABLE_MQTT
	// update mqtt values as well:
	if (mqttClient) {
	mqttClient->publish((*ch), rds[i]);
	}
	#endif

Is it possible to add this code section also during aggregation?

vzlogger/src/threads.cpp

Lines 163 to 168 in df03cf6

	#ifdef LOCAL_SUPPORT
	if (options.local()) {
	shrink_localbuffer(); // remove old/outdated data in the local buffer
	add_ch_to_localbuffer(*(*ch)); // add this ch data to the local buffer
	}
	#endif

[r00t-]

Is it possible to add this code section also during aggregation?

why don't you just try it, worst case it will crash...

Is MQTT really updated during aggregation, as there is another update after aggregation?

i'm not so familiar with the MQTT integration, but i think it publishes BOTH the raw and aggregated values.

[r00t-]

@indus:
if the holidays have me bored enough to attempt implementing this, would you be available to test? [can you build a branch from source or are you using binaries?]

[indus]

@r00t- I used to use to compile from source on an arm SoC and switched to a docker container recently with a transition to amd64 hardware. I probably could test between the holidays between Christmas and New Year’s Day (“zwischen den Jahren”)

I guess it should be easy to let the Dockerfile check-out a branch.

[r00t-]

(i find it irritating to see a long listing of mostly boilerplate code in an obscure language that this project does not use posted here.)

[indus]

(i find it irritating to see a long listing of mostly boilerplate code in an obscure language that this project does not use posted here.)

sorry - i deleted it (my idea was that it may have helped others to build a workaround in a similar situation)

[r00t-]

sorry, i didn't mean you have to delete it.
but it's so overly specific to that language/library you used and contains so little code specific to vzlogger.
now if it was at least very short and concise, it could serve as an advertisement for using that language, but it's not even that.

the reference to the push option was very valid, as it provides a way to extract the un-aggregated data.
your providing of some code is also correct in the sense that there is no specific documentation of the "push" data format (as far as i checked).
(just that that was only three lines of the twenty lines of code you posted, and not specifically marked as such.)

actually, now that i read the code for "push" in vzlogger, it has a problem similar to mqtt, => #687
(which is also why it can (intentionally) bypass aggregation.)

[indus]

No problem - and I understand your objection. I think I found a compromise for my deleted post:

I've now switched to using a custom middleware and the push option, so there's no longer any need for this feature on my side.

Deno Code Example

import { Hono } from 'npm:hono';

const app = new Hono();

app.post("/", async (c) => {
    const body = await c.req.json();

    for (const channel of body.data) {
        if (channel.uuid === UUID) {
            // business logic
        }
    }

    // respond with a JSON or the push sever throws an error
    return c.json({}, 200);
})

Deno.serve({ port: SERVER_PORT }, app.fetch);

[r00t-]

i still think your example would be much more useful if it would show the format of the data, i.e. with a debug printf,
and it's surely excessive to echo back the whole request?

[indus]

it's surely excessive to echo back the whole request?

I'm really not good at PHP (nor C++), but I had a look at the official code, and it seems to me it is doing something very similar.

To my understanding it recreates the channel data here (with the difference that it may be transformed by some interpreter):
https://github.com/volkszaehler/volkszaehler.org/blob/9d83e1d6c6ae85f7dd4445b5f6e79586124872e3/lib/Server/MiddlewareAdapter.php#L186-L189

and sends it back here:
https://github.com/volkszaehler/volkszaehler.org/blob/9d83e1d6c6ae85f7dd4445b5f6e79586124872e3/lib/Server/HttpReceiver.php#L71

I don't know why the official code does so, but sending the whole body back comes close and shoudn't be more excessive. What I really don't understand is what the push server does with the information?

I think the relevant code is in this check here:

vzlogger/src/PushData.cpp

Lines 174 to 191 in ae0372d

	size_t PushDataServer::curl_custom_write_callback(void *ptr, size_t size, size_t nmemb,
	void *data) {
	size_t realsize = size * nmemb;
	CURLresponse *response = static_cast<CURLresponse *>(data);
	response->data = (char *)realloc(response->data, response->size + realsize + 1);
	if (response->data == NULL) { // out of memory!
	print(log_alert, "Cannot allocate memory data=%p response->size=%zu realsize=%zu", "push",
	data, response->size, realsize);
	exit(EXIT_FAILURE);
	}
	memcpy(&(response->data[response->size]), ptr, realsize);
	response->size += realsize;
	response->data[response->size] = 0;
	return realsize;
	}

But its purpose? 😶 I have no idea!