Grundig Sonoclock 890A-Web fails reading the station list

[rorso]

I successfully convinced the above radio to communicate with YCast. It walks through the menus and directories that are fetched from "radio-browser.info" and it displays the configured stations.yaml.

But on every level that lists the actual station, it only displays a "communication error, please retry". It's the same when tryng to list the configured favorites.

I verified the communication with wireshark and the radio gets a station listing and confirms the recipt. It obviously does not like the data.

Cross checking with the original data captured from a communication with "grundig.radiosetup.com" (which IS vTuner), I see some content differences:

Original:

GET /setupapp/grundig/asp/browseXML/navXML.asp?gofile=LocationLevelFive-Europe-Germany-AllStations&bkLvl=9237&startItems=1&endItems=100&mac=7e1b12345678a47e5c907f5e6a9164c2&dlang=ger&fver=1&ven=grn6 HTTP/1.1

<?xml version="1.0" encoding="iso-8859-1" standalone="yes" ?>
<ListOfItems>
	<ItemCount>7429</ItemCount>
	<Item>
		<ItemType>Previous</ItemType>
		<UrlPrevious>http://grundig.vtuner.com/setupapp/grundig/asp/browseXML/navXML.asp?gofile=LocationLevelFour-Europe-Germany&amp;bkLvl=9237</UrlPrevious>
		<UrlPreviousBackUp>http://grundig2.vtuner.com/setupapp/grundig/asp/browseXML/navXML.asp?gofile=LocationLevelFour-Europe-Germany&amp;bkLvl=9237</UrlPreviousBackUp>
	</Item>
	<Item>
		<ItemType>Station</ItemType>
		<StationId>81163</StationId>
		<StationName>!!! 0nline-disco !!!</StationName>
		<StationUrl>http://grundig.vtuner.com/setupapp/grundig/asp/func/dynamODFS.asp?ex45v=7e1b858b9671a47e5c907f5e6a9164c2&amp;id=81163</StationUrl>
		<Bookmark>http://grundig.vtuner.com/setupapp/grundig/asp/browseXML/AddFav.asp?empty=&amp;stationid=81163</Bookmark>
	</Item>
        <!-- more items -->
</ListOfItems>

YCast version:

GET /ycast/radiobrowser/country/Austria?vtuner=true&startItems=1&endItems=100&mac=c123456789e56dad9aa7d69a827c1d6a&dlang=ger&fver=1&ven=grn6 HTTP/1.1

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<ListOfItems>
	<ItemCount>320</ItemCount>
	<Item>
		<ItemType>Station</ItemType>
		<StationId>RB_37cd2212-ac24-4002-802d-f6c05c488708</StationId>
		<StationName>  _01 SALZBURG FM</StationName>
		<StationUrl>http://canli.arabeskinmerkezi.com/9180/stream</StationUrl>
		<StationDesc>arabesk, arabic music, pop, pop music</StationDesc>
		<Logo>http://grundig.radiosetup.com/ycast/icon?id=RB_37cd2212-ac24-4002-802d-f6c05c488708</Logo>
		<StationFormat>arabesk</StationFormat>
		<StationLocation>AT</StationLocation>
		<StationBandWidth>128</StationBandWidth>
		<StationMime>AAC+</StationMime>
		<Relia>3</Relia>
		<Bookmark />
	</Item>
        <!-- more items -->
</ListOfItems>

While it should not harm to send more information, it seems a little odd to send the StationId as a string instead of the original integer. Due to the prefix it disqualifies as a GUID.

Is it granted that all(?) vTuner clients accept a station id as 39-character string? No problem report so far on this level? I did not find any reports in the past issues. But I don't see any "Grundig" devices either.

The different XML encoding "should" pose no problem as long as the client is sufficiently XML compliant.

Maybe the added parameters &fver=1&ven=grn6 have a special meaning in preparing the output for that device.

But then ... the StationId does not seem to carry valuable info in this collection anyway. It IS contained in the Bookmark URL and will get the right stream when queried from a saved entry. At least my device has no option to search by StationId, so I think it is dumped anyway.

[rorso]

And it turns out to be "complicated"...

The good news: It's not the ID.
The bad news: It's everything else

This device breaks on the station list if:

• the Bookmark tag is empty
• there is any additional tag in the station list besides ItemType, StationId, StationUrl and Bookmark

Even when stripped down, it fails playing the station, because it does not call StationUrl by itself but instead issues a search on the submitted ID to get more details. It's the same when playing "the last tuned station" on startup:

/setupapp/grundig/asp/BrowseXML/Search.asp?sSearchtype=3&Search=RB_f91aa755-2979-451e-a4fe-1393f&mac=c497f96ed9e56dad9aa7d69a827c1d6a&dlang=eng&fver=1&ven=grn6

And it expects a return of this kind with all the additional values that need to be missing in the first list:

GET /setupapp/grundig/asp/BrowseXML/Search.asp?sSearchtype=3&Search=5952&mac=7e1234567871a47e5c907f5e6a9164c2&dlang=eng&fver=1&ven=grn6

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<ListOfItems>
	<ItemCount>1</ItemCount>
	<Item>
		<ItemType>Previous</ItemType>
		<UrlPrevious>http://grundig.vtuner.com/setupapp/grundig/asp/browseXML/loginXML.asp?gofile=</UrlPrevious>
		<UrlPreviousBackUp>http://grundig2.vtuner.com/setupapp/grundig/asp/browseXML/loginXML.asp?gofile=</UrlPreviousBackUp>
	</Item>
	<Item>
		<ItemType>Station</ItemType>
		<StationId>5952</StationId>
		<StationName>Radio Swiss Classic</StationName>
		<StationUrl>http://grundig.vtuner.com/setupapp/grundig/asp/func/dynamODFS.asp?ex45v=&amp;id=5952</StationUrl>
		<StationDesc>Das Klassikradio zum Entspannen</StationDesc>
		<StationFormat>Classical</StationFormat>
		<StationLocation>Basel Switzerland</StationLocation>
		<StationBandWidth>128</StationBandWidth>
		<StationMime>MP3</StationMime>
		<StationProto>HTTP</StationProto>
		<Relia>5</Relia>
		<Bookmark>http://grundig.vtuner.com/setupapp/grundig/asp/browseXML/AddFav.asp?empty=&amp;stationid=5952</Bookmark>
	</Item>
</ListOfItems>

Since this "search" is not implemented yet, it fails with a 404 and results in "network error, please retry".

As soon as this reply is there, it issues ANOTHER call get the final channel URL within a special Content-Type. It seems weird, that this call is NOT protected by any token, so one could try to iterate through the station numbers.

I have yet to try, if this dynamic URL, that is submitted by the previous call can be exchanged against the true station URL, successfully skipping the last step:

GET /setupapp/grundig/asp/func/dynamODFS.asp?ex45v=&id=5952

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: audio/x-mpegurl
Server: Microsoft-IIS/10.0
Set-Cookie: ASPSESSIONIDCAQABSSS=FIMGIPPCMLPKDPAPBLOKEPHH; path=/
X-Powered-By: ASP.NET
Date: Fri, 29 Dec 2023 10:08:37 GMT
Connection: close
Content-Length: 41

http://stream.srg-ssr.ch/m/rsc_de/mp3_128

Given this, it does not wonder that encoded characters within the station name are not decoded for the display either. So a station

<StationName>Retro FM (M &#233;rida) - 103.1 FM - XHPYM-FM - Cadena RASA - M &#233;rida, Yucat &#225;n</StationName>

displays as

Retro FM (M érida) - 103.1 FM - XHPYM-FM - Cadena RASA - M érida, Yucat án

Since the reply is marked as UTF-8, the character encoding would be necessary for just " <, &, >" and might be superfluous by XML definition.

I guess this would need a big addition that is likely to break things for all others. :-(

Btw. What was the idea about prefixing the UID with additional characters? Aren't GUIDs meant to be "globally unique" by definition? Any current GUID creation routine is guarranteed to not create any duplicate.

[rorso]

And more complication, but with some brute force I got it working.

I could have noted in the previous post, that the ID is truncated to 32 characters by my device before it is placed into the search. Formatted UUIDs are too long, adding a prefix does not make it better.

If I am forced to use UUIDs and I'm forced to use a prefix, then I have to shorten the URL.

I did this, by converting the UUID string into a real hex number (32 byte as string, 16 byte as binary) and then base64 encode that -> 24 characters for the ID + 3 for the prefix -> 27 characters in total. That easily fits into the 32-character limit of my device. This requires importing the base64 library.

To encode and decode the UUID on the fly, I had to pimp two functions in generic.py in a way that should not break the current function:

import base64

def generate_stationid_with_prefix(uid, prefix):
    if not prefix or len(prefix) != 2:
        logging.error("Invalid station prefix length (must be 2)")
        return None
    if not uid:
        logging.error("Missing station id for full station id generation")
        return None
    
    #if UUID formatted string, compress by base64 encoding
    if len(uid) == 36 and uid[8] == "-":
        uid = base64.b64encode(bytes.fromhex(uid.replace("-",""))).decode('ascii')
        
    return str(prefix) + '_' + str(uid)

and

def get_stationid_without_prefix(uid):
    if len(uid) < 4:
        logging.error("Could not extract stationid (Invalid station id length)")
        return None
    if uid[2] == "_":
        _uid = uid[3:]
        
        #check for String in UUID format: RB_c86d26e1-0fdd-48e8-b362-67145770d981
        if len(_uid) == 36 and _uid[8] == "-":
            return _uid
        
        #check for string in HEX() format --> convert to UUID string: RB_c86d26e10fdd48e8b36267145770d981
        if len(_uid) == 32 and all(c in "0123456789abcdefABCDEF" for c in _uid):
            return "-".join([_uid[:8],_uid[8:12],_uid[12:16],_uid[16:20],_uid[20:32]])
        
        #check for base64 encoded UUID: RB_yG0m4Q/dSOizYmcUV3DZgQ==
        if len(_uid) == 24 and all(c in "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/= " for c in _uid):
            #account for dumb devices not URLENCODING IDs in parameters by default ...
            _uid = _uid.replace(" ","+")
            _uid = bytes(base64.b64decode(_uid)).hex()
            return "-".join([_uid[:8],_uid[8:12],_uid[12:16],_uid[16:20],_uid[20:32]])
        
        #if nothing else, return entire remainder: RB_67145770d981
        return _uid
    
    # > 3 characters but no prefix? Return original: 67145770d981
    return uid

I added a new function in radiobrowser.py that delivers a station-list when queried with one or more UUIDs.

def search_uid(uid):
    stations = []
    stations_json = request('stations/byuuid/' + str(uid))
    for station_json in stations_json:
        if SHOW_BROKEN_STATIONS or get_json_attr(station_json, 'lastcheckok') == 1:
            stations.append(Station(station_json))
    return stations

and I added a case to get this called within server.py when the URL contains "Search". This most likely specific to my device.

def upstream(path):
    if request.args.get('token') == '0':
        return vtuner.get_init_token()
    if request.args.get('search'):
        return station_search()
    if 'statxml.asp' in path and request.args.get('id'):
        return get_station_info()
    if 'Search.asp' in path and request.args.get('Search'):
        return station_by_id()
    if 'navXML.asp' in path:
        return radiobrowser_landing()
    if 'FavXML.asp' in path:
        return my_stations_landing()
    if 'loginXML.asp' in path:
        return landing()
    logging.error("Unhandled upstream query (/setupapp/%s)", path)
    abort(404)

def station_by_id():
    query = request.args.get('Search')
    if not query or len(query) < 3:
        page = vtuner.Page()
        page.add(vtuner.Display("Search query too short"))
        page.set_count(1)
        return page.to_string()
    else:
        uuid = generic.get_stationid_without_prefix(query)  # should take care to explode compressed UUids
        stations = radiobrowser.search_uid(uuid)
        return get_stations_page(stations, request, details=True).to_string()

You might note the additional parameter details=True. I invented that to either produce a reduced set of tags in the station list vs. a full set when queried by ID. This is definitely specific to my device and would have to get properly masked to "only when Grundig / and or specific URL parameters". Since all of the classes in vtuner.py are looped with their specific to_xml, I had to invent this dummy "details" to all of them, but I just used it in the Station.to_xml

class Station:
    def __init__(self, uid, name, description, url, icon, genre, location, mime, bitrate, bookmark):
        self.uid = uid
        self.name = name
        self.description = description
        self.url = strip_https(url)
        self.trackurl = None
        self.icon = icon
        self.genre = genre
        self.location = location
        self.mime = mime
        self.bitrate = bitrate
        self.bookmark = bookmark

    def set_trackurl(self, url):
        self.trackurl = url

    def to_xml(self, details=False):
        item = ET.Element('Item')
        logging.debug("Station.to_xml() details: %s", details)
        ET.SubElement(item, 'ItemType').text = 'Station'
        ET.SubElement(item, 'StationId').text = self.uid
        ET.SubElement(item, 'StationName').text = self.name
        if self.trackurl:
            ET.SubElement(item, 'StationUrl').text = self.trackurl
        else:
            ET.SubElement(item, 'StationUrl').text = self.url
        if details:
            ET.SubElement(item, 'StationDesc').text = self.description
            ET.SubElement(item, 'Logo').text = self.icon
            ET.SubElement(item, 'StationFormat').text = self.genre
            ET.SubElement(item, 'StationLocation').text = self.location
            ET.SubElement(item, 'StationBandWidth').text = str(self.bitrate)
            ET.SubElement(item, 'StationMime').text = self.mime
            ET.SubElement(item, 'Relia').text = '3'
        #ET.SubElement(item, 'Bookmark').text = self.bookmark
        ET.SubElement(item, 'Bookmark').text = self.url
        return item

Ah, yes - I faked the Bookmark entry. Does not "work", but I could not leave it empty for my device...

This made my code work for the "Grundig SonoClock 890A", at least with the RadioBrowser stations. I'm still thinking about how to handle my_stations.

This certainly does not make a good pull request by now, but eventually someone can still comment about it if/how this eventually could enhance the current codebase.

Btw: I stumbled over a possible bug in get_stations_page that replaces the URL to the station Icons, overwriting the RB supplied URL. Might be intended, but maybe this should be done only in case of station_tracking (indentation problem). But even then, parameters in URLs should get URLENCODED. You never know for sure what the value contains. A base64 encoded UUID for instance :-)

The "search" functions use unencoded search arguments too. Might break easily.

[rorso]

Some final tweaks:

The entitiy decoding in vtuner.py Class Page: did not work out. That one does it right. This does not guarantee that the characters are correct in the display - most foreign (Unicode) characters are still "?" but at least the Ӓ are gone now and at least some umlauts and accented characters do show correct:

return XML_HEADER + ET.tostring(self.to_xml(), encoding='unicode')

After setting some defaults in case a value was not defined, my device now accepts the bookmarked entries too. It definitely does NOT like empty tags:

        if details:
            ET.SubElement(item, 'StationDesc').text = self.description
            ET.SubElement(item, 'Logo').text = self.icon
            ET.SubElement(item, 'StationFormat').text = self.genre
            ET.SubElement(item, 'StationLocation').text = self.location if self.location != None else "XX"
            ET.SubElement(item, 'StationBandWidth').text = str(self.bitrate) if self.bitrate != None else "128"
            ET.SubElement(item, 'StationMime').text = self.mime if self.mime != None else "mp3"
            ET.SubElement(item, 'Relia').text = '3'
        ET.SubElement(item, 'Bookmark').text = self.bookmark if self.bookmark != None else self.url

I still could need some queries from other devices to confine all those changes to my model and not disturb others...