VT_GETSTATE invalid argument on embedded system

[allebot]

Originally reported by: tobi_v

Original Ticket: alleg/bugs/367

I'm trying to use Allegro 4.4 on an embedded platform (ea3250 development board), using the Linux console and fbcon subsystems.

The Linux console subsystem tries to open /dev/console to call the VT_GETSTATE ioctl on it, but the kernel returns invalid argument when the /dev/console fd is passed to it, thus init_console fails, and hence the console subsystem prevents setting any graphics mode.

Bypassing the /dev/console fixes this, i.e. I hacked Allegro like this:

--- a/src/linux/lconsole.c
+++ b/src/linux/lconsole.c
@@ -118,7 +118,7 @@ static int init_console(void)
* opening each unused one in turn.
*/

•  if ((console_fd = open ("/dev/console", O_WRONLY)) < 0) {
  

•  if (TRUE || (console_fd = open ("/dev/console", O_WRONLY)) < 0) {
   int n;
   uszprintf (allegro_error, ALLEGRO_ERROR_SIZE, uconvert_ascii("%s /dev/console: %s", tmp),
              get_config_text("Unable to open"), ustrerror (errno));
  

Possibly /dev/console no longer has the function it had when this code was written?

A better solution may be desired but this hack at least makes the Linux console subsystem work on devices where the current VT can't be found. (The terminal on which the Allegro program is started is typically /dev/pts/N for ssh connections or /dev/ttySN for serial connections on such a device.)

Related: http://www.allegro.cc/forums/thread/605565/890751