Gtools commands are limited by matsize

[mcaceresb]

The maximum number of variables that can be passed to gtools commands is the maximum value of matsize in the user's system.

disp c(matsize)

Will show the user how many variables can be used with gtools. This limitation is due to the way Stata's C API is designed. I use matrices to pass various necessary information to and from C, so this limitation will almost certainly not change unless the Stata C API changes.

So, for example, the following fails in Stata/IC:

. clear

. set obs 10
obs was 0, now 10

. forvalues i = 1/800 {
  2.     gen x`i' = _n
  3. }
 
. * This is fine

. gisid x*

. gen x801 = 10
 
. * But now this fails

. gisid x*

# variables > matsize (801 > 800). Tried to run

    set matsize 801

but the command failed. Try setting matsize manually.
r(908);

[mcaceresb]

There is actually a fix for this, which blows my mind and am somewhat hesitant to implement. Error:

matrix a = J(1, 20000, 1)

No error (!!!)

mata st_matrix("a", J(1, 20000, 1))

The latter, howver, takes 8 seconds (!) and does seem to consume a lot more memory than it should. It seems that matrix for whatever reason is very slow and memory hungry. Both of these are instantaneous:

mata a = J(1, 20000, 1)
set obs 20000
gen a = 1

[mcaceresb]

Another fix: Save from mata to tempfile; implement function in C: sf_read_vector_tempfile