GitHub search results on Web UI seem to be too less

[dalance]

I believe the condition for adding a new language to this project requires more than 2,000 files to be found via GitHub search.
However, it appears that a problem has arisen where the number of search results visible on the Web UI is far too low.
For example, the TMDL files that were recently merged were reported in #7750 to exceed 19k files, but the current search result shows only 106.

https://github.com/search?type=code&q=NOT+is%3Afork+path%3A*.tmdl

Even Rust shows only 109 results, which is clearly far too low.

https://github.com/search?type=code&q=NOT+is%3Afork+path%3A*.rs

From my recollection, this phenomenon started around the end of last year to the beginning of this year.

On the other hand, searches performed via the GitHub API do not seem to be affected by this issue.
For my language, I regularly perform searches via the API to check whether the 2,000-file threshold is met, and I consistently get stable results as shown below.
(Additionally, since we use the files returned by these searches as test cases for the language, we can be certain that the files actually exist.)

I suspect this is likely a GitHub-side issue, but there is a possibility that the Linguist merge condition has effectively become impossible to satisfy. It will still take some time for our language to meet the conditions, but once the API-based search results clearly exceed the threshold, would it be okay to submit a PR at that point?

[lildude]

I suspect this is likely a GitHub-side issue

It is. We discovered it late last week and it's due to a change associated with the path operator. GitHub's Search team are investigating the issue.

It will still take some time for our language to meet the conditions, but once the API-based search results clearly exceed the threshold, would it be okay to submit a PR at that point?

I'm not accepting any new submissions until this issue is fixed but I'd be interested in seeing your code for this as it might prove useful in coming up with a much more predictable and reliable method everyone (even AI and CI) can use. Previous attempts were inefficient and subject to severe rate limiting.

[dalance]

I see. I'm relieved to hear that the investigation has already been started.
The code we are currently using is as follows.

• The function to call search API
  https://github.com/veryl-lang/discovery/blob/be484acfe02e1c89c2da2c9731defb1f6fe8ff9f/src/db.rs#L155

• call the above funciton and get total_count
  https://github.com/veryl-lang/discovery/blob/be484acfe02e1c89c2da2c9731defb1f6fe8ff9f/src/db.rs#L176

We call this program through GitHub Actions and use GITHUB_TOKEN passed from GitHub Action runner.

https://github.com/veryl-lang/discovery/blob/be484acfe02e1c89c2da2c9731defb1f6fe8ff9f/.github/workflows/periodic.yml

[lildude]

Thanks for that. I see it's quite simple and effectively just using the same figure the UI should return. My previous attempts were more involved attempting to ensure a good distribution across unique :user/:repo combos and this is what hit the rate limiting.

[dalance]

This issue seems to be fixed.

[lildude]

Yes. The fix rolled out yesterday.