fix(go): scope SHOW COLUMNS to the requested object in GetObjects - #170
Conversation
GetObjects at Columns and All depth dispatched a SHOW COLUMNS command whose scope resolved to IN ACCOUNT whenever the catalog name contained an underscore, because isWildcardStr treats '_' and '%' as wildcards. For an ordinary database name such as DB_NAME this scanned every column in the entire account, taking 80+ seconds on large accounts, even though the request targeted a single table. Scope the SHOW COLUMNS query to the narrowest concrete object (IN TABLE / IN SCHEMA / IN DATABASE), treating '_' as a literal and only '%', '.*', and nil as broad. The column list still comes from information_schema.columns and the downstream ILIKE filtering is unchanged, so the returned metadata is identical; SHOW COLUMNS is only used to enrich xdbc_column_size for BINARY columns. Also treat Snowflake error 2003 (object does not exist or not authorized) like 2043 (no match) so a narrowly scoped query for a missing object falls back to an empty result set instead of failing. Add a unit test covering the new scope selection.
Address review feedback on the SHOW COLUMNS scoping change: - Limit the "object does not exist" (2003) empty-result fallback to the optional SHOW COLUMNS enrichment query via getQueryID's new variadic alsoEmptyOn parameter, instead of applying it to every getQueryID caller. Required object and constraint discovery keep their prior 2043-only behavior so genuine errors there are not masked. - Document that scopeIdentifier/showColumnsScope treat '_' as a literal by design: SHOW COLUMNS is only a best-effort source for BINARY xdbc_column_size, the column list still comes from information_schema.columns via ILIKE, so a narrow scope never drops columns or changes non-BINARY metadata. - Add a test case covering underscore-as-literal scope selection.
zeroshade
added
lang-go
|
@frbvianna can you take a look and test this out with your config? |
There was a problem hiding this comment.
Pull request overview
Fixes a performance regression in GetObjects at ObjectDepthColumns / ObjectDepthAll where SHOW COLUMNS could incorrectly broaden to IN ACCOUNT (notably when catalog names contain underscores), causing extremely slow metadata fetches on large Snowflake accounts. The change introduces scope narrowing specifically for the optional SHOW COLUMNS enrichment query while preserving existing behavior for required discovery queries.
Changes:
- Add
scopeIdentifierandshowColumnsScopeto choose the narrowest validSHOW COLUMNS ... IN {TABLE|SCHEMA|DATABASE|ACCOUNT}suffix, treating_as literal and only%/.*/ nil as broadening signals. - Extend
getQueryIDwith analsoEmptyOnvariadic parameter so only the optionalSHOW COLUMNSenrichment can treat error2003as an empty-result fallback (required queries keep prior behavior). - Add
TestShowColumnsScopeto validate scoping rules and identifier quoting/escaping.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| go/connection.go | Introduces showColumnsScope and applies it to the SHOW COLUMNS enrichment query; refines empty-result fallback behavior via alsoEmptyOn. |
| go/get_objects_scope_test.go | Adds unit coverage for SHOW COLUMNS scope selection and quote escaping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
I agree, but I don't think that should block this change, should it? |
|
Didn't say it was blocking :) But ideally @frbvianna can confirm that this actually fixes the problem. |
|
I was able to reproduce the issue reported and prove this fixed it for myself, but I agree that it would be better for @frbvianna to confirm |
|
Hey @zeroshade. Thanks for the fix. I gave it a test, and now it's taking a reasonable ~12 seconds to get objects at column depth. |
Summary
Fixes #169.
GetObjectsatObjectDepthColumns/ObjectDepthAlldispatched aSHOW COLUMNScommand whose scope resolved toIN ACCOUNTwhenever the catalog name contained an underscore, becauseisWildcardStrtreats_and%as wildcards. SinceSHOW COLUMNShas no table-levelLIKEfilter,SHOW COLUMNS ... IN ACCOUNTscans every column in the entire account — 80–150s on large accounts — even for a single-table request likeDB_NAME.TEST_SCHEMA.LINEITEM.Changes
scopeIdentifier/showColumnsScopeand scope theSHOW COLUMNSquery to the narrowest concrete object (IN TABLE/IN SCHEMA/IN DATABASE), treating_as a literal and only%/.*/ nil as broadening.2003(object does not exist / not authorized) empty-result fallback to only the optionalSHOW COLUMNSenrichment query, via a new variadicalsoEmptyOnparameter ongetQueryID. Required object/constraint discovery keeps its prior2043-only behavior.Why this is safe
SHOW COLUMNSis only a best-effort source for BINARYxdbc_column_sizeinget_objects_all.sql(aLEFT JOINforbyte_length). The authoritative column list comes frominformation_schema.columnsfiltered byILIKE, so a narrowerSHOW COLUMNSscope never drops columns or alters non-BINARY metadata.Tradeoff: a BINARY column reached only through a genuine
_-as-wildcard match (or a case-insensitive pattern differing from the stored identifier) may report a NULLxdbc_column_size, matching the behavior beforeSHOW COLUMNSenrichment was added in #152. This is documented onscopeIdentifier/showColumnsScope.Scope timing (small test account; worse on large accounts)
SHOW COLUMNSscopeIN ACCOUNTIN DATABASEIN SCHEMAIN TABLETesting
go build,go vet,gofmt: clean.TestShowColumnsScope: passes (concrete underscore names →IN TABLE;%/.*/ nil → broader; quote escaping).GetObjects(Columns, "SNOWFLAKE_SAMPLE_DATA", "TPCH_SF1", "LINEITEM")returns correctly and drops from the ~18sIN ACCOUNTfloor to a table-scoped query; a non-existent table returns an empty result rather than erroring.