Open
Conversation
…t_cohorts_metadata_query
updating the datamodel branch with the latest changes form main
Latest SPARQL query for the new data model
…ies for better performance visibility
…ery expectations - Remove primary/secondary outcome specs from generic field mapping - Add custom logic in handle_special_fields to create nested structure: protocol -> outcome_specification -> primary/secondary_outcome_specification - This matches the structure expected by the studies metadata SPARQL query - Fixes issue where outcomes were not being fetched from the triplestore
…ER BY) - Change lines[2:290] to lines[2:291] to include line 290 (GROUP BY clause) - Change lines[297:542] to lines[297:543] to include line 542 (ORDER BY clause) - Python slice notation excludes the end index, causing queries to be malformed - This was causing 'QueryBadFormed' errors when executing the queries
- Add query_endpoint.setMethod('POST') to explicitly use POST requests
- Prevents 'headers size should fit in 8kb' error for large queries
- GET method puts query in URL/headers which has size limitations
- POST method sends query in request body with no practical size limit
…_metadata - Add logging at each major step to identify where processing is slow/stuck - Optimize study_name to cohort_id matching using pre-built dictionary - Avoid O(n*m) loop by creating cohort_id_map for fast lookups - Log counts of studies, variables, and processing time at each step - Helps diagnose performance issues with large datasets
Critical fixes to make uploaded data structure match what queries expect: - Change study type to use iao:is_about relationship (was is_described_by) - Change RDF type to sio:descriptor (was sio:study_descriptor) - Add rdfs:label with actual study type value for query matching - Add use_rdfs_label config option to support value-based labels This fixes the issue where queries returned 0 results because: Query expected: ?study iao:is_about ?descriptor . ?descriptor rdfs:label ?study_type Upload created: ?study is_described_by ?descriptor (wrong predicate)
Cross mapping
briniging latest changes and fixes from Komal's branch
Cross mapping latest changes
…ed get_variables_query
… branch structure
merging latest changes from the main branch
… and value context
…uery from main branch
…-based query from main branch" This reverts commit e60b9bb.
…ent instead of ICARE ontology
…e correct field names from CMEO query (study_name, var_name, var_label, etc.)
…studies_metadata graph, Query 2 retrieves variables from study graphs
…ELECT subquery and update line ranges in query extraction functions
…ace in 4 locations
…sing SPARQL parser to fail
…on_process/category structure
…d statistics (count, min, max)
…nly strings and skip categories with empty labels
…et_var_uri and skip empty variable names
…load.py data storage formats for varType and units
…le both CohortVarLinker and upload.py data storage formats
…remove auto-populated mapped_label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bringing in the small fixes from my branch to the cross_mapping