diff --git a/.github/workflows/pkgdown.yaml b/.github/workflows/pkgdown.yaml index ce79bca..76e3167 100644 --- a/.github/workflows/pkgdown.yaml +++ b/.github/workflows/pkgdown.yaml @@ -2,7 +2,7 @@ # Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help on: push: - branches: [main, master] + branches: [main, master, playground] pull_request: branches: [main, master] release: diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9c3def6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,113 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository Overview + +tree-sitter-r is an R grammar implementation for [tree-sitter](https://github.com/tree-sitter/tree-sitter), a parser generator tool and incremental parsing library. This repository provides: + +1. The core grammar implementation for the R language +2. Node.js bindings for use in JavaScript/TypeScript projects +3. Rust bindings for use in Rust projects +4. R package bindings for use in R projects + +The grammar precisely implements R language syntax with a few documented deviations, focusing on providing a useful syntax tree structure for tooling. + +## Common Development Commands + +### Core Grammar Development + +```bash +# Install dependencies +npm install + +# Build the grammar and generate parser +tree-sitter generate + +# Start the tree-sitter playground for testing +npm run start + +# Run parser tests +tree-sitter test + +# Generate WASM build +tree-sitter build --wasm +``` + +### Node.js Package + +```bash +# Install dependencies +npm install + +# Run Node bindings tests +node --test bindings/node/*_test.js + +# Build Node bindings +node-gyp rebuild +``` + +### Rust Crate + +```bash +# Build Rust crate +cargo build + +# Test Rust crate +cargo test +``` + +### R Package + +```bash +# Build and check R package +cd bindings/r +R CMD build . +R CMD check treesitter.r_*.tar.gz + +# Run R package tests +cd bindings/r +R -e 'devtools::test()' +``` + +## Repository Structure + +- `/grammar.js` - The main grammar definition file defining R syntax rules +- `/src/` - Contains parser implementation files generated by tree-sitter + - `parser.c` - The core parser implementation + - `scanner.c` - Custom scanner for handling context-sensitive tokens +- `/bindings/` - Language-specific bindings + - `/bindings/node/` - Node.js bindings + - `/bindings/rust/` - Rust bindings + - `/bindings/r/` - R package implementation +- `/test/` - Test files for the grammar + - `/test/corpus/` - Syntax test cases + - `/test/highlight/` - Syntax highlighting test cases + - `/test/tags/` - Symbol tagging test cases + +## Architecture + +The tree-sitter-r project follows the standard tree-sitter architecture: + +1. **Grammar Definition**: The core of the project is the `grammar.js` file that defines the grammar rules for parsing R code. It defines all language constructs like expressions, statements, operators, etc. + +2. **Generated Parser**: The tree-sitter CLI uses the grammar definition to generate the actual parser implementation in C, which is stored in `src/parser.c`. + +3. **Custom Scanner**: For handling context-sensitive aspects of the R syntax, a custom scanner is implemented in `src/scanner.c`. This handles special cases that cannot be expressed using the grammar alone. + +4. **Language Bindings**: The project provides bindings for Node.js, Rust, and R, allowing the parser to be used from these languages. + +5. **Tests**: Comprehensive tests in the `/test` directory verify the grammar works as expected with different R constructs. + +## Working with the Grammar + +When modifying the grammar: + +1. Edit `grammar.js` to make your changes to the grammar definition +2. Run `tree-sitter generate` to regenerate the parser +3. Test your changes with `tree-sitter test` +4. Use `npm run start` to launch the playground for interactive testing + +## Known Grammar Deviations + +As noted in the README, there are a few intentional deviations from the R grammar, particularly around how `]]` tokens are handled. See the README.md for details. diff --git a/bindings/r/.Rbuildignore b/bindings/r/.Rbuildignore index e008552..1dd9f8c 100644 --- a/bindings/r/.Rbuildignore +++ b/bindings/r/.Rbuildignore @@ -14,3 +14,4 @@ ^cran-comments\.md$ ^[\.]?air\.toml$ ^\.vscode$ +^playground$ diff --git a/bindings/r/_pkgdown.yml b/bindings/r/_pkgdown.yml index a2c3285..c4d9e46 100644 --- a/bindings/r/_pkgdown.yml +++ b/bindings/r/_pkgdown.yml @@ -1,8 +1,26 @@ -url: ~ - +url: https://jennybc.github.io/tree-sitter-r/ development: mode: auto - template: package: tidytemplate bootstrap: 5 + includes: + in_header: | + + assets: playground +navbar: + structure: + left: + - intro + - reference + - articles + - tutorials + - playground + right: + - search + - github + components: + playground: + text: Playground + href: playground/index.html + diff --git a/bindings/r/pkgdown/assets/playground/assets/playground.css b/bindings/r/pkgdown/assets/playground/assets/playground.css new file mode 100644 index 0000000..3c44ff8 --- /dev/null +++ b/bindings/r/pkgdown/assets/playground/assets/playground.css @@ -0,0 +1,167 @@ +/* Basic playground CSS */ +:root { + --primary-color: #0550ae; + --bg-color: #ffffff; + --text-color: #333333; + --border-color: #cccccc; + --highlight-color: #e3f2fd; + --error-color: #f44336; +} + +body.dark-theme { + --primary-color: #4f83cc; + --bg-color: #1e1e1e; + --text-color: #eaeaea; + --border-color: #555555; + --highlight-color: #284766; + --error-color: #ff6b6b; +} + +.playground { + display: flex; + flex-direction: column; + height: calc(100vh - 100px); + min-height: 500px; + font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; +} + +.playground-header { + padding: 10px; + font-size: 16px; + font-weight: bold; + border-bottom: 1px solid var(--border-color); +} + +.playground-body { + display: flex; + flex: 1; + overflow: hidden; +} + +.playground-section { + display: flex; + flex-direction: column; + overflow: hidden; + border-right: 1px solid var(--border-color); +} + +.playground-section:last-child { + border-right: none; +} + +.playground-section-header { + padding: 5px 10px; + background-color: #f5f5f5; + border-bottom: 1px solid var(--border-color); + font-weight: bold; + font-size: 14px; +} + +body.dark-theme .playground-section-header { + background-color: #2d2d2d; +} + +.playground-section-content { + flex: 1; + overflow: auto; + position: relative; +} + +/* Code editor section */ +.code-section { + width: 50%; +} + +/* Tree output section */ +.tree-section { + width: 50%; +} + +/* Optional query section */ +.query-section { + width: 100%; + height: 150px; + flex: 0 0 auto; + border-top: 1px solid var(--border-color); + border-right: none; +} + +/* Tree nodes */ +.tree-row { + padding: 2px 0; + cursor: pointer; + font-family: monospace; + white-space: nowrap; +} + +.tree-row:hover { + background-color: var(--highlight-color); +} + +.field-name { + color: #9a6700; +} + +body.dark-theme .field-name { + color: #d9bf8c; +} + +.node-type { + color: #004d40; +} + +body.dark-theme .node-type { + color: #4db6ac; +} + +.anonymous-node-type { + color: #7d7d7d; + font-style: italic; +} + +.tree-node-position { + color: #7d7d7d; + font-size: 80%; +} + +/* CodeMirror customizations */ +.CodeMirror { + height: 100%; + font-family: monospace; + font-size: 14px; +} + +/* Error message */ +.error-message { + color: var(--error-color); + padding: 10px; + font-family: monospace; + white-space: pre-wrap; + font-size: 14px; +} + +/* Query captures */ +.capture-node { + background-color: rgba(86, 156, 214, 0.2); +} + +/* Media queries for responsive layout */ +@media (max-width: 768px) { + .playground-body { + flex-direction: column; + } + + .code-section, .tree-section { + width: 100%; + height: 50%; + } + + .playground-section { + border-right: none; + border-bottom: 1px solid var(--border-color); + } + + .playground-section:last-child { + border-bottom: none; + } +} \ No newline at end of file diff --git a/bindings/r/pkgdown/assets/playground/assets/playground.js b/bindings/r/pkgdown/assets/playground/assets/playground.js new file mode 100644 index 0000000..12e428f --- /dev/null +++ b/bindings/r/pkgdown/assets/playground/assets/playground.js @@ -0,0 +1,283 @@ +// Simplified Tree-sitter R playground implementation +// Based on the tree-sitter playground but focused only on R language + +(function() { + // DOM elements we'll use + let codeEditor, tree, codeContainer, treeContainer, queryContainer, queryEditor; + // Libraries we depend on + let CodeMirror, TreeSitter, Clusterize; + // The tree-sitter objects we'll use + let parser, rLanguage; + + // Wait for page to load then initialize the playground + document.addEventListener('DOMContentLoaded', async function() { + // Wait until all required libraries are loaded + await initializeLibraries(); + + // Get the DOM elements + codeContainer = document.getElementById('code-container'); + treeContainer = document.getElementById('tree-container'); + queryContainer = document.getElementById('query-container'); + + // Create code editor + codeEditor = CodeMirror(codeContainer, { + mode: 'r', + lineNumbers: true, + showCursorWhenSelecting: true, + tabSize: 2, + theme: 'default', + value: getStoredState('code') || 'function(x) {\n if (x > 0) {\n return(sqrt(x))\n } else {\n return(NA)\n }\n}' + }); + + // Create query editor if the container exists + if (queryContainer) { + queryEditor = CodeMirror(queryContainer, { + mode: 'scm', + lineNumbers: false, + tabSize: 2, + theme: 'default', + placeholder: 'Enter a tree-sitter query...', + value: getStoredState('query') || '' + }); + } + + // Initialize tree-sitter + initializeParser(); + + // Set up event handlers + setupEvents(); + }); + + // Load required libraries + async function initializeLibraries() { + // These would typically be loaded from CDN or directly in the HTML + // Here we're just ensuring they're available + CodeMirror = window.CodeMirror; + TreeSitter = window.TreeSitter; + Clusterize = window.Clusterize; + + if (!CodeMirror || !TreeSitter || !Clusterize) { + console.error('Required libraries not loaded'); + return; + } + + // Load CodeMirror R mode if not already loaded + if (!CodeMirror.modes.r) { + // We could load it here if needed + } + } + + // Initialize the Tree-sitter parser with R language + async function initializeParser() { + try { + parser = new TreeSitter(); + + // Load the R language WASM file + const wasmUrl = 'assets/tree-sitter-r.wasm'; + console.log('Attempting to load WASM from:', wasmUrl); + rLanguage = await TreeSitter.Language.load(wasmUrl); + + // Set language for parser + parser.setLanguage(rLanguage); + + // Initial parse + parseCode(); + } catch (error) { + console.error('Error initializing parser:', error); + displayError('Failed to load R grammar. Check console for details.'); + } + } + + // Parse current code and update the tree + function parseCode() { + try { + const code = codeEditor.getValue(); + localStorage.setItem('tree-sitter-r-playground-code', code); + + // Parse the code + const tree = parser.parse(code); + + // Render the tree + renderTree(tree); + + // Run query if there's one in the editor + if (queryEditor && queryEditor.getValue().trim()) { + runQuery(); + } + } catch (error) { + console.error('Error parsing code:', error); + displayError('Error parsing code. Check console for details.'); + } + } + + // Render the syntax tree + function renderTree(tree) { + if (!treeContainer) return; + + const root = tree.rootNode; + const rows = []; + const maxRow = codeEditor.lineCount() - 1; + + // Format tree into HTML rows + let formatTree = function(node, depth) { + if (!node) return; + + const start = node.startPosition; + const end = node.endPosition; + const maxColumn = codeEditor.getLine(Math.min(maxRow, end.row)).length; + + // Create row for this node + const row = document.createElement('div'); + row.className = 'tree-row'; + row.style.marginLeft = (depth * 12) + 'px'; + + // Field name if present + if (node.parent && node.parent.childCount > 1) { + const fieldName = node.parent.children.find(n => n === node)?.fieldName; + if (fieldName) { + const fieldNameSpan = document.createElement('span'); + fieldNameSpan.className = 'field-name'; + fieldNameSpan.textContent = fieldName + ': '; + row.appendChild(fieldNameSpan); + } + } + + // Node type + const typeSpan = document.createElement('span'); + typeSpan.className = node.isNamed() ? 'node-type' : 'anonymous-node-type'; + typeSpan.textContent = node.type; + row.appendChild(typeSpan); + + // Position display + const position = document.createElement('span'); + position.className = 'tree-node-position'; + position.textContent = ` [${start.row},${start.column} - ${end.row},${end.column}]`; + row.appendChild(position); + + // Add event listener to highlight the corresponding code when clicked + row.addEventListener('click', () => { + codeEditor.focus(); + codeEditor.setSelection( + { line: start.row, ch: start.column }, + { line: end.row, ch: end.column } + ); + }); + + rows.push(row); + + // Process children recursively + if (node.children && node.children.length > 0) { + node.children.forEach(child => formatTree(child, depth + 1)); + } + }; + + formatTree(root, 0); + + // Clear the tree container + treeContainer.innerHTML = ''; + + // Add all rows to the tree container + rows.forEach(row => treeContainer.appendChild(row)); + } + + // Run a query on the current code + function runQuery() { + if (!queryEditor) return; + + try { + const code = codeEditor.getValue(); + const queryString = queryEditor.getValue(); + + localStorage.setItem('tree-sitter-r-playground-query', queryString); + + if (!queryString.trim()) { + // Clear highlights but don't show error + codeEditor.getAllMarks().forEach(m => m.clear()); + return; + } + + // Parse the code + const tree = parser.parse(code); + + // Create and run query + const query = rLanguage.query(queryString); + const matches = query.matches(tree.rootNode); + + // Clear previous highlights + codeEditor.getAllMarks().forEach(m => m.clear()); + + // Highlight matching nodes + matches.forEach(match => { + match.captures.forEach(capture => { + const { node } = capture; + const start = { line: node.startPosition.row, ch: node.startPosition.column }; + const end = { line: node.endPosition.row, ch: node.endPosition.column }; + + codeEditor.markText(start, end, { + className: `capture-${capture.name || 'node'}` + }); + }); + }); + + } catch (error) { + console.error('Query error:', error); + displayError('Invalid query. Check console for details.'); + } + } + + // Set up event handlers + function setupEvents() { + // Update tree when code changes + codeEditor.on('changes', debounce(parseCode, 250)); + + // Run query when query changes + if (queryEditor) { + queryEditor.on('changes', debounce(runQuery, 250)); + } + + // Add keyboard shortcuts + codeEditor.setOption('extraKeys', { + 'Ctrl-Enter': parseCode, + 'Cmd-Enter': parseCode, + }); + + if (queryEditor) { + queryEditor.setOption('extraKeys', { + 'Ctrl-Enter': runQuery, + 'Cmd-Enter': runQuery, + }); + } + } + + // Helper function to get stored state + function getStoredState(key) { + try { + return localStorage.getItem(`tree-sitter-r-playground-${key}`); + } catch (error) { + console.error('Error accessing localStorage:', error); + return null; + } + } + + // Helper function to display errors + function displayError(message) { + // Simple error display - could be improved + const errorDiv = document.createElement('div'); + errorDiv.className = 'error-message'; + errorDiv.textContent = message; + + treeContainer.innerHTML = ''; + treeContainer.appendChild(errorDiv); + } + + // Debounce function to limit how often a function is called + function debounce(func, wait) { + let timeout; + return function() { + const context = this; + const args = arguments; + clearTimeout(timeout); + timeout = setTimeout(() => func.apply(context, args), wait); + }; + } +})(); \ No newline at end of file diff --git a/bindings/r/pkgdown/assets/playground/index.html b/bindings/r/pkgdown/assets/playground/index.html new file mode 100644 index 0000000..7b290a2 --- /dev/null +++ b/bindings/r/pkgdown/assets/playground/index.html @@ -0,0 +1,418 @@ + + + + + + Tree-sitter R Playground YO + + + + + + + + + + + + + + + +
+
+ Tree-sitter R Grammar Playground + +
+
+ + +
+ +
+ + +
+ +
+ + +
+ +
+ + +
+ +
+ + +
+
+
+ +
+ +
+
R Code
+
+
+ + +
+
Syntax Tree
+
+
+
+ + +
+
Query
+
+
+
+ + + + + + diff --git a/bindings/r/pkgdown/assets/test.html b/bindings/r/pkgdown/assets/test.html new file mode 100644 index 0000000..c40b3b8 --- /dev/null +++ b/bindings/r/pkgdown/assets/test.html @@ -0,0 +1,66 @@ + + + + Tree-sitter WASM Test + + + +

Tree-sitter WASM Test

+
Testing if we can load the Tree-sitter WASM file...
+ + + + \ No newline at end of file