Skip to content

Commit a2d0a64

Browse files
rodrigoprimorodrigoprimo
committed
Add script to validate code examples in sniff documentation
This commit introduces a new script that verifies code examples in PHPCS sniff documentation. The script checks that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff, ensuring code examples correctly demonstrate the behavior of the sniff. Noteworthy implementation decisions: - The script will be available in the Composer vendor/bin directory as `phpcs-check-doc-examples`. - To check for syntax errors in the code examples, the code uses `token_get_all()` with the flag TOKEN_PARSE. This means that PHP >= 7.0 is required to run this script while the rest of the repository requires PHP >= 5.4. A requirement check was added to `phpcs-check-doc-examples` to alert users if they are running PHP < 7.0, and a warning was added to the top of the file informing that `phpcs-check-doc-examples` must remain compatible with PHP 5.4 for the requirement check to work. - The script is unable to differentiate between multiple code examples in the same <code> block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same <code> block, there is at least one example that does trigger the sniff. - The tests for this script run on a separate GitHub action job as they need to be tested against a different set of PHP versions than the rest of the tests in this repository, and they need to be tested against multiple PHPCS versions and OSes. The DebugSniff tests need to be tested against multiple PHPCS versions but not multiple OSes, while the tests for the other tools need to be tested against multiple OSes but not multiple PHPCS versions. How to use the script: - Run it from the root directory of a PHPCS standard: ``` $ /path/to/script/phpcs-check-doc-examples ``` The script will search for all sniff documentation files and check their code examples. For other ways to run the script check the help page: ``` $ /path/to/script/phpcs-check-doc-examples --help ```
1 parent 0e2e7b5 commit a2d0a64

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+3774
-6
lines changed

‎.github/workflows/quicktest.yml

Lines changed: 89 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,14 @@ jobs:
6868
if: ${{ matrix.php >= '7.2' }}
6969
run: composer lint
7070

71-
- name: Lint against parse errors (PHP < 7.2)
72-
if: ${{ matrix.php < '7.2' }}
71+
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
72+
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
7373
run: composer lintlt72
7474

75+
- name: Lint against parse errors (PHP < 7.0)
76+
if: ${{ matrix.php < '7.0' }}
77+
run: composer lintlt70
78+
7579
# Check that any sniffs available are feature complete.
7680
# This acts as an integration test for the feature completeness script,
7781
# which is why it is run against various PHP versions.
@@ -182,3 +186,86 @@ jobs:
182186
183187
- name: Run the unit tests for the PHPCSDebug sniff
184188
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}
189+
190+
# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
191+
quicktest-doc-code-examples:
192+
runs-on: ${{ matrix.os }}
193+
194+
strategy:
195+
matrix:
196+
os: ['ubuntu-latest', 'windows-latest']
197+
php: ['7.0', 'latest']
198+
phpcs_version: ['dev-master']
199+
200+
include:
201+
- os: 'ubuntu-latest'
202+
php: '7.0'
203+
phpcs_version: '3.1.0'
204+
- os: 'windows-latest'
205+
php: '7.0'
206+
phpcs_version: '3.1.0'
207+
208+
name: "QTest: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"
209+
210+
steps:
211+
- name: Prepare git to leave line endings alone
212+
run: git config --global core.autocrlf input
213+
214+
- name: Checkout code
215+
uses: actions/checkout@v4
216+
217+
# On stable PHPCS versions, allow for PHP deprecation notices.
218+
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
219+
- name: Setup ini config
220+
id: set_ini
221+
shell: bash
222+
run: |
223+
if [ "${{ matrix.phpcs_version }}" != "dev-master" ]; then
224+
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
225+
else
226+
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
227+
fi
228+
229+
- name: Install PHP
230+
uses: shivammathur/setup-php@v2
231+
with:
232+
php-version: ${{ matrix.php }}
233+
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
234+
coverage: none
235+
236+
- name: 'Composer: adjust dependencies'
237+
run: |
238+
# Set the PHPCS version to be used in the tests.
239+
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
240+
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
241+
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction
242+
243+
# Install dependencies and handle caching in one go.
244+
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
245+
- name: Install Composer dependencies
246+
uses: "ramsey/composer-install@v3"
247+
with:
248+
# Bust the cache at least once a month - output format: YYYY-MM.
249+
custom-cache-suffix: $(date -u "+%Y-%m")
250+
251+
- name: Grab PHPUnit version
252+
id: phpunit_version
253+
shell: bash
254+
# yamllint disable rule:line-length
255+
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
256+
# yamllint enable rule:line-length
257+
258+
- name: Determine PHPUnit composer script to use
259+
id: phpunit_script
260+
shell: bash
261+
run: |
262+
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
263+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
264+
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
265+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
266+
else
267+
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
268+
fi
269+
270+
- name: Run the unit tests for the DocCodeExamples script
271+
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}

‎.github/workflows/test.yml

Lines changed: 196 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,13 @@ jobs:
7373
if: ${{ matrix.php >= '7.2' }}
7474
run: composer lint -- --checkstyle | cs2pr
7575

76-
- name: Lint against parse errors (PHP < 7.2)
77-
if: ${{ matrix.php < '7.2' }}
78-
run: composer lintlt72 -- --checkstyle | cs2pr
76+
- name: Lint against parse errors (PHP < 7.2 && PHP >= 7.0)
77+
if: ${{ matrix.php < '7.2' && matrix.php >= '7.0' }}
78+
run: composer lintlt72
79+
80+
- name: Lint against parse errors (PHP < 7.0)
81+
if: ${{ matrix.php < '7.0' }}
82+
run: composer lintlt70
7983

8084
# Check that any sniffs available are feature complete.
8185
# This also acts as an integration test for the feature completeness script,
@@ -229,3 +233,192 @@ jobs:
229233
230234
- name: Run the unit tests for the PHPCSDebug sniff
231235
run: composer test-sniff${{ steps.phpunit_script.outputs.SUFFIX }}
236+
237+
# Run CI checks/tests for the DocCodeExamples script as it requires PHP >= 7.0.
238+
test-doc-code-examples:
239+
runs-on: ${{ matrix.os }}
240+
241+
strategy:
242+
matrix:
243+
os: ['ubuntu-latest', 'windows-latest']
244+
# IMPORTANT: test runs shouldn't fail because of PHPCS being incompatible with a PHP version.
245+
# - PHPCS will run without errors on PHP 7.0 - 7.2 on any version.
246+
# - PHP 7.3 needs PHPCS 3.3.1+ to run without errors.
247+
# - PHP 7.4 needs PHPCS 3.5.0+ to run without errors.
248+
# - PHP 8.0 needs PHPCS 3.5.7+ to run without errors.
249+
# - PHP 8.1 needs PHPCS 3.6.1+ to run without errors.
250+
# - PHP 8.2 needs PHPCS 3.6.1+ to run without errors.
251+
# - PHP 8.3 needs PHPCS 3.8.0+ to run without errors (though the errors don't affect this package).
252+
# - PHP 8.4 needs PHPCS 3.8.0+ to run without errors (officially 3.11.0, but 3.8.0 will work fine).
253+
php: ['7.0', '7.1', '7.2']
254+
phpcs_version: ['3.1.0', 'dev-master']
255+
256+
include:
257+
# Complete the matrix, while preventing issues with PHPCS versions incompatible with certain PHP versions.
258+
- os: 'ubuntu-latest'
259+
php: '7.3'
260+
phpcs_version: 'dev-master'
261+
- os: 'windows-latest'
262+
php: '7.3'
263+
phpcs_version: 'dev-master'
264+
- os: 'ubuntu-latest'
265+
php: '7.3'
266+
phpcs_version: '3.3.1'
267+
- os: 'windows-latest'
268+
php: '7.3'
269+
phpcs_version: '3.3.1'
270+
271+
- os: 'ubuntu-latest'
272+
php: '7.4'
273+
phpcs_version: 'dev-master'
274+
- os: 'windows-latest'
275+
php: '7.4'
276+
phpcs_version: 'dev-master'
277+
- os: 'ubuntu-latest'
278+
php: '7.4'
279+
phpcs_version: '3.5.0'
280+
- os: 'windows-latest'
281+
php: '7.4'
282+
phpcs_version: '3.5.0'
283+
284+
- os: 'ubuntu-latest'
285+
php: '8.0'
286+
phpcs_version: 'dev-master'
287+
- os: 'windows-latest'
288+
php: '8.0'
289+
phpcs_version: 'dev-master'
290+
- os: 'ubuntu-latest'
291+
php: '8.0'
292+
phpcs_version: '3.5.7'
293+
- os: 'windows-latest'
294+
php: '8.0'
295+
phpcs_version: '3.5.7'
296+
297+
- os: 'ubuntu-latest'
298+
php: '8.1'
299+
phpcs_version: 'dev-master'
300+
- os: 'windows-latest'
301+
php: '8.1'
302+
phpcs_version: 'dev-master'
303+
- os: 'ubuntu-latest'
304+
php: '8.1'
305+
phpcs_version: '3.6.1'
306+
- os: 'windows-latest'
307+
php: '8.1'
308+
phpcs_version: '3.6.1'
309+
310+
- os: 'ubuntu-latest'
311+
php: '8.2'
312+
phpcs_version: 'dev-master'
313+
- os: 'windows-latest'
314+
php: '8.2'
315+
phpcs_version: 'dev-master'
316+
- os: 'ubuntu-latest'
317+
php: '8.2'
318+
phpcs_version: '3.6.1'
319+
- os: 'windows-latest'
320+
php: '8.2'
321+
phpcs_version: '3.6.1'
322+
323+
- os: 'ubuntu-latest'
324+
php: '8.3'
325+
phpcs_version: 'dev-master'
326+
- os: 'windows-latest'
327+
php: '8.3'
328+
phpcs_version: 'dev-master'
329+
- os: 'ubuntu-latest'
330+
php: '8.3'
331+
phpcs_version: '3.8.0'
332+
- os: 'windows-latest'
333+
php: '8.3'
334+
phpcs_version: '3.8.0'
335+
336+
- os: 'ubuntu-latest'
337+
php: '8.4'
338+
phpcs_version: 'dev-master'
339+
- os: 'windows-latest'
340+
php: '8.4'
341+
phpcs_version: 'dev-master'
342+
- os: 'ubuntu-latest'
343+
php: '8.4'
344+
phpcs_version: '3.8.0'
345+
- os: 'windows-latest'
346+
php: '8.4'
347+
phpcs_version: '3.8.0'
348+
349+
# Experimental builds. These are allowed to fail.
350+
- os: 'ubuntu-latest'
351+
php: '7.4'
352+
phpcs_version: '4.0.x-dev'
353+
354+
- os: 'ubuntu-latest'
355+
php: '8.5' # Nightly.
356+
phpcs_version: 'dev-master'
357+
358+
name: "Test: PHP ${{ matrix.php }} - PHPCS ${{ matrix.phpcs_version }} (${{ matrix.os == 'windows-latest' && 'Win' || 'Linux' }})"
359+
360+
continue-on-error: ${{ matrix.php == '8.5' || matrix.phpcs_version == '4.0.x-dev' }}
361+
362+
steps:
363+
- name: Prepare git to leave line endings alone
364+
run: git config --global core.autocrlf input
365+
366+
- name: Checkout code
367+
uses: actions/checkout@v4
368+
369+
- name: Setup ini config
370+
id: set_ini
371+
shell: bash
372+
run: |
373+
# On stable PHPCS versions, allow for PHP deprecation notices.
374+
# Unit tests don't need to fail on those for stable releases where those issues won't get fixed anymore.
375+
if [[ "${{ matrix.phpcs_version }}" != "dev-master" && "${{ matrix.phpcs_version }}" != "4.0.x-dev" ]]; then
376+
echo 'PHP_INI=error_reporting=E_ALL & ~E_DEPRECATED, display_errors=On' >> "$GITHUB_OUTPUT"
377+
else
378+
echo 'PHP_INI=error_reporting=-1, display_errors=On' >> "$GITHUB_OUTPUT"
379+
fi
380+
381+
- name: Install PHP
382+
uses: shivammathur/setup-php@v2
383+
with:
384+
php-version: ${{ matrix.php }}
385+
ini-values: ${{ steps.set_ini.outputs.PHP_INI }}
386+
coverage: none
387+
388+
- name: 'Composer: adjust dependencies'
389+
run: |
390+
# Set the PHPCS version to be used in the tests.
391+
composer require --no-update squizlabs/php_codesniffer:"${{ matrix.phpcs_version }}" --no-scripts --no-interaction
392+
# Remove the PHPCSDevCS dependency as it has different PHPCS requirements and would block installs.
393+
composer remove --no-update --dev phpcsstandards/phpcsdevcs --no-scripts --no-interaction
394+
395+
# Install dependencies and handle caching in one go.
396+
# @link https://github.com/marketplace/actions/install-php-dependencies-with-composer
397+
- name: Install Composer dependencies
398+
uses: "ramsey/composer-install@v3"
399+
with:
400+
composer-options: ${{ matrix.php == '8.5' && '--ignore-platform-req=php+' || '' }}
401+
# Bust the cache at least once a month - output format: YYYY-MM.
402+
custom-cache-suffix: $(date -u "+%Y-%m")
403+
404+
- name: Grab PHPUnit version
405+
id: phpunit_version
406+
shell: bash
407+
# yamllint disable rule:line-length
408+
run: echo "VERSION=$(vendor/bin/phpunit --version | grep --only-matching --max-count=1 --extended-regexp '\b[0-9]+\.[0-9]+')" >> "$GITHUB_OUTPUT"
409+
# yamllint enable rule:line-length
410+
411+
- name: Determine PHPUnit composer script to use
412+
id: phpunit_script
413+
shell: bash
414+
run: |
415+
if [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '11.' ) }}" == "true" ]; then
416+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
417+
elif [ "${{ startsWith( steps.phpunit_version.outputs.VERSION, '10.' ) }}" == "true" ]; then
418+
echo 'SUFFIX=' >> "$GITHUB_OUTPUT"
419+
else
420+
echo 'SUFFIX=-lte9' >> "$GITHUB_OUTPUT"
421+
fi
422+
423+
- name: Run the unit tests for the DocCodeExamples script
424+
run: composer test-doc-code-examples${{ steps.phpunit_script.outputs.SUFFIX }}

‎README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ This is a set of tools to assist developers of sniffs for [PHP CodeSniffer][phpc
2727
+ [Stand-alone Installation](#stand-alone-installation)
2828
* [Features](#features)
2929
+ [Checking whether all sniffs in a PHPCS standard are feature complete](#checking-whether-all-sniffs-in-a-phpcs-standard-are-feature-complete)
30+
+ [Checking code examples in sniff documentation are correct](#checking-code-examples-in-sniff-documentation-are-correct)
3031
+ [Sniff Debugging](#sniff-debugging)
3132
+ [Documentation XSD Validation](#documentation-xsd-validation)
3233
* [Contributing](#contributing)
@@ -133,6 +134,49 @@ directories <dir> One or more specific directories to examine.
133134
-V, --version Display the current version of this script.
134135
```
135136

137+
### Checking code examples in sniff documentation are correct
138+
139+
You can verify that the code examples in your sniff documentation XML files correctly demonstrate the behavior of the sniffs. In other words, check that "Valid" examples don't trigger the sniff and "Invalid" examples do trigger the sniff.
140+
141+
Note that while this repository requires PHP >= 5.4, this script requires PHP >= 7.0.
142+
143+
Be aware that this script is unable to differentiate between multiple code examples in the same `<code>` block. This means that for invalid examples, the script might miss ones that don't trigger the sniff if, in the same `<code>` block, there is at least one example that does trigger the sniff.
144+
145+
To use the tool, run it from the root of your standards repo like so:
146+
```bash
147+
# When installed as a project dependency:
148+
vendor/bin/phpcs-check-doc-examples
149+
150+
# When installed globally with Composer:
151+
phpcs-check-doc-examples
152+
153+
# When installed as a git clone or otherwise:
154+
php path/to/PHPCSDevTools/bin/phpcs-check-doc-examples
155+
```
156+
157+
If all is good, the script will exit with code 0 and a summary of what was checked.
158+
159+
If there are issues with the code examples, you will see error messages for each problematic example, like so:
160+
```text
161+
Errors found while processing path/to/project/StandardName/Docs/Category/SniffNameStandard.xml:
162+
163+
ERROR: Code block is valid and PHPCS should have returned nothing, but instead it returned an error.
164+
Code block title: "Valid: invalid valid code example."
165+
Code block content: "function sniffValidationWillFail() {}"
166+
```
167+
168+
#### Options
169+
```text
170+
directories|files One or more specific directories or files to examine.
171+
Defaults to the directory from which the script is run.
172+
--exclude=<dir1,dir2> Comma-delimited list of relative paths of directories to
173+
exclude from the scan.
174+
--ignore-sniffs=<sniff1,sniff2> Comma-delimited list of sniffs to ignore.
175+
--colors Enable colors in console output. (disables auto detection of color support)
176+
--no-colors Disable colors in console output.
177+
--help Print the script help page.
178+
-V, --version Display the current version of this script.
179+
```
136180

137181
### Sniff Debugging
138182

0 commit comments

Comments
 (0)