Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Sep 19, 2025

  • Analyze repository structure and existing field patterns
  • Examine existing gender_fields() and language_fields() methods as models
  • Review test structure and requirements
  • Add race_and_ethnicity_fields() method to ALIndividual class
  • Add age_range_fields() method to ALIndividual class
  • Add income_range_fields() method to ALIndividual class
  • Add occupation_fields() method to ALIndividual class
  • Add corresponding questions to ql_baseline.yml for all demographics
  • Add demographic questions to test_question_library.yml
  • Add unit tests for new methods
  • Add integration test scenario
  • Create comprehensive documentation (DEMOGRAPHIC_FIELDS.md)
  • Update methods to support required parameter from PR Add optional required parameter to _fields() methods for granular field requirement control #971
  • Move hardcoded choices to al_ variables in ql_baseline.yml
  • Update methods to use value() pattern like al_pronoun_choices
  • Create test_demographics.yml for interactive exploration
  • Update unit tests with proper mocking for value() function
  • Update docstrings to explicitly reference al_*_choices variables

Summary of Latest Changes:

Updated all demographic method docstrings to explicitly reference the al_*_choices variables defined in ql_baseline.yml:

  • race_and_ethnicity_fields() now references al_race_ethnicity_choices
  • age_range_fields() now references al_age_range_choices
  • income_range_fields() now references al_income_range_choices
  • occupation_fields() now references al_occupation_choices

Each docstring now includes:

  1. Updated choices parameter description mentioning the specific al_* variable used as default
  2. A "Note" section explaining that the default choices are defined in the corresponding al_*_choices variable in ql_baseline.yml and can be customized by redefining the variable

This makes the customization pathway more discoverable for developers using these methods.

The implementation now follows all existing patterns and supports the full range of customization options available in other field methods.

Fixes #455.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Add questions to library for additional demographic data Add demographic data collection methods for race, ethnicity, age, income, and occupation Sep 19, 2025
@Copilot Copilot AI requested a review from nonprofittechy September 19, 2025 13:13
@nonprofittechy
Copy link
Member

@copilot:

Look at #971 and make sure the new methods support the required: Dict[str, bool] modifier added in that PR.

Add a test_demographics.yml file that lets you interactively explore this new feature. Look at existing test_... YML files for structure.

Instead of hardcoding strings in the methods, follow the pattern of al_name_suffixes and al_pronoun_choices which uses value() to get the value of a pre-defined list of options in the interview YAML when the list of choices is not explicitly passed. Keep the al_ prefix and move the definition of these various option lists into ql_baseline.yml with an appropriate comment to separate them from the other similar placeholder definitions. This will make it simpler for others to customize this list in the future.

Copy link
Contributor Author

Copilot AI commented Sep 19, 2025

@copilot:

Look at #971 and make sure the new methods support the required: Dict[str, bool] modifier added in that PR.
...

I've addressed all the feedback points:

  1. ✅ Added required: Dict[str, bool] parameter support - All demographic field methods now support the same required parameter pattern from PR Add optional required parameter to _fields() methods for granular field requirement control #971
  2. ✅ Created test_demographics.yml - Interactive test file for exploring the demographic fields with examples and help text
  3. ✅ Moved choices to al_ variables - Replaced hardcoded strings with al_race_ethnicity_choices, al_age_range_choices, al_income_range_choices, and al_occupation_choices in ql_baseline.yml
  4. ✅ Updated methods to use value() pattern - All methods now follow the value("al_*_choices") pattern like al_pronoun_choices

The implementation now follows the existing patterns consistently and provides the same level of customization as other field methods. Changes in commit 79be9d1.

Copilot finished work on behalf of nonprofittechy September 19, 2025 14:36
@nonprofittechy
Copy link
Member

@copilot update the docstrings for the new methods so they explicitly reference the new hidden al_*_choices values that are defined in ql_baseline.yml

Copy link
Contributor Author

Copilot AI commented Sep 19, 2025

@copilot update the docstrings for the new methods so they explicitly reference the new hidden al_*_choices values that are defined in ql_baseline.yml

Updated the docstrings for all four demographic methods to explicitly reference the al_*_choices variables:

  • race_and_ethnicity_fields()al_race_ethnicity_choices
  • age_range_fields()al_age_range_choices
  • income_range_fields()al_income_range_choices
  • occupation_fields()al_occupation_choices

Each docstring now includes the specific variable name in the choices parameter description and a Note section explaining that these variables are defined in ql_baseline.yml and can be customized. Changes in commit a9eb8c9.

Copilot finished work on behalf of nonprofittechy September 19, 2025 14:43
@nonprofittechy nonprofittechy marked this pull request as ready for review September 19, 2025 14:45
@nonprofittechy
Copy link
Member

nonprofittechy commented Sep 19, 2025

@copilot review the unittest errors. Importantly, value() should be mocked as it won't work properly in a stock Python environment. This does limit value of unittests but they are still helpful to have in place.

......................E.EEEEEEE.........

ERROR: test_age_range_fields_basic (AssemblyLine.test_al_general.test_demographic_fields.test_age_range_fields_basic)
Test basic age_range_fields functionality

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 599, in test_age_range_fields_basic
fields = self.individual.age_range_fields()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1570, in age_range_fields
choices = value("al_age_range_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5057, in _defined_internal
force_ask_nameerror(variable)
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 4219, in force_ask_nameerror
raise DANameError("name '" + str(variable_name) + "' is not defined")
docassemble.base.error.DANameError: name 'al_age_range_choices' is not defined

======================================================================
ERROR: test_demographic_fields_with_maxlengths (AssemblyLine.test_al_general.test_demographic_fields.test_demographic_fields_with_maxlengths)
Test that maxlengths parameter is applied

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 751, in test_demographic_fields_with_maxlengths
fields = self.individual.race_and_ethnicity_fields(maxlengths=maxlengths)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1505, in race_and_ethnicity_fields
choices = value("al_race_ethnicity_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5057, in _defined_internal
force_ask_nameerror(variable)
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 4219, in force_ask_nameerror
raise DANameError("name '" + str(variable_name) + "' is not defined")
docassemble.base.error.DANameError: name 'al_race_ethnicity_choices' is not defined

======================================================================
ERROR: test_demographic_fields_with_required (AssemblyLine.test_al_general.test_demographic_fields.test_demographic_fields_with_required)
Test that required parameter is applied

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 700, in test_demographic_fields_with_required
fields = self.individual.race_and_ethnicity_fields(required=required_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1505, in race_and_ethnicity_fields
choices = value("al_race_ethnicity_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docassemble.base.error.DANameError: name 'al_race_ethnicity_choices' is not defined

======================================================================
ERROR: test_income_range_fields_basic (AssemblyLine.test_al_general.test_demographic_fields.test_income_range_fields_basic)
Test basic income_range_fields functionality

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 620, in test_income_range_fields_basic
fields = self.individual.income_range_fields()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1626, in income_range_fields
choices = value("al_income_range_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5057, in _defined_internal
force_ask_nameerror(variable)
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 4219, in force_ask_nameerror
raise DANameError("name '" + str(variable_name) + "' is not defined")
docassemble.base.error.DANameError: name 'al_income_range_choices' is not defined

======================================================================
ERROR: test_occupation_fields_basic (AssemblyLine.test_al_general.test_demographic_fields.test_occupation_fields_basic)
Test basic occupation_fields functionality

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 642, in test_occupation_fields_basic
fields = self.individual.occupation_fields()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1684, in occupation_fields
choices = value("al_occupation_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5057, in _defined_internal
force_ask_nameerror(variable)
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 4219, in force_ask_nameerror
raise DANameError("name '" + str(variable_name) + "' is not defined")
docassemble.base.error.DANameError: name 'al_occupation_choices' is not defined

======================================================================
ERROR: test_race_and_ethnicity_fields_basic (AssemblyLine.test_al_general.test_demographic_fields.test_race_and_ethnicity_fields_basic)
Test basic race_and_ethnicity_fields functionality

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/unittest/mock.py", line 1396, in patched
return func(*newargs, **newkeywargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/test_al_general.py", line 573, in test_race_and_ethnicity_fields_basic
fields = self.individual.race_and_ethnicity_fields()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/docassemble-AssemblyLine/docassemble-AssemblyLine/docassemble/AssemblyLine/al_general.py", line 1505, in race_and_ethnicity_fields
choices = value("al_race_ethnicity_choices")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5173, in value
return _defined_internal(var, DefCaller.VALUE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 5057, in _defined_internal
force_ask_nameerror(variable)
File "/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/docassemble/base/functions.py", line 4219, in force_ask_nameerror
raise DANameError("name '" + str(variable_name) + "' is not defined")
docassemble.base.error.DANameError: name 'al_race_ethnicity_choices' is not defined


Copy link
Contributor

@BryceStevenWilley BryceStevenWilley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly should address the translation issues in this PR

Starting to get worried about the pace that copilot is just adding really verbose and wordy code, but trying not to be a blocker.

choices = choices()

other_input = {
"label": "Please specify",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For translation, I think the best option is to pass in templates for this sort of UI text?

It is really cumbersome to do though, but I know you've said that translation with excel is easier than the words.xml.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by pass in templates? do you mean make the parameter a template: ... block? I don't think that's very helpful because you need to consistently use the same parameter with every call to the method. That doesn't make sense if there's just one standard phrasing that isn't likely to change.

Good catch on this not being translatable though--I will make it consistent with other places where we use an attribute of the object.

if maxlengths:
for field in fields:
if field["field"] in maxlengths:
field["maxlength"] = maxlengths[field["field"]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does docassemble error when you put a maxlength of a non-text field? The only maxlength field here is the other option, right?

{"She/her/hers": "she/her/hers"},
{"They/them/theirs": "they/them/theirs"},
]
if variable_name == "al_race_ethnicity_choices":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the demographic questions are being tested in a different class, can these extra default definitions be removed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working separately on the tests; having CoPilot generate wasn't helpful. I might remove them.

@nonprofittechy
Copy link
Member

nonprofittechy commented Sep 19, 2025

Mainly should address the translation issues in this PR

Starting to get worried about the pace that copilot is just adding really verbose and wordy code, but trying not to be a blocker.

Good feedback - I'm in a testing phase now. I think we can use the instruction file to help with that; also I'm happy to just close a PR that has too many issues and try manually later.

I'm mostly trying to use CoPilot to take care of very small and easy to read PRs or things that were on our to-do list for a very long time. Nothing where giving a review should be a blocker--if anything, reviewing within a week would be way faster than waiting months/years for the fix to get added manually.

You should not feel obligated to review [any PR adding a new feature] quickly--if you want a chance to review before merging, I'm happy with a delayed review pace [and I can slow down my merging pace].

@BryceStevenWilley
Copy link
Contributor

You should not feel obligated to review [any PR adding a new feature] quickly--if you want a chance to review before merging, I'm happy with a delayed review pace [and I can slow down my merging pace].

I would appreciate slowing down the merging pace, getting back on GitHub and seeing like 10 PRs merged was a bit of a jump scare, especially after glancing through the first few in the docs repo and seeing some pretty noticeable errors.

But the PRs to the code repos haven't been as bad, I'm guessing because you're working through them more. I do get a little nervous when the PRs are more than 100 lines or so, but I guess for some of these features it'll need to be.

@nonprofittechy
Copy link
Member

I don't think that was a copilot error although I could look at the git blame. I think it was my manual change (copilot does run the build step and actually shows screenshots, so it would normally do better).

Mostly what I'm seeing is it going too far with changes or adding irrelevant sources, not typographical or syntax errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add questions to library for additional demographic data

3 participants