opencl memory errors during fits (Trac #1015)

[pkienzle]

The following OpenCL memory error was reported from !SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.

2017-10-23 17:39:12 : Traceback (most recent call last):
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
 File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit
 File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory
 File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate
 File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals
 File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in __call__
 File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/__init__.py", line 1088, in kernel_call
 File "<generated function enqueue_knl_cylinder_Iq>", line 169, in enqueue_knl_cylinder_Iq
LogicError: when processing argument #233 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

=== Steps: 161 of 159  chisq: 70.4  ETA: -86s
  M2.p2_radius: 71.5       | M2.p2_sld_shell: -0.108     | M2.p2_thickness: 75.8      
   M1.p1_scale: 0.000499   | M1.p1_sld_solvent: 1.38       |    M1.p2_radius: 136       
M1.p2_radius.width: 0.851      |     M1.p2_scale: 0.0181     |  M1.p2_sld_core: 2.08      
M1.p2_thickness: 14.1       |    M4.p2_radius: 64.8       | M4.p2_sld_shell: -0.0451   
M4.p2_thickness: 79.6       |    M3.p2_radius: 69         | M3.p2_sld_shell: -0.199    
M3.p2_thickness: 83.7       | M3.p2_thickness.width: 0.794      |  
2017-10-23 17:42:59 : Traceback (most recent call last):
 File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run
LogicError: when processing argument #233 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

	2017-10-23 17:43:49 : ('   0.357', '   0.697')
	2017-10-23 17:43:50 : ('   0.357', '   0.697')
	2017-10-23 17:43:51 : ('   0.505', '      15')
	2017-10-23 17:43:51 : ('   0.505', '      15')
	2017-10-23 17:43:55 : ('  0.0847', '   0.268')
	2017-10-23 17:43:55 : ('  0.0847', '   0.268')
	2017-10-23 17:45:05 : Computation  completed!
	2017-10-23 17:45:07 : Computation  completed!
	2017-10-23 17:45:20 : Computation  completed!
	2017-10-23 17:45:21 : Computation  completed!
	2017-10-23 17:45:22 : Computation  completed!
	2017-10-23 17:45:25 : Computation  completed!
	2017-10-23 17:45:26 : Computation  completed!
	2017-10-23 17:45:27 : Computation  completed!
	2017-10-23 17:45:29 : Computation  completed!
	2017-10-23 17:45:40 : Computation  completed!
	2017-10-23 17:45:45 : Computation  completed!
	2017-10-23 17:46:15 : Computation  completed!
	2017-10-23 17:48:42 : Fitting is in progress...
	2017-10-23 17:48:42 : Fitting is in progress...
	2017-10-23 17:48:42 : Traceback (most recent call last):
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
 File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
 File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit
 File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps
AttributeError: 'FitDriver' object has no attribute 'fitter'

None 

Migrated from http://trac.sasview.org/ticket/1015

{
    "status": "new",
    "changetime": "2019-03-29T00:07:14",
    "_ts": "2019-03-29 00:07:14.288941+00:00",
    "description": "The following OpenCL memory error was reported from !SasView.  This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits.  Restructuring the calculator so that the q values are held for the life of the model may fix the error.\n\n{{{\n2017-10-23 17:39:12 : Traceback (most recent call last):\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 82, in compute\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 14, in map_apply\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 11, in map_getattr\n File \"sas/sascalc/fit/BumpsFitting.pyc\", line 289, in fit\n File \"sas/sascalc/fit/BumpsFitting.pyc\", line 202, in theory\n File \"sas/sascalc/fit/BumpsFitting.pyc\", line 192, in _recalculate\n File \"sas/sascalc/fit/AbstractFitEngine.pyc\", line 242, in residuals\n File \"/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py\", line 272, in evalDistribution\n File \"/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py\", line 582, in evalDistribution\n File \"/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py\", line 617, in calculate_Iq\n File \"/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py\", line 638, in _calculate_Iq\n File \"/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py\", line 567, in __call__\n File \"/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/__init__.py\", line 1088, in kernel_call\n File \"<generated function enqueue_knl_cylinder_Iq>\", line 169, in enqueue_knl_cylinder_Iq\nLogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT\n\n=== Steps: 161 of 159  chisq: 70.4  ETA: -86s\n  M2.p2_radius: 71.5       | M2.p2_sld_shell: -0.108     | M2.p2_thickness: 75.8      \n   M1.p1_scale: 0.000499   | M1.p1_sld_solvent: 1.38       |    M1.p2_radius: 136       \nM1.p2_radius.width: 0.851      |     M1.p2_scale: 0.0181     |  M1.p2_sld_core: 2.08      \nM1.p2_thickness: 14.1       |    M4.p2_radius: 64.8       | M4.p2_sld_shell: -0.0451   \nM4.p2_thickness: 79.6       |    M3.p2_radius: 69         | M3.p2_sld_shell: -0.199    \nM3.p2_thickness: 83.7       | M3.p2_thickness.width: 0.794      |  \n2017-10-23 17:42:59 : Traceback (most recent call last):\n File \"sas/sascalc/data_util/calcthread.pyc\", line 268, in _run\nLogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT\n\n\t2017-10-23 17:43:49 : ('   0.357', '   0.697')\n\t2017-10-23 17:43:50 : ('   0.357', '   0.697')\n\t2017-10-23 17:43:51 : ('   0.505', '      15')\n\t2017-10-23 17:43:51 : ('   0.505', '      15')\n\t2017-10-23 17:43:55 : ('  0.0847', '   0.268')\n\t2017-10-23 17:43:55 : ('  0.0847', '   0.268')\n\t2017-10-23 17:45:05 : Computation  completed!\n\t2017-10-23 17:45:07 : Computation  completed!\n\t2017-10-23 17:45:20 : Computation  completed!\n\t2017-10-23 17:45:21 : Computation  completed!\n\t2017-10-23 17:45:22 : Computation  completed!\n\t2017-10-23 17:45:25 : Computation  completed!\n\t2017-10-23 17:45:26 : Computation  completed!\n\t2017-10-23 17:45:27 : Computation  completed!\n\t2017-10-23 17:45:29 : Computation  completed!\n\t2017-10-23 17:45:40 : Computation  completed!\n\t2017-10-23 17:45:45 : Computation  completed!\n\t2017-10-23 17:46:15 : Computation  completed!\n\t2017-10-23 17:48:42 : Fitting is in progress...\n\t2017-10-23 17:48:42 : Fitting is in progress...\n\t2017-10-23 17:48:42 : Traceback (most recent call last):\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 82, in compute\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 14, in map_apply\n File \"sas/sasgui/perspectives/fitting/fit_thread.pyc\", line 11, in map_getattr\n File \"sas/sascalc/fit/BumpsFitting.pyc\", line 275, in fit\n File \"sas/sascalc/fit/BumpsFitting.pyc\", line 375, in run_bumps\nAttributeError: 'FitDriver' object has no attribute 'fitter'\n\nNone \n}}}",
    "reporter": "pkienzle",
    "cc": "",
    "resolution": "",
    "workpackage": "SasView Bug Fixing",
    "time": "2017-10-24T20:05:10",
    "component": "SasView",
    "summary": "opencl memory errors during fits",
    "priority": "major",
    "keywords": "",
    "milestone": "SasView 4.3.0",
    "owner": "",
    "type": "defect"
}

[butlerpd]

Trac update at 2017/10/27 10:09:33: butler changed description from:

The following OpenCL memory error was reported from SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.

{{{
2017-10-23 17:39:12 : Traceback (most recent call last):
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit
File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory
File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate
File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals
File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in call
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/init.py", line 1088, in kernel_call
File "", line 169, in enqueue_knl_cylinder_Iq
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

=== Steps: 161 of 159 chisq: 70.4 ETA: -86s
M2.p2_radius: 71.5 | M2.p2_sld_shell: -0.108 | M2.p2_thickness: 75.8
M1.p1_scale: 0.000499 | M1.p1_sld_solvent: 1.38 | M1.p2_radius: 136
M1.p2_radius.width: 0.851 | M1.p2_scale: 0.0181 | M1.p2_sld_core: 2.08
M1.p2_thickness: 14.1 | M4.p2_radius: 64.8 | M4.p2_sld_shell: -0.0451
M4.p2_thickness: 79.6 | M3.p2_radius: 69 | M3.p2_sld_shell: -0.199
M3.p2_thickness: 83.7 | M3.p2_thickness.width: 0.794 |
2017-10-23 17:42:59 : Traceback (most recent call last):
File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

2017-10-23 17:43:49 : (' 0.357', ' 0.697')
2017-10-23 17:43:50 : (' 0.357', ' 0.697')
2017-10-23 17:43:51 : (' 0.505', ' 15')
2017-10-23 17:43:51 : (' 0.505', ' 15')
2017-10-23 17:43:55 : (' 0.0847', ' 0.268')
2017-10-23 17:43:55 : (' 0.0847', ' 0.268')
2017-10-23 17:45:05 : Computation completed!
2017-10-23 17:45:07 : Computation completed!
2017-10-23 17:45:20 : Computation completed!
2017-10-23 17:45:21 : Computation completed!
2017-10-23 17:45:22 : Computation completed!
2017-10-23 17:45:25 : Computation completed!
2017-10-23 17:45:26 : Computation completed!
2017-10-23 17:45:27 : Computation completed!
2017-10-23 17:45:29 : Computation completed!
2017-10-23 17:45:40 : Computation completed!
2017-10-23 17:45:45 : Computation completed!
2017-10-23 17:46:15 : Computation completed!
2017-10-23 17:48:42 : Fitting is in progress...
2017-10-23 17:48:42 : Fitting is in progress...
2017-10-23 17:48:42 : Traceback (most recent call last):
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit
File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps
AttributeError: 'FitDriver' object has no attribute 'fitter'

None
}}}

to:

The following OpenCL memory error was reported from !SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.

{{{
2017-10-23 17:39:12 : Traceback (most recent call last):
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit
File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory
File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate
File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals
File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in call
File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/init.py", line 1088, in kernel_call
File "", line 169, in enqueue_knl_cylinder_Iq
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

=== Steps: 161 of 159 chisq: 70.4 ETA: -86s
M2.p2_radius: 71.5 | M2.p2_sld_shell: -0.108 | M2.p2_thickness: 75.8
M1.p1_scale: 0.000499 | M1.p1_sld_solvent: 1.38 | M1.p2_radius: 136
M1.p2_radius.width: 0.851 | M1.p2_scale: 0.0181 | M1.p2_sld_core: 2.08
M1.p2_thickness: 14.1 | M4.p2_radius: 64.8 | M4.p2_sld_shell: -0.0451
M4.p2_thickness: 79.6 | M3.p2_radius: 69 | M3.p2_sld_shell: -0.199
M3.p2_thickness: 83.7 | M3.p2_thickness.width: 0.794 |
2017-10-23 17:42:59 : Traceback (most recent call last):
File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run
LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

2017-10-23 17:43:49 : (' 0.357', ' 0.697')
2017-10-23 17:43:50 : (' 0.357', ' 0.697')
2017-10-23 17:43:51 : (' 0.505', ' 15')
2017-10-23 17:43:51 : (' 0.505', ' 15')
2017-10-23 17:43:55 : (' 0.0847', ' 0.268')
2017-10-23 17:43:55 : (' 0.0847', ' 0.268')
2017-10-23 17:45:05 : Computation completed!
2017-10-23 17:45:07 : Computation completed!
2017-10-23 17:45:20 : Computation completed!
2017-10-23 17:45:21 : Computation completed!
2017-10-23 17:45:22 : Computation completed!
2017-10-23 17:45:25 : Computation completed!
2017-10-23 17:45:26 : Computation completed!
2017-10-23 17:45:27 : Computation completed!
2017-10-23 17:45:29 : Computation completed!
2017-10-23 17:45:40 : Computation completed!
2017-10-23 17:45:45 : Computation completed!
2017-10-23 17:46:15 : Computation completed!
2017-10-23 17:48:42 : Fitting is in progress...
2017-10-23 17:48:42 : Fitting is in progress...
2017-10-23 17:48:42 : Traceback (most recent call last):
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply
File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr
File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit
File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps
AttributeError: 'FitDriver' object has no attribute 'fitter'

None
}}}

[butlerpd]

Trac update at 2017/10/27 16:17:27: butler changed milestone from "SasView 4.2.0" to "SasView 4.3.0"

[pkienzle]

Trac update at 2018/10/23 21:45:02: pkienzle commented:

The resulting error occurs also when a model is displayed on the fit page then you test a new OpenCL device:

LogicError: when processing argument #233 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT

In this case, I believe it is because the program was compiled in a different OpenCL context than the one used to allocate the q vectors. We need to keep all references to the buffers and kernels in the context object and retrieve them from when running. That way when the context is reset we can force a new set of buffers to be allocated and recompile the kernels.

Note: may want to store the compiled model as a class variable in the SasviewModel wrapper so that it doesn't get recompiled each time you select a model on the fit page. At the top of SasviewModel._calculate_Iq use self.__class__._model = core.build_model(self._model_info). The model info attribute is already stored at the class level in _generate_model_attributes, but the build model step was being done lazily.

[pkienzle]

Trac update at 2018/10/24 14:23:28: pkienzle commented:

The work-around is to tell the user to reselect the model.

[wpotrzebowski]

Trac update at 2019/03/28 14:56:34: wojciech commented:

I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.

[pkienzle]

Trac update at 2019/03/29 00:07:14: pkienzle commented:

Replying to [comment:5 wojciech]:

I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.

Are you working from sasmodels/master or a tagged revision?

There was some additional cleanup that didn't get merged until the beta_approx branch was merged, though it's been too long for me to remember the details.