add cuopt direct solver

[Iroy30]

Fixes #3626

Summary/Motivation:

Add cuOpt math optimization (includes LP and MILP) solver backend to Pyomo so users can solve pyomo models with cuOpt

Changes proposed in this PR:

• Add cuopt_direct solver plugin

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[codecov]

Codecov Report

❌ Patch coverage is 77.18631% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.38%. Comparing base (13facca) to head (0e598ee).

Files with missing lines	Patch %	Lines
pyomo/solvers/plugins/solvers/cuopt_direct.py	77.01%	60 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3620      +/-   ##
==========================================
- Coverage   89.40%   89.38%   -0.03%     
==========================================
  Files         909      910       +1     
  Lines      105541   105804     +263     
==========================================
+ Hits        94364    94574     +210     
- Misses      11177    11230      +53     

Flag	Coverage Δ
builders	29.09% <19.77%> (-0.02%)	⬇️
default	86.00% <77.18%> (?)
expensive	35.72% <20.15%> (?)
linux	86.55% <19.01%> (-2.63%)	⬇️
linux_other	86.55% <19.01%> (-0.18%)	⬇️
osx	82.71% <19.01%> (-0.17%)	⬇️
win	84.82% <19.01%> (-0.16%)	⬇️
win_other	84.82% <19.01%> (-0.16%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mrmundt]

We are working on making cuopt available in our testing infrastructure; can you please add tests to this PR?

[mrmundt]

@Iroy30 - We've been able to make cuopt available on our internal testing machines. Can you please add tests to this PR?

[Iroy30]

@mrmundt Thanks! We have added tests by enabling testing cuopt with LP and MILP capabilities in tests/solvers.py. Let us know if:

1. There is any other testing that needs to be enabled.
2. For testing cuOpt backend in CI, it needs a GPU, is that something pyomo team intends to add or will it be disabled in github CI?

The following is the testing output I get relevant to cuOpt

Testing scenario generation
------------------------------
LP_block, cuopt, python
   ok: 
LP_block_kernel, cuopt, python
   ok: 
LP_compiled, cuopt, python
   ok: 
LP_compiled_dense_kernel, cuopt, python
   ok: 
LP_compiled_sparse_kernel, cuopt, python
   ok: 
LP_constant_objective1, cuopt, python
   ok: 
LP_constant_objective1_kernel, cuopt, python
   ok: 
LP_constant_objective2, cuopt, python
   ok: 
LP_constant_objective2_kernel, cuopt, python
   ok: 
LP_duals_maximize, cuopt, python
   ok: 
LP_duals_maximize_kernel, cuopt, python
   ok: 
LP_duals_minimize, cuopt, python
   ok: 
LP_duals_minimize_kernel, cuopt, python
   ok: 
LP_inactive_index, cuopt, python
   ok: 
LP_inactive_index_kernel, cuopt, python
   ok: 
LP_infeasible1, cuopt, python
   ok: 
LP_infeasible1_kernel, cuopt, python
   ok: 
LP_infeasible2, cuopt, python
   ok: 
LP_infeasible2_kernel, cuopt, python
   ok: 
LP_piecewise, cuopt, python
   ok: 
LP_piecewise_nosuffixes, cuopt, python
   ok: 
LP_simple, cuopt, python
   ok: 
LP_simple_kernel, cuopt, python
   ok: 
LP_trivial_constraints, cuopt, python
   ok: 
LP_trivial_constraints_kernel, cuopt, python
   ok: 
LP_unbounded, cuopt, python
   ok: 
LP_unbounded_kernel, cuopt, python
   ok: 
LP_unused_vars, cuopt, python
   ok: 
LP_unused_vars_kernel, cuopt, python
   ok: 
MILP_discrete_var_bounds, cuopt, python
   ok: 
MILP_discrete_var_bounds_kernel, cuopt, python
   ok: 
MILP_infeasible1, cuopt, python
   ok: 
MILP_infeasible1_kernel, cuopt, python
   ok: 
MILP_simple, cuopt, python
   ok: 
MILP_simple_kernel, cuopt, python
   ok: 
MILP_unbounded, cuopt, python
   ok: 
MILP_unbounded_kernel, cuopt, python
   ok: 
MILP_unused_vars, cuopt, python
   ok: 
MILP_unused_vars_kernel, cuopt, python
   ok: 

[mrmundt]

This is fine, but just so you're aware, we have this lovely little utility called TicTocTimer that you may want to consider using: https://pyomo.readthedocs.io/en/latest/api/pyomo.common.timing.TicTocTimer.html

[whart222]

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

[mrmundt]

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@whart222 - I am evenly split. Because we are still messing with what the new solver interfaces are going to actually do / how they will handle input and present output, I don't know if we want to put "new" solvers there or just "well-established" ones that we can robustly test / really know what they are supposed to do and return.

[mrmundt]

@Iroy30 - I forgot to post this last week, but all of the failures are of the variety:

self = <pyomo.solvers.tests.checks.test_no_solution_behavior.Test_LP_unbounded_kernel testMethod=test_cuopt_python>

    def return_test(self):
>       return failed_solve_test(self)

/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:61: in failed_solve_test
    opt, results = model_class.solve(
/tester/python/python39/pyomo/pyomo/solvers/tests/models/base.py:123: in solve
    results = opt.solve(
/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:99: in solve
    self._solver_error('solve')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.opt.base.solvers.UnknownSolver object at 0x7f86ad9ab040>
method_name = 'solve'

        def _solver_error(self, method_name):
>           raise RuntimeError(
                """Attempting to use an unavailable solver.
    
    The SolverFactory was unable to create the solver "%s"
    and returned an UnknownSolver object.  This error is raised at the point
    where the UnknownSolver object was used as if it were valid (by calling
    method "%s").
    
    The original solver was created with the following parameters:
    \t"""
                % (self.type, method_name)
                + "\n\t".join("%s: %s" % i for i in sorted(self._kwds.items()))
                + "\n\t_args: %s" % (self._args,)
                + "\n\toptions: %s" % (self.options,)
            )
E           RuntimeError: Attempting to use an unavailable solver.
E           
E           The SolverFactory was unable to create the solver "cuopt"
E           and returned an UnknownSolver object.  This error is raised at the point
E           where the UnknownSolver object was used as if it were valid (by calling
E           method "solve").
E           
E           The original solver was created with the following parameters:
E           	executable: cuopt
E           	solver_io: python
E           	type: cuopt
E           	_args: ()
E           	options: {}

/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:116: RuntimeError

[mrmundt]

@Iroy30 - Two more things:

1. Please run black -S -C on your files
2. Something in our testing environment isn't loading correctly. I'm looking into it.

[Iroy30]

@jsiirola Did the instructions fix the cuOpt import?

[mrmundt]

This is the problem line. If cuopt isn't actually available, then this panics. You should probably wrap this in a try block or move it into its own function, e.g.:

try:
    self._version = cuopt.__version__.split('.')
except DeferredImportError:
    self._version = None

[mrmundt]

Actually, upon chatting in the dev call more about this just now, this should probably just be moved into its own function, not at all in the init, because if it's in the init, it will import cuopt unconditionally no matter what.

[Iroy30]

Seems to run successfully now in the branch pipeline . Could we trigger CI @mrmundt

[jsiirola]

OK. We are getting close. cuopt finally appears to be running on Jenkins. We are seeing 9 failures:

FAILED pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py::Test_MILP_unbounded_kernel::test_cuopt_python - AssertionError: <TerminationCondition.infeasible: 'infeasible'> not found in (<TerminationCondition.unbounded: 'unbounded'>, <TerminationCondition.infeasibleOrUnbounded: 'infeasibleOrUnbounded'>)
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_LP_duals_maximize::test_cuopt_python_nonsymbolic_labels - AssertionError: Solution mismatch for plugin test_cuopt_python_nonsymbolic_labels, python interface and problem type LP_duals_maximize
Difference in solution for x[1].rc:
	Baseline - -1.0
	Current - 1.0
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_LP_duals_maximize::test_cuopt_python_symbolic_labels - AssertionError: Solution mismatch for plugin test_cuopt_python_symbolic_labels, python interface and problem type LP_duals_maximize
Difference in solution for x[1].rc:
	Baseline - -1.0
	Current - 1.0
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_LP_duals_maximize_kernel::test_cuopt_python_nonsymbolic_labels - AssertionError: Solution mismatch for plugin test_cuopt_python_nonsymbolic_labels, python interface and problem type LP_duals_maximize
Difference in solution for x[1].rc:
	Baseline - -1.0
	Current - 1.0
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_LP_duals_maximize_kernel::test_cuopt_python_symbolic_labels - AssertionError: Solution mismatch for plugin test_cuopt_python_symbolic_labels, python interface and problem type LP_duals_maximize
Difference in solution for x[1].rc:
	Baseline - -1.0
	Current - 1.0
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_MILP_unbounded::test_cuopt_python_nonsymbolic_labels - AssertionError: <TerminationCondition.infeasible: 'infeasible'> not found in (<TerminationCondition.unbounded: 'unbounded'>, <TerminationCondition.infeasibleOrUnbounded: 'infeasibleOrUnbounded'>)
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_MILP_unbounded::test_cuopt_python_symbolic_labels - AssertionError: <TerminationCondition.infeasible: 'infeasible'> not found in (<TerminationCondition.unbounded: 'unbounded'>, <TerminationCondition.infeasibleOrUnbounded: 'infeasibleOrUnbounded'>)
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_MILP_unbounded_kernel::test_cuopt_python_nonsymbolic_labels - AssertionError: <TerminationCondition.infeasible: 'infeasible'> not found in (<TerminationCondition.unbounded: 'unbounded'>, <TerminationCondition.infeasibleOrUnbounded: 'infeasibleOrUnbounded'>)
FAILED pyomo/pyomo/solvers/tests/checks/test_writers.py::Test_MILP_unbounded_kernel::test_cuopt_python_symbolic_labels - AssertionError: <TerminationCondition.infeasible: 'infeasible'> not found in (<TerminationCondition.unbounded: 'unbounded'>, <TerminationCondition.infeasibleOrUnbounded: 'infeasibleOrUnbounded'>)