update_obsdb commits too often

[mhasself]

It seems like "add_obs_columns" calls in the site pipeline script create excess sqlite traffic, even though they do nothing almost all of the time. Ideally we'd make a single update per obs -- and actually we should even consider batching those.

This is conflicting with obsdb consumers -- so general goal here is to block for write less often.

[mmccrackan]

Following up on this, but I realized those calls don't actually get run as I originally thought since the obs columns already exist, although the actual source of the problem is similar. Looking at the tracebacks, it occurs on the commit of the obs info here:

sotodlib/sotodlib/site_pipeline/update_obsdb.py

Line 381 in d133f84

bookcartobsdb.update_obs(obs_id, very_clean, tags=tags)

I have managed to reproduce this consistently by doing an obsdb query in jupyter while update_obsdb is running. If I interrupt that query, it seems to fail the flow. I also managed to cause this on SATp3 by doing the same. I need to do more testing to see if it is only that interruption that causes this or if it is any obsdb query at all (I think it might be the latter but not consistently since doing the former may keep things locked).

For now I think we can:

• Use Josh's profiler to look for improvements to speed this up so it isn't running for hours.
• Increase the obsdb sqlite timeout time.
• Wrap the above line in a try/except so it either retries or just keeps going (I think the latter in conjunction with a longer timeout is probably best).