Race between layer and Lambda update

[achave11-ucsc]

The example below is from anvildev, however the same errors occurred on dev and anvilprod.

During the deploy job for the merge commit of PR #5909 (which updated the elasticsearch client from 7.10.1 to 7.17.9), an UnsupportedProductError occurred for both the indexercachehealth and servicecachehealth lambdas.

CloudWatch Insights logs:

[ERROR] UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
Traceback (most recent call last):
  File "/var/task/azul/chalice.py", line 166, in patched_event_source_handler
    return old_handler(self_, event, context)
  File "/var/task/chalice/app.py", line 1756, in __call__
    return self.handler(event_obj)
  File "/var/task/app.py", line 212, in update_health_cache
    app.health_controller.update_cache()
  File "/var/task/azul/health.py", line 138, in update_cache
    health_object = dict(time=time.time(), health=self._health.as_json_fast())
  File "/var/task/azul/health.py", line 308, in as_json_fast
    return self.as_json(p.key for p in self.fast_properties[self.lambda_name])
  File "/var/task/azul/health.py", line 181, in as_json
    json = {k: getattr(self, k) for k in sorted(keys)}
  File "/var/task/azul/health.py", line 181, in <dictcomp>
    json = {k: getattr(self, k) for k in sorted(keys)}
  File "/var/task/azul/health.py", line 73, in __get__
    return super().__get__(obj, objtype=objtype)
  File "/var/task/azul/caching.py", line 189, in __get__
    value = obj.__dict__[self.fget.__name__] = self.fget(obj)
  File "/var/task/azul/health.py", line 273, in elasticsearch
    'up': ESClientFactory.get().ping(),
  File "/opt/python/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/python/elasticsearch/client/__init__.py", line 280, in ping
    return self.transport.perform_request(
  File "/opt/python/elasticsearch/transport.py", line 421, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/opt/python/elasticsearch/transport.py", line 638, in raise_error
    raise UnsupportedProductError(message)

[achave11-ucsc]

Assignee to populate description with symptoms.

[achave11-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

Turns out that the function code and the layer are updated with different API actions:

The screenshot is of a spreadsheet that was imported from CloudWatch Insights rsults and massaged (the @timestamp of trail events is not the event time). The query used was

fields eventTime, @timestamp, eventName, requestParameters.functionName, @message
| filter @message like /ERROR|Task|INIT_START/ or eventSource = 'lambda.amazonaws.com'
| limit 1000

This shows that the errors occurred after the layer was updated with UpdateFunctionConfiguration20150331v2 and before the function code was updated with UpdateFunctionCode20150331v2. During that time, the new ES client library was used by the old code that didn't contain the monkey patch for disabling the server version check in the ES client library.

[hannes-ucsc]

Spike to try publish. Use the trail to show the API actions used by Terraform. It should include a call to PublishVersion after UpdateFunctionConfiguration and UpdateFunctionCode.