Integrate Google cloud hosted Pubsub

[rupeshpoojary]

Hi team,
we are integrating feature-hub into our ecosystem for all our product line, we wanted to integrate the google cloud hosted pub-sub into the feature-hub(hem chart) which we have deployed into our GKE cluster, currently the official documentation shows the implementation of a docker-compose based solution which has a pub-sub emulator as a container and passing the details of the pub-sub in the environment for edge and MR.

How do i integrate/pass the details for pub-sub that is hosted in GCP, Please help me how do i integrate it to connect to google cloud pub-sub.

[rvowles]

Hiya @rupeshpoojary !

The docker-compose is basically what you need to function - it tells FeatureHub that it should be using Pubsub - but you will need your infrastructure (terraform, pulumi, crossplane, etc) to actually create the topics and subscriptions before hand. The only issue is that you need to ensure that it is allowed to create subscriptions because each Edge instance needs to create a brand new subscriber to the topic coming out of the Management Repository.

If we look at the config in the docker stuff, there is some things you would never use, e.g. cloudevents.pubsub.local.* is never used in "real" GCP because these exist to connect to the local pubsub and create the topics/subscriptions and pair them together. The topics listed in "cloudevents.pubsub.local.topics" you need to create in your infrastructure code, the "cloudevents.pubsub.local.sub-pairs" holds subcriber/topic pairs your infrastructure code needs to attach.

All of the rest of the configuration indicates what names are going to be used, so your infrastructure code will need to feed these into your values.yaml file in the k8s chart.

We use GCP Pub/Sub in our SaaS version and these are the settings we use as well.

# this indicates it will use pubsub
cloudevents.pubsub.enabled=true
# this is your GCP project name
cloudevents.pubsub.project=featurehub
cloudevents.pubsub.min-backoff-delay-seconds=20

# this is a comma separated list of keys, it will take each key and look for cloudevents.enricher.pubsub.channel.<key> to find the name of the subscription it needs to publish on (it expects a topic)
cloudevents.enricher.pubsub.channels=webhooks
cloudevents.enricher.pubsub.channel.webhooks=featurehub-enriched-events-webhooks
# this is "dacha2" pubsub has to have a subscription to listen for the original feature update coming from MR, it can't use the original name
cloudevents.enricher.pubsub.subscription-name=enricher-updates-sub

# this is "edge" - listening to the cloudevents.enricher.channel-name
cloudevents.enricher.pubsub.enriched-subscription-name=featurehub-enriched-events

cloudevents.edge-mr.pubsub.topic-name=featurehub-edge-updates
cloudevents.mr-edge.pubsub.topic-name=featurehub-mr-edge
# featurehub must dynamically create subscribers for Edge and Dacha2 as PubSub is not actually... PubSub
cloudevents.mr-edge.pubsub.subscription-prefix=featurehub-edge-listener
cloudevents.stats.pubsub.topic-name=featurehub-stats
cloudevents.mr-dacha2.pubsub.topic-name=featurehub-mr-dacha2
cloudevents.mr-dacha2.pubsub.subscription-prefix=featurehub-dacha2-listener

# don't use any of this in your configuration, however it informs you of how you need to connect them together
# start Google PubSub emulator config
# this is internal FeatureHub code that pre-creates these topics on the emulator, it shouldn't be used in production code
cloudevents.pubsub.local.host=pubsub:8075
cloudevents.pubsub.local.topics=featurehub-mr-dacha2,featurehub-mr-edge,featurehub-edge-updates,featurehub-stats,featurehub-enriched-events-webhooks,enriched-feature-data

# dacha2 and edge make their own subscribers
cloudevents.pubsub.local.sub-pairs=featurehub-edge-updates-mr-sub=featurehub-edge-updates,featurehub-enriched-events=featurehub-enriched-events-webhooks,enricher-updates-sub=featurehub-mr-edge
# end Google PubSub emulator config

[rvowles]

I presume you are also familiar with the need for registering the workload identity against the service account the helm chart is creating?

[rupeshpoojary]

Hi Richard, Thankyou for your prompt reply, We have already Created the Topics on pubsub gcloud , also the nodes on which the helm chart is deployed has the Permission Role attached to it. can you help me with "cloudevents.enricher.pubsub.channels=webhooks" what is this channel ? and I am not familiar with the registering of workload, can you Please help our here ? and the config you sent above , where should that be put ?

…

On Fri, Jun 9, 2023 at 2:18 PM Richard Vowles ***@***.***> wrote: I presume you are also familiar with the need for registering the workload identity against the service account the helm chart is creating? — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BAOCIM6VAK25D4GTRNIWVALXKLPNFANCNFSM6AAAAAAZAII43E> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This email and any attachments may contain confidential, proprietary, or legally privileged information and are for the intended recipient only. Any other use of emails, such as unauthorized access, disclosure, copying, distribution, or reliance on any contents by anyone else, is prohibited and unlawful. If you are not the intended recipient, please notify the sender immediately and delete the email

[rupeshpoojary]

Hi Richard, I have been coordinating with Nilesh from the Reliance team, we are continuously working with the Nilesh and team for integrating this featurehub into our fynd ecosystem. Could you add me into the slack channel of feature hub? i am currently not able to join On Fri, Jun 9, 2023 at 3:19 PM Rupesh Poojary ***@***.***> wrote:

…

Hi Richard, Thankyou for your prompt reply, We have already Created the Topics on pubsub gcloud , also the nodes on which the helm chart is deployed has the Permission Role attached to it. can you help me with "cloudevents.enricher.pubsub.channels=webhooks" what is this channel ? and I am not familiar with the registering of workload, can you Please help our here ? and the config you sent above , where should that be put ? On Fri, Jun 9, 2023 at 2:18 PM Richard Vowles ***@***.***> wrote: > I presume you are also familiar with the need for registering the > workload identity against the service account the helm chart is creating? > > — > Reply to this email directly, view it on GitHub > <#49 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BAOCIM6VAK25D4GTRNIWVALXKLPNFANCNFSM6AAAAAAZAII43E> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

-- This email and any attachments may contain confidential, proprietary, or legally privileged information and are for the intended recipient only. Any other use of emails, such as unauthorized access, disclosure, copying, distribution, or reliance on any contents by anyone else, is prohibited and unlawful. If you are not the intended recipient, please notify the sender immediately and delete the email

[rvowles]

Heya @rupeshpoojary - our Slack is here: https://join.slack.com/t/anyways-labs/shared_invite/zt-frxdx34x-ODs_XmLh6BCvqiNeBRx0hA