Retrieve mapping values in custom integration

[roland-d]

Hello,

As you may know by now I am building a custom integration and making progress :) I am now at the point where I want to map the submission values to the field mapping I made in the form.

So the flow I am looking at now is:

1. Run the Formie queue manager to send out submissions with the run-integration
2. This triggers the sendPayload($submission) function
3. In the sendPayload() I want to load the field mappings with $formValues = $this->getFieldMappingValues($submission, $this->formFieldMapping, 'fields'); but this returns an empty array. If I see it correct that is because of the $this->formFieldMapping is empty.

I am stuck as I do not see how the $this->formFieldMapping is filled. I did see it tries to get it from cache but this is also empty. It is like I am missing a sort of populate function that pre-fills some data.

Here is the class as I have it now (shortened for readability):

<?php

namespace client\modules;

use Craft;
use craft\errors\InvalidFieldException;
use craft\helpers\App;
use Exception;
use fmREST;
use RuntimeException;
use Throwable;
use verbb\formie\base\Crm;
use verbb\formie\base\Integration;
use verbb\formie\elements\Submission;
use verbb\formie\models\IntegrationField;
use verbb\formie\models\IntegrationFormSettings;

class Filemaker extends Crm
{
	public string $hostName;
	public string $database;
	public string $username;
	public string $password;
	public ?array $contactFieldMapping = null;
	public ?array $dealFieldMapping = null;
	public ?array $leadFieldMapping = null;
	public ?array $companyFieldMapping = null;
	public ?array $formFieldMapping = null;
	public bool $debug = false;

	public static function displayName(): string
	{
		return Craft::t('formie', 'FileMaker');
	}

	public function getIconUrl(): string
	{
		return Craft::$app->getAssetManager()->getPublishedUrl("@client/web/assets/cp/filemaker.svg", true);
	}

	public function getDescription(): string
	{
		return Craft::t('formie', 'Provides FileMaker integration.');
	}

	public function getSettingsHtml(): string
	{
		return Craft::$app->getView()->renderTemplate('client/integrations/crm/filemaker/_plugin-settings', [
			'integration' => $this,
		]);
	}

	public function getFormSettingsHtml($form): string
	{
		return Craft::$app->getView()->renderTemplate('client/integrations/crm/filemaker/_form-settings', [
			'integration' => $this,
			'form' => $form
		]);
	}

	public function fetchFormSettings(): IntegrationFormSettings
	{
		$settings = [];

		try
		{
			if (!Craft::$app->getRequest()->getParam('sendFormPayload')) {
				throw new \InvalidArgumentException('A layout must be specified');
			}

			$settings['syncFormSuccess'] = '';
			$settings['syncFormError'] = '';
			$layout = Craft::$app->getRequest()->getParam('layout');

			if (!$layout) {
				$settings['syncFormError'] = Craft::t('formie', 'Layout required');
			}
			else
			{
				$response = $this->getFmClient()->layoutMetadata($layout);

				if ($response['messages'][0]['code'] > 0)
				{
					Integration::error($this, $response['messages'][0]['message'], true);
				}

				$filemakerFields = [];

				foreach ($response['response']['fieldMetaData'] as $field)
				{
					$filemakerFields[] =
						new IntegrationField([
							'handle' => $field['name'],
							'name' => Craft::t('formie', $field['name']),
							'required' => false,
						]);
				}
				$settings['syncFormSuccess'] = Craft::t('formie', 'Successfully synced FileMaker fields');

				$settings['fields'] = $filemakerFields;
			}
		} catch (Throwable $e) {
			Integration::apiError($this, $e);
		}

		return new IntegrationFormSettings($settings);
	}

	public function sendPayload(Submission $submission): bool
	{
		$formValues = $this->getFieldMappingValues($submission, $this->formFieldMapping, 'fields');
print_r($formValues);
		return true;
	}

	public function fetchConnection(): bool
	{
		try {
			$this->getFmClient()->productInfo();
		} catch (Throwable $e) {
			Integration::apiError($this, $e);

			return false;
		}

		return true;
	}
}

In the database I have seen that the field mapping is stored for the form but the code cannot find it for some reason.

Where am I missing something?

Thank you.

[engram-design]

So firstly, you should supply settings for the mapping screen for a particular name. You've done this in your fetchFormSettings() function (combined for brevity):

return new IntegrationFormSettings([
    'fields' => [
        new IntegrationField([
            'handle' => $field['name'],
            'name' => Craft::t('formie', $field['name']),
            'required' => false,
        ]),
    ]
]);

Here, you create a collection of IntegrationField's, grouped into fields. This could actually be called whatever you like. For example, our ActiveCampaign integration has many different objects to map to: contacts, deals and accounts. This fetchFormSettings() function returns 3 collections, one for each of these objects. They're named as such to make it easy to grab later, when we get to sending payloads.

I'd recommend you change this from fields to form for clarity. So in your code:

$settings['form'] = $filemakerFields;

Because those fields represent the settings for mapping to a "form object", just like a "contact object", deal, etc.

In sendPayload() you'll clearly want to fetch the mapping settings for the data object you're after. Calling $formValues = $this->getFieldMappingValues($submission, $this->formFieldMapping, 'fom'); should be exactly what you need.

If you run $this->formFieldMapping, you're saying that that's empty? That'd be the first issue, as it sounds like you mapping values aren't saving?

Again, looking at ActiveCampaign as an example, you should have something like this in your form template.

<integration-field-mapping
    label="{{ 'Form Field Mapping' | t('formie') }}"
    instructions="{{ 'Choose how your form fields should map to your {name} Form fields.' | t('formie', { name: integration.displayName() }) }}"
    name-label="{{ integration.displayName() }}"
    id="form-field-mapping"
    name="formFieldMapping"
    :value="get(form, 'settings.integrations.{{ handle }}.formFieldMapping')"
    :rows="get(settings, 'form')"
></integration-field-mapping>

Notice the name="formFieldMapping" which saves the settings of your mapping to the formFieldMapping attribute in your class.

Can you let me know the template side of things, and maybe adjust from fields to form?

[roland-d]

If you run $this->formFieldMapping, you're saying that that's empty? That'd be the first issue, as it sounds like you mapping values aren't saving?

Correct, but my mappings are stored in the database and I see them in the UI.

This is what I have in my form template after changing it per your suggestion, I was missing the word form in the id, name and value:

<integration-field-mapping
	label="{{ 'Field Mapping' | t('formie') }}"
	instructions="{{ 'Choose how your form fields should map to your {name} fields.' | t('formie', { name: integration.displayName() }) }}"
	name-label="{{ integration.displayName() }}"
	id="form-field-mapping"
	name="formFieldMapping"
	:value="get(form, 'settings.integrations.{{ handle }}.formFieldMapping')"
	:rows="get(settings, 'fields')"
></integration-field-mapping>

This is how I run the command line:

 php craft formie/submissions/run-integration --integration=filemaker --submission-id=102

When I check the value of $this->formFieldMapping it is NULL.

Not sure if it is relevant but I must also declare these properties otherwise the page errors out with an unknown property error:

	public ?array $contactFieldMapping = null;
	public ?array $dealFieldMapping = null;
	public ?array $leadFieldMapping = null;

Let me know what else I can supply to solve this mystery. Thanks.

[engram-design]

Ah yes, the run-integration command is actually brand-new, but there's a bug with that. Fixed for the next release. To get this early, change your verbb/formie requirement in composer.json to:

"require": {
  "verbb/formie": "dev-craft-4 as 2.0.4",
  "...": "..."
}

Then run composer update.

[roland-d]

I have visual 😄

Glad to know it wasn't just me 🙃

Thank you for the fix, I can confirm it works for me. Now I can implement the rest.

[roland-d]

I would like to revisit this somewhat because I have noticed a possible side-effect.

Now when I refresh the list of fields on the integration tab, the table is no longer populated. When I click the refresh button I am getting this JSON response:

{"syncFormSuccess":"Successfully synced FileMaker fields","syncFormError":"","form":[{"handle":"db_accountcreate","name":"db_accountcreate","type":null,"required":"","options":[]},{"handle":"db_accountmod","name":"db_accountmod","type":null,"required":"","options":[]},{"handle":"db_datecreate","name":"db_datecreate","type":null,"required":"","options":[]},{"handle":"db_datemod","name":"db_datemod","type":null,"required":"","options":[]},{"handle":"db_linkall","name":"db_linkall","type":null,"required":"","options":[]},{"handle":"db_recid","name":"db_recid","type":null,"required":"","options":[]},{"handle":"db_usercreate","name":"db_usercreate","type":null,"required":"","options":[]},{"handle":"db_usermod","name":"db_usermod","type":null,"required":"","options":[]},{"handle":"db_inlog","name":"db_inlog","type":null,"required":"","options":[]},{"handle":"Username","name":"Username","type":null,"required":"","options":[]},{"handle":"UserId","name":"UserId","type":null,"required":"","options":[]},{"handle":"naam","name":"naam","type":null,"required":"","options":[]},{"handle":"UserIp","name":"UserIp","type":null,"required":"","options":[]},{"handle":"db_status","name":"db_status","type":null,"required":"","options":[]},{"handle":"BARCODE","name":"BARCODE","type":null,"required":"","options":[]},{"handle":"aanvraag_voor","name":"aanvraag_voor","type":null,"required":"","options":[]},{"handle":"school_adres","name":"school_adres","type":null,"required":"","options":[]},{"handle":"school_naam","name":"school_naam","type":null,"required":"","options":[]}]}

That all looks good, except the UI is not updated.

In my _form-settings.html I have this integration field:

<integration-field-mapping
	label="{{ 'Field Mapping' | t('formie') }}"
	instructions="{{ 'Choose how your form fields should map to your {name} fields.' | t('formie', { name: integration.displayName() }) }}"
	name-label="{{ integration.displayName() }}"
	id="form-field-mapping"
	name="formFieldMapping"
	:value="get(form, 'settings.integrations.{{ handle }}.formFieldMapping')"
	:rows="get(settings, 'fields')"
></integration-field-mapping>

My fetchFormSettings function looks like this:

public ?array $contactFieldMapping = null;
public ?array $dealFieldMapping = null;
public ?array $leadFieldMapping = null;
public ?array $formFieldMapping = null;

public function fetchFormSettings(): IntegrationFormSettings
	{
		$settings = [];

		try
		{
			if (!Craft::$app->getRequest()->getParam('sendFormPayload')) {
				throw new IntegrationException('A layout must be specified');
			}

			$settings['syncFormSuccess'] = '';
			$settings['syncFormError'] = '';
			$layout = Craft::$app->getRequest()->getParam('layout');

			if (!$layout) {
				$settings['syncFormError'] = Craft::t('formie', 'Layout required');
			}
			else
			{
				$response = $this->getFmClient()->layoutMetadata($layout);

				if ((int) $response['messages'][0]['code'] !== 0)
				{
					Integration::error($this, $response['messages'][0]['message'], true);
				}

				$filemakerFields = [];

				foreach ($response['response']['fieldMetaData'] as $field)
				{
					$filemakerFields[] =
						new IntegrationField([
							'handle' => $field['name'],
							'name' => Craft::t('formie', $field['name']),
							'required' => false,
						]);
				}
				$settings['syncFormSuccess'] = Craft::t('formie', 'Successfully synced FileMaker fields');

				$settings['form'] = $filemakerFields;
			}
		} catch (Throwable $e) {
			Integration::apiError($this, $e);
		}

		return new IntegrationFormSettings($settings);
	}

I see no JavaScript errors in my console.
Has something fallen over or do I need to change something else? Do I need to provide more info?

[engram-design]

Can you adjust :rows="get(settings, 'fields')" to :rows="get(settings, 'form')" as I mentioned?

[roland-d]

@engram-design I promise I triple-checked that block of code 🙈 Thank you, that does it.

One other thing I found out, I have now 2 forms and when I load the fields in form 1 and go to form 2, I see the fields from form 1. I reload the fields and see my form 2 fields but when I then go to form 1, I see the fields from form 2 🙃

Is there some kind of caching setting I have to set?

[engram-design]

Not quite sure what you mean sorry! You mean your integration settings (for your Filemaker integration) you set on one form are showing on another? Specifically your mappings?

So for instance, you might have this configured on one form

And then on the other form you don't see this, where they should be blank?

That shouldn't be possible at all, as they're stored in the form, and a form only knows about itself.

[roland-d]

Hello, sorry for the late reply but I had no access again to the CRM until today. So to explain what I see I have created a short video, this hopefully explains what I mean. As you can see when I switch forms and refresh the fields I get the correct fields but going back to the form I see the fields from the other form. Does this video help?

formie-fields.mp4

.

[engram-design]

Right, I've never seen that happen before with any other integration. I'm not sure if it's down to your custom integration, or something you've setup different, but I'm not quite sure how that's even possible.

As for the fields being empty, would you be running PHP 8.1? If so, might be similar to #979

[roland-d]

@engram-design I figured it out, it is not PHP 8.1 as I am running PHP 8.0 but something else.

So when I retrieve the field mapping it is eventually stored in the formie_integrations tables cache field. When I go to another form, this updates the same cache field. Going back to my previous form I see the fields from the form I just stored in the cache.

My question now is, how do I store these fields in a form specific cache? Is that even supported?

[engram-design]

Field mappings are stored in the form settings and the integrations key in that. Take a look at the ‘settings’ column for a form which is a large JSON object. In that, there’ll be an ‘integrations’ key which inside that will store the settings for the integrations. That’s why integrations are stored per form.

The cache column for the integration is updated whenever you refresh the integration. These are not the saves mappings but the available things to map. Think of it like the rows in that table to map values against.

Think of this cache in practical terms like the Mailchimp integration, where you might want to pick which list and which fields in each list to map to. This is fetches for all lists and saved in a cache so that the next time you want to that integration on another form you don’t need to fetch that data again. It also helps when reloading the form not to hit the API all the time.

as for why you’re getting your saved mapped values mixed up, I have no idea what’s going on there. I’d start by inspecting the form settings from the database. Then adding debug statement ‘{% dd…’ to the Twig templates to double check the integration mapped settings are also correct.

whilst some information is saved to the cache for the integration this is global. It has no bearing on the mapped values saved on a form (the values might be there, they’re just never used in practice)

[roland-d]

Hello @engram-design Apologies for getting back this late but other priorities took over.

Field mappings are stored in the form settings and the integrations key in that. Take a look at the ‘settings’ column for a form which is a large JSON object. In that, there’ll be an ‘integrations’ key which inside that will store the settings for the integrations. That’s why integrations are stored per form.

I completely follow that. So my question now is, where or what or when is something triggered to fill the contactFieldMapping in the settings column? This is what I have now in that field:

{"contactFieldMapping":null,"database":"$FM_DATABASE","dealFieldMapping":null,"debug":true,"formFieldMapping":null,"hostName":"$FM_HOSTNAME","leadFieldMapping":null,"password":"$FM_PASSWORD","username":"$FM_USERNAME"}

This is probably also my problem, nothing is stored in this field. I only see the getFormSettingsHtml() triggered when I click on the Refresh button.

I am checking it but not finding my golden egg yet :)

[engram-design]

When you save the form is when those values are populated, provided you actually are sending data in the payload for contactFieldMapping.

For instance, here I'm saving mapping for the Active Campaign CRM integration.

I've mapped the "Form Name" to the "Email" attribute (aware that's invalid - just a quick example)

Can you compare on your end if this is happening, but looking in the network tab of web developer tools and inspect the save-form request? It sounds like it might not be.

settings[integrations][myIntegrationHandle][contactFieldMapping][myAttribute]

[roland-d]

@engram-design Thank you for that, it is starting to make some sense now. So in a previous post we renamed the contactFieldMapping to formFieldMapping. Using that I saw only the form field that was assigned a mapping was being stored. So I reverted it all back to contactFieldMapping and now I see all fields, whether they have a form field assigned or not being stored in the formie_forms.settings field.

In this field I see the correct fields for the given layout. Up to here all is good.

Can you compare on your end if this is happening, but looking in the network tab of web developer tools and inspect the save-form request? It sounds like it might not be.

That is looking good now as I see the name of my integration in the POST request.

Now, when rendering the table with fields, I still see the fields from the previous form I was. So somehow it is not taking the fields from the settings field of the form I am editing. Where would I check that to see what is happening? This almost feels like some kind of caching somewhere.

[engram-design]

Yeah that behaviour is very, very strange. What I would do first is debug the settings for the form.

formie/src/templates/forms/_panes/integrations.html

Line 49 in 5f689f2

Add the following:

{% if handle == 'myIntegrationHandle' %}
    {% dd form.settings.integrations[handle] %}
{% endif %}

What does that show? The correct or incorrect settings for the integration for the form? That should be correct, because that's literally grabbing the settings for the from, straight from the column in the database.

[roland-d]

@engram-design Here is the debug info

[
    'enabled' => '1'
    'layout' => 'Spelend_Leren'
    'contactFieldMapping' => [
        'aanvrager_adres' => '{aanvrager_adres}'
        'aanvrager_postcode' => ''
        'aanvrager_woonplts' => ''
        'aanvraag_voor' => ''
        'aanvrager_wie' => ''
        'aanvrager_naam' => ''
        'aanvrager_naam_voorv' => ''
        'aanvrager_voornaam' => ''
        'aanvrager_geslacht' => ''
        'aanvrager_adres_voorv' => ''
        'aanvrager_huisno_toev' => ''
        'aanvrager_huisno' => ''
        'aanvrager_email' => ''
        'aanvrager_telefoon1' => ''
        'kind_naam' => ''
        'kind_naam_voorv' => ''
        'kind_voornaam' => ''
        'kind_geslacht' => ''
        'kind_geb_dag' => ''
        'kind_geb_mnd' => ''
        'kind_geb_jaar' => ''
        'kind_ovp' => ''
        'school_naam' => ''
        'school_adres' => ''
        'school_postcode' => ''
        'school_leerjaar' => ''
        'UserIp' => ''
        'User_browser' => ''
        'Lang' => ''
        'RSF_id' => ''
    ]
]

These are the correct fields for this form. To be sure I checked another form and this also shows the correct fields in the dump at this point.

[engram-design]

Okay, so this content is fed into the Vue component

formie/src/templates/integrations/crm/active-campaign/_form-settings.html

Line 19 in 5f689f2

<integration-form-settings handle="{{ handle }}" :form-settings="{{ formSettings | json_encode }}">

just to check that they're the same there?

Then, a little further down

formie/src/templates/integrations/crm/active-campaign/_form-settings.html

Line 46 in 5f689f2

:value="get(form, 'settings.integrations.{{ handle }}.contactFieldMapping')"

You could test on the Vue end:

${ get(form, 'settings.integrations.{{ handle }}.contactFieldMapping') }

[roland-d]

just to check that they're the same there?

Yes, they are exactly the same the integration-form-settings.

So I tested the getForm() in Vue and that output looks good, so I tested it a bit further with the getSettings() and that output is wrong. That has the incorrect fields in it. This is how I tested that:

${ get(settings, 'contact') }

I checked further but I don't see so quickly where this data is pulled from.

Where do I look next?

[engram-design]

So settings In the Vue context here (in ${ get(settings, 'contact') }) is being passed into the integration-form-settings Vue component, and accessible to the v-slot. The source of this content is largely the same, coming from Twig:

{% set formSettings = integration.getFormSettings().getSettings() %}

Which is all in the

formie/src/templates/integrations/crm/active-campaign/_form-settings.html

Line 46 in 5f689f2

:value="get(form, 'settings.integrations.{{ handle }}.contactFieldMapping')"

file (as an example integration file).

integration.getFormSettings() should indeed fetch from the cache, but maybe test this using integration.getFormSettings(false) to check that behaviour.

[roland-d]

We are getting closer to the truth :)

So I set the cache to false and it died on me 😆 That is because I require a layout to be specified, I need this layout to retrieve the fields from an API server. It looks like this:

Doing the call from the twig template, does not have the layout field in the sendFormPayload.

Every form that I have has a different layout. I guess this is the root cause of what we are seeing, the cache is called contact for all forms instead of just one form.

I would need a cache per form based on my handle I guess. So I tried to change $settings['contact'] = $filemakerFields; to $settings[$layout] = $filemakerFields; so this is unique however that does not work because Formie is pulling from the last time the cache was filled and this is the previous form I was. I checked that by running this:

{% dd(formSettings) %}

and this is the partial output:

[
    'syncFormSuccess' => 'Successfully synced FileMaker fields'
    'syncFormError' => ''
    'rs_wkp' => [
        0 => verbb\formie\models\IntegrationField#1
        (
            [handle] => 'telefoon'
            [name] => 'telefoon'
            [type] => null
            [required] => null

So to summarize, in my case each form has a unique set of fields coming in from the API, there is not one superset that can be chosen from. The caching of the fields does not expect that.

Is this something we can fix? I hope this is clear :)

[engram-design]

Right, so that makes sense compared to every other integration out there. Deal objects have set fields, Contacts likewise. For your use-case, these need to be specific and per-form.

Unfortunately, this would be a bit of a major refactor on our end to pass an instance of the form to the cache function, which is going to annoy other custom integrations that'll break.

But I think you can get around this in your integration, with some changes. You'll need to switch from the generic contact key to store your data, and use something like contact-__formHandle__ or you could use your contact-__layout__ if you wanted to cache including your layout.

You can do this by firstly defining the data in your fetchFormSettings() method to use the unique key:

return new IntegrationFormSettings([
    'contact-__layout__' => $myFields,
]);

Then, ensure that you grab those values in Vue.

<integration-field-mapping
    ...
    :rows="get(settings, 'contact-__layout__')"
></integration-field-mapping>

Where replacing __layout__ with your actual layout variable from the form settings. This is a long thread, but judging from your past code sample, you seem to require that when fetching settings anyway.

You'd grab __layout__ in Twig/Vue the same way that you populate the text field in your screenshot above (which is also for settings).

Hopefully, that'll help? Otherwise, you're best to disable the cache altogether and modify your code to check if it's the first-run (because fetchFormSettings() is called as soon as the form builder loads) or if it's a POST request.

// For the initial render, return an empty array, until we try and fetch again with the layout provided
if (!Craft::$app->getRequest()->getIsPost()) {
    return new IntegrationFormSettings([]);
}

if (!Craft::$app->getRequest()->getParam('sendFormPayload')) {
    throw new \Exception('A layout must be specified');
}

The major drawback with this is that because you have no saved rows for the mapping, you have to fetch the fields every time you want to manage the integration mapping. It'll always show no fields available to map against until you hit "Refresh Integration"

[roland-d]

Right, so that makes sense compared to every other integration out there.

😄 This is a Filemaker connection and in FIlemaker you create a layout per form. So you can map your website form to the Filemaker form.

But I think you can get around this in your integration, with some changes.

I actually made those changes as well and that works. Like you said, you will see no fields until I click Refresh Integration.

This workaround is workable for me, as I can map my fields and the mapping can be used when posting the payload. The integration is something I setup only once (most of the time at least) and after that it is set.

Thank you for your help in this and hopefully the option to set fields per form in cache can be added to the wishlist.

[engram-design]

Nice one, and yep will give it some further thought for the next major version. We're likely to run into this ourselves at some point for a similar integration requirement.