SharePoint Graph API support

[Drakonian]

Summary

This is the first version of the SharePoint Graph API module that leverages the Microsoft Graph module.

After analyzing the current REST implementation, I concluded that it cannot be changed without breaking backward compatibility. In addition, the REST and Graph APIs have completely different data models and sets of fields, which makes it impossible to reuse the same tables for entities that REST uses.

Therefore, adding a separate SharePoint Graph Client seemed the simplest and most natural solution. I also considered factory/builder patterns and dependency injection, but for the reasons mentioned above these approaches did not seem optimal, they don’t fit well and would greatly complicate the overall use of the SharePoint module.

I have already tested this code locally in various scenarios, and it behaves well. My plans are:

• Detail review of all the code with improvements where necessary
• Gathering feedback from the community and Microsoft
• Test pagination
• Test large file uploads
• Restructuring the folders so that REST and Graph reside in separate, peer-level folders
• Writing tests

Examples of how to use:

local procedure GetDrivesFromSite()
var
    TempGraphDrive: Record "SharePoint Graph Drive" temporary;
    SharePointGraphClient: Codeunit "SharePoint Graph Client";
    GraphAuthorization: Codeunit "Graph Authorization";
    GraphAuthInterface: Interface "Graph Authorization";
begin
    GraphAuthInterface := GraphAuthorization.CreateAuthorizationWithClientCredentials(
        TenantIdValue,
        ClientIdValue,
        ClientSecretValue,
        GraphScopeLbl);

    SharePointGraphClient.Initialize(SharePointUrlValue, GraphAuthInterface);

    if not SharePointGraphClient.GetDrives(TempGraphDrive) then
        Message('No drives found.');

    if TempGraphDrive.FindSet() then
        repeat
            Message('Drive Name: %1\Type: %2\Owner: %3\Quota Used: %4 of %5\Created: %6',
                TempGraphDrive.Name,
                TempGraphDrive.DriveType,
                TempGraphDrive.OwnerName,
                TempGraphDrive.QuotaUsed,
                TempGraphDrive.QuotaTotal,
                TempGraphDrive.CreatedDateTime);
        until TempGraphDrive.Next() = 0;
end;

local procedure UploadFileToRootDriveFolder()
var
    TempGraphDriveItem: Record "SharePoint Graph Drive Item" temporary;
    SharePointGraphClient: Codeunit "SharePoint Graph Client";
    GraphAuthorization: Codeunit "Graph Authorization";
    GraphAuthInterface: Interface "Graph Authorization";
    Diagnostics: Interface "HTTP Diagnostics";
    FileInStream: InStream;
    FileName: Text;
    FolderPath: Text;
begin
    GraphAuthInterface := GraphAuthorization.CreateAuthorizationWithClientCredentials(
        TenantIdValue,
        ClientIdValue,
        ClientSecretValue,
        GraphScopeLbl);

    SharePointGraphClient.Initialize(SharePointUrlValue, GraphAuthInterface);

    FolderPath := 'TestFolder'; //To root drive folder as we not specified DriveId
    if not UploadIntoStream('Select a file to upload', '', '', FileName, FileInStream) then
        exit;

    if SharePointGraphClient.UploadFile(FolderPath, FileName, FileInStream, TempGraphDriveItem) then
        Message('File uploaded successfully: %1', TempGraphDriveItem.Name)
    else begin
        Diagnostics := SharePointGraphClient.GetDiagnostics();
        Error('Error: %1 - %2', Diagnostics.GetHttpStatusCode(), Diagnostics.GetResponseReasonPhrase());
    end;
end;

Work Item(s)

Fixes #3198

Fixes AB#568746

[Drakonian]

@pri-kise

[JesperSchulz]

Very interesting contribution! Please allow me to take a much closer look at your code in the coming days. Unfortunately I will be on the road next week, so this might take me a short while to get to - but I'll be on it soon!

[Drakonian]

Very interesting contribution! Please allow me to take a much closer look at your code in the coming days. Unfortunately I will be on the road next week, so this might take me a short while to get to - but I'll be on it soon!

Thank you, I’ll look forward to your feedback -)

[pri-kise]

First of all: I really appreciate the work you have done for this PR.

I'm unsure if maybe some of the functionalities in the graph helper codeunits should maybe better be placed into the graph client module.
I only took a short look into the new code so far. I will need some more time to look into the changes in more detail.

Update:
I'm not sure if it's really helpful to provide a boolean value as return value on all those functions.
How should anyone know why the got false.

I think that it would be more helpful to return errors when something didn't went well.

Furthermore on your planned improvements for pagination:
I'm not sure if this is possible with the GraphOptionalParameters, but I think that it would be helpful if we could specify a range from outside, too.
What I mean is that sometimes we might want to control the pagination steps from outside.

[Drakonian]

@pri-kise

Thank you for such a helpful review of the PR. Your remarks are very useful.

1. I added an HTTP client handler, of course, it is absolutely essential here.
2. I completely agree with your point about the boolean, it's a relic from an older code version where I was experimenting with building a SharePoint Graph module (trying to merge with the SharePoint REST module). I prefer the approach used in the Azure Blob Storage API, where a method returns an object containing detailed information about the operation, so I’ve applied a similar pattern here.

Regarding managing pagination externally, that’s an excellent observation, but I’m still considering the best way to implement it. I’ll think through an approach that is convenient.

I believe pagination would be best handled at the Microsoft Graph module level, this needs some further thought.

My plans: at the moment I intend to keep improving the code and finish writing the tests.

[JesperSchulz]

@Drakonian, wow, that's impressive work! I must admit that I got "code review fatigue" about 1/3 into the review process - that's A LOT of code 🤣 I will need to see the code in action, rather than just read my way through it.

1. I will try to compile and use your module updates
2. I will wait for your tests to fully understand how the code is to be used in praxis

Generally though, I have no objections of accepting your changes. They seem to follow all of our best practices for system application modules!

[Drakonian]

@Drakonian, wow, that's impressive work! I must admit that I got "code review fatigue" about 1/3 into the review process - that's A LOT of code 🤣 I will need to see the code in action, rather than just read my way through it.

1. I will try to compile and use your module updates
2. I will wait for your tests to fully understand how the code is to be used in praxis

Generally though, I have no objections of accepting your changes. They seem to follow all of our best practices for system application modules!

Thanks, I honestly spent quite a bit of effort to make the code match the quality and style of System Application :)

Of course, the best thing to look at is in action. I'll tag you when I'm done with the tests and also will prepare several examples of usage.

[JesperSchulz]

@Drakonian, wow, that's impressive work! I must admit that I got "code review fatigue" about 1/3 into the review process - that's A LOT of code 🤣 I will need to see the code in action, rather than just read my way through it.

1. I will try to compile and use your module updates
2. I will wait for your tests to fully understand how the code is to be used in praxis

Generally though, I have no objections of accepting your changes. They seem to follow all of our best practices for system application modules!

Thanks, I honestly spent quite a bit of effort to make the code match the quality and style of System Application :)

Of course, the best thing to look at is in action. I'll tag you when I'm done with the tests and also will prepare several examples of usage.

I can tell! Really nice work! Let's get this reeled in in time for the 2025 wave 2 release 🥳 Of course we will need documentation updates and release notes, but we can work together on that!

Friday I'll try to take this for an actual spin! Exciting 👏

[tinestaric]

@Drakonian, to echo @JesperSchulz, really, really nice work! And it's precisely what I'll need soon! Thank you! :D

I'll take this module for a spin; however, I see a couple of gaps in the code.

1. UploadLargeFile sends the file by chunks, but there's no DownloadLargeFile equivalent. I recently discovered that BC has a limit on the size of the response content HttpClient can receive, set to 150 MB by default (controlled by the NavHttpClientMaxResponseContentSize server setting). That means that you can't download files larger than that, so we'd need a chunking option for download as well.

2. I'm missing any Delete operations for either Files or Folders

3. No existence check operations for files or folders

And there are two nice-to-haves, which I can work around with a combination of existing functions, but it would be cleaner to use the Graph API directly
4. Copy operation (but I can use Download+Upload)
5. Move operation - via PATCH parentReference (but Download+Upload+Delete gets me there as well)

[Drakonian]

@tinestaric
Excellent observations!

I expect the next few weeks to be less busy, and I will be able to work on this PR. Thank you for your comments, they will help make the module much better.

[Drakonian]

@tinestaric
All right, if that is not an issue, I will focus and try to finish this module together with the tests a bit earlier. I am already eager to start using this module on projects myself :)

Do I understand correctly that, for some unclear reason, the output InStream is causing an error in the FileStorage interface? Could you please indicate where you are using the DownloadFile method from the Graph SharePoint module in your PR?

[JesperSchulz]

I'll let the two of you figure it out when this PR is ready to go out of DRAFT mode. Once it is, I'll be ready to assist to get it in as quickly as possible - and I'd also be open to backport it to 27.x, if need be!

Let me know when / if I'm needed.

[tinestaric]

@Drakonian
Great, thank you again for all the work you've already done!

As for the issue, here's the boiled-down code that's causing it. I've tried it on 27.1 today without any of the changes I've made to the module.

                    trigger OnAssistEdit()
                    var
                        GraphAuthorization: Codeunit "Graph Authorization";
                        GraphAuthInterface: Interface "Graph Authorization";
                        SharePointGraphClient: Codeunit "SharePoint Graph Client";
                        Response: Codeunit "SharePoint Graph Response";
                        Stream: InStream;
                        FileName: Text;
                        Path: Text;
                        TenantId: Text;
                        ClientId: Text;
                        ClientSecret: SecretText;
                        SharePointUrl: Text;
                        Scopes: List of [Text];
                    begin
                        // Hardcoded configuration
                        TenantId := '***';
                        ClientId := '***';
                        ClientSecret := SecretText.SecretStrSubstNo('***');
                        SharePointUrl := 'https://mySharepoint.sharepoint.com/sites/testnl/';
                        Path := 'BC/Tine/combined-picture.png';

                        // Initialize authorization
                        Scopes.Add('https://graph.microsoft.com/.default');
                        GraphAuthInterface := GraphAuthorization.CreateAuthorizationWithClientCredentials(
                            TenantId,
                            ClientId,
                            ClientSecret,
                            Scopes);

                        // Initialize SharePoint Graph Client
                        SharePointGraphClient.Initialize(SharePointUrl, GraphAuthInterface);

                        // Download file - this fails
                        Response := SharePointGraphClient.DownloadFileByPath(Path, Stream);

                        if not Response.IsSuccessful() then
                            Error('Download failed: %1', Response.GetError());

                        FileName := 'DownloadedFile.png';
                        DownloadFromStream(Stream, '', '', '', FileName);
                    end;

The request succeeds. But the DownloadFromStream fails with "Stream not initialized". When digging a bit deeper, Stream is still fine while within the ProcessStreamResponse, but it becomes uninitialized as it exits that procedure.

Here, the length is still as I'd expect, but in the next part, this message already breaks because InStream isn't initialized.

In my PR, I'm using DownloadLargeFileByPath in the GetFile method of ExtSharePointGraphHelper. It's a function I've added, but it breaks for the same issue: the ProcessStreamResponse I used to read each chunk.

[pri-kise]

Just some notes:
The InStream is always bound to the Blob Field or the HttpContent.

Out could be the issue that the HttpResponseMessage is local and therefore the InStream is lost after the procedure is left. (See also: microsoft/AL#7172)

There's another issue that reports similiar problems with the external file storage module:
#5439

I switched to TempBlob as return value to pass around files for this reason.

In the already exisitng SharePoint Client the InStream is created from a global variable that still exist when the local procedure is left.

[Drakonian]

@tinestaric @pri-kise

Thank you for the additional context, I have successfully reproduced the issue locally.

Interestingly, it seems to me that this issue did not exist when I last tested the module, but since that was five months ago, a lot has changed in the code base/platform, or I may simply have missed it.

It looks more like a platform issue, I will try to experiment a bit and will most likely also switch to TempBlob.

[Drakonian]

@tinestaric

I tried to reuse your code as much as possible so that your time would not be wasted.

I also added automated tests, but only for part of the functionality. I still need to test everything thoroughly and complete automated tests. (I will move library/handlers codeunits for tests in separate application later)

You could also help me with testing if you want to speed up the process.

[tinestaric]

@Drakonian
Thank you, and yes. Let me know when it's ready for testing, and I'll go ahead.

[Drakonian]

I have good news, I added automated tests and manually tested the current implementation including new methods from @tinestaric
I believe this PR is ready for review, so it is no longer a draft.

Therefore, I would appreciate any code review
@tinestaric @pri-kise @JesperSchulz

[duichwer]

Do we those new tables to extensible (at the first release)?

What I mean is, if it weren't better to make the new tables extensible false at first

[Drakonian]

To be honest, I am not even sure. Are there any best practices? Why prevent others from extending the temporary tables?

[tinestaric]

@Drakonian
Great! I didn't spot anything on the code review, but I'll take all these changes for a spin tomorrow and report back if I find anything.

[tinestaric]

@Drakonian
I tested the following scenarios using your framework, and they all work without a problem.
List Files/Folders
Download File
Create File
Copy File
Move File
File Exists
Delete File
Create Folder
Folder Exists
Delete Folder

The one notable thing I did spot (but it is not directly related to your changes) is that using the Graph API is considerably slower than the REST API when listing files/folders.

The issue is that the REST API retrieves all at once, while the Graph API paginates requests. Pagination is fine, but the default page size set on GetAllPages in Graph Client Impl. is 100, and there's no way to change it because GraphPaginationData isn't passed as a parameter.

In my tests, for a folder of 850 files, I'm noticing about 3-4 seconds slower performance.

It shouldn't be worth changing in the scope of this PR, but for anyone coming across this PR, it might be worth noting.

[duichwer]

@Drakonian I tested the following scenarios using your framework, and they all work without a problem. List Files/Folders Download File Create File Copy File Move File File Exists Delete File Create Folder Folder Exists Delete Folder

The one notable thing I did spot (but it is not directly related to your changes) is that using the Graph API is considerably slower than the REST API when listing files/folders.

The issue is that the REST API retrieves all at once, while the Graph API paginates requests. Pagination is fine, but the default page size set on GetAllPages in Graph Client Impl. is 100, and there's no way to change it because GraphPaginationData isn't passed as a parameter.

In my tests, for a folder of 850 files, I'm noticing about 3-4 seconds slower performance.

It shouldn't be worth changing in the scope of this PR, but for anyone coming across this PR, it might be worth noting.

@tinestaric aren't you able to set the page size via GraphOptionalParameters: Codeunit "Graph Optional Parameters" ?

Then the page size of the optional parameters is used here:

BCApps/src/System Application/App/MicrosoftGraph/src/GraphClientImpl.Codeunit.al

Line 107 in 5c5e218

GraphPaginationHelper.ApplyPageSize(GraphOptionalParameters, GraphPaginationData);

[tinestaric]

@Drakonian I tested the following scenarios using your framework, and they all work without a problem. List Files/Folders Download File Create File Copy File Move File File Exists Delete File Create Folder Folder Exists Delete Folder
The one notable thing I did spot (but it is not directly related to your changes) is that using the Graph API is considerably slower than the REST API when listing files/folders.
The issue is that the REST API retrieves all at once, while the Graph API paginates requests. Pagination is fine, but the default page size set on GetAllPages in Graph Client Impl. is 100, and there's no way to change it because GraphPaginationData isn't passed as a parameter.
In my tests, for a folder of 850 files, I'm noticing about 3-4 seconds slower performance.
It shouldn't be worth changing in the scope of this PR, but for anyone coming across this PR, it might be worth noting.

@tinestaric aren't you able to set the page size via GraphOptionalParameters: Codeunit "Graph Optional Parameters" ?

Then the page size of the optional parameters is used here:

BCApps/src/System Application/App/MicrosoftGraph/src/GraphClientImpl.Codeunit.al

Line 107 in 5c5e218

GraphPaginationHelper.ApplyPageSize(GraphOptionalParameters, GraphPaginationData);

That's what I initially thought, and I wanted to set the "top" query parameter for Graph Optional Parameters.
But within ApplyPageSize, it actually checks the GraphPaginationData for a page size (which defaults to 100) and then overrides the GraphOptionalParameter's "top" value.

So, regardless of what I set in the optional parameters, it's always overwritten by the default page size of 100.

[Drakonian]

@tinestaric

Thank you for your discovery and testing. Overall, I agree that improving pagination most likely relates to the context of the Graph API module.

Therefore, I suggest moving forward with this module.
I will create separate issue/PR regarding your discovery in the Graph API module a little later.

I think we are close to completing this PR.
What do you think about it?
@JesperSchulz

[JesperSchulz]

@tinestaric

Thank you for your discovery and testing. Overall, I agree that improving pagination most likely relates to the context of the Graph API module.

Therefore, I suggest moving forward with this module. I will create separate issue/PR regarding your discovery in the Graph API module a little later.

I think we are close to completing this PR. What do you think about it? @JesperSchulz

Let's push forward 🚀 I'm super busy these days, but I'll make room for this one 😊

[JesperSchulz]

@Drakonian, looks like we've got some missing permissions and some issues with accessibility. Let's get all lights 🟢 and then we'll continue!