Skip to content

Commit 6473ce1

Browse files
c200bzhc200bzh
authored
apple_store: new custom connector (#29)
* apple_store: new custom connector Remove reports_examples folder * app_store : add report ids in config * app_store : delete default report ids
1 parent 9bca8f8 commit 6473ce1

20 files changed

+1285
-0
lines changed

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,4 +158,9 @@ cython_debug/
158158
# and can be added to the global gitignore or merged into this file. For a more nuclear
159159
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160160
#.idea/
161+
162+
# Ignore reports examples with sensitive data
163+
reports_examples/
164+
161165
*.ps1
166+

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,10 @@ Repository holding the custom Airbyte connector using in Status-im.
1717
* `source-discourse-fetcher`: Source connector to fetch data from Discourse API
1818
* `source-simplecast-fetcher`: Source connector to fetch data from Simplecast API
1919
* `source-twitter-fetcher`: Source connector to fetch data from Twitter API
20+
21+
### Status Analytics related
22+
23+
* `source-app-store`: Source connector to fetch data from Apple App Store
24+
* `source-google-play`: Source connector to fetch data from Google Play Store
25+
26+

‎source-app-store/Dockerfile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
FROM --platform=linux/amd64 airbyte/python-connector-base:1.1.0
2+
3+
COPY . ./airbyte/integration_code
4+
RUN pip install ./airbyte/integration_code google-api-python-client
5+
6+
7+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
8+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

‎source-app-store/README.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Apple Store Source
2+
3+
This is the repository for the Apple Store source connector, written in Python.
4+
For information the Apple Store connect API [see here](https://developer.apple.com/documentation/AppStoreConnectAPI/downloading-analytics-reports).
5+
6+
## App Store Connect API Reports
7+
8+
This connector extracts reports from the App Store Connect API.
9+
10+
### Report Structure
11+
12+
- Each report type has a unique ID which can be found in `reports.csv`
13+
- Reports are generated daily by Apple and typically contain data from the current day and previous 3 days,due to this rolling window, reports from consecutive days may contain duplicate data so in these connector for each report we only load data that corresonds to the day before the report was generated.
14+
15+
### Sync Behavior
16+
17+
- Each sync processes reports between `start_date` and `end_date`
18+
- `end_date` defaults to today
19+
- `start_date` defaults to 4 days ago
20+
- The connector downloads fresh copies of reports on each sync
21+
- If a sync is run multiple times in the same day, newer downloads will replace older ones
22+
23+
### Report Processing
24+
25+
- For each report type, the connector finds available instances (a report instance represents a specific date)
26+
- The processing date in the report metadata is the date when Apple generated the report
27+
- Records are filtered to include only data matching the expected date (1 day before the day of the report)
28+
29+
### Available Reports
30+
31+
#### App Install Performance
32+
- Documents app installation performance metrics
33+
- Documentation: https://developer.apple.com/documentation/analytics-reports/app-installs-performance
34+
- ID: r5-1032fee7-dfb3-4a4a-b24d-e603c95f5b09
35+
36+
#### Additional Reports
37+
- App Downloads Detailed
38+
- App Installation and Deletion Detailed
39+
- App Sessions Detailed
40+
- App Discovery and Engagement Detailed
41+
42+
### Technical Notes
43+
44+
- The connection may take several minutes to complete due to the API response times
45+
- A deduplication mechanism is implemented in the `read_records` function to handle overlapping data
46+
47+
## Overview
48+
49+
This connector extracts extract data from diferent apple store reports using the App store connect API.
50+
51+
### Output schema
52+
53+
This connector outputs the following streams that corresponds each to a specific report:
54+
55+
1. **App Installs Performance**
56+
2. **App Downloads Detailed**
57+
3. **App Installation and Deletion Detailed**
58+
4. **App Sessions Detailed**
59+
5. **App Discovery and Engagement Detailed**
60+
61+
### Configuration
62+
63+
The connector requires the following configuration parameters:
64+
65+
```yaml
66+
key_id:
67+
description: Your App Store Connect API Key ID.
68+
issuer_id:
69+
description: Your App Store Connect API Issuer ID (found in the API Keys section of App Store Connect).
70+
private_key:
71+
description: The private key content for the App Store Connect API. Include the entire key, including the BEGIN and END lines.
72+
start_date (optional):
73+
description: The date to start syncing data from, in YYYY-MM-DD format. If not provided, defaults to 3 days ago. The oldest date available is the 2025-04-02 (because no reports request were made before this date).
74+
examples:
75+
- "2023-01-01"
76+
end_date (optional):
77+
description: The date to sync data until, in YYYY-MM-DD format. If not provided, defaults to the current date.
78+
examples:
79+
- "2023-04-30"
80+
report_ids (optional):
81+
description: Custom report IDs to use for fetching data. If not provided, default IDs will be used.
82+
83+
```
84+
85+
86+
87+
## Local development
88+
89+
### Prerequisites
90+
91+
#### Activate Virtual Environment and install dependencies
92+
From this connector directory, create a virtual environment:
93+
```
94+
python -m venv .venv
95+
```
96+
```
97+
source .venv/bin/activate
98+
pip install -r requirements.txt
99+
```
100+
101+
### Locally running the connector
102+
```
103+
python main.py spec
104+
python main.py check --config sample_files/config-example.json
105+
python main.py discover --config sample_files/config-example.json
106+
python main.py read --config sample_files/config-example.json --catalog sample_files/configured_catalog.json
107+
```
108+
109+
### Locally running the connector docker image
110+
111+
```bash
112+
docker build -t airbyte/source-app-store:dev .
113+
# Running the spec command against your patched connector
114+
docker run airbyte/source-app-store:dev spec
115+
```
116+
117+
#### Run
118+
Then run any of the connector commands as follows:
119+
```
120+
docker run --rm airbyte/source-app-store:dev spec
121+
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/source-app-store:dev check --config /sample_files/config-example.json
122+
docker run --rm -v $(pwd)/sample_files:/sample_files airbyte/source-app-store:dev discover --config /sample_files/config-example.json
123+
docker run --rm -v $(pwd)/sample_files:/sample_files -v $(pwd)/sample_files:/sample_files airbyte/source-app-sore:dev read --config /sample_files/config-example.json --catalog /sample_files/configured_catalog.json
124+
```
125+
126+
## Notes
127+
128+
- reports.csv is the list of all the reports that exist with theyr name, category and id
129+
- be aware that the scripts can take some time to execute (some minutes)
130+
131+
* App Install Performance
132+
a new report is generated every day, in this report we have data from the current day and the previous 3 days. so we can have duplicates in the data for example between 2025-04-01 and 2025-04-02 reports.
133+
This is why we implemented a deduplication mechanism in the read_records function.
134+
135+
Doc about the report: https://developer.apple.com/documentation/analytics-reports/app-installs-performance
136+
137+
if the sync is tun twice the same day, the second run will override the first one. (ir ancient downloads will be replaced by the new ones)
138+
139+
Process every report with date between start_date and end_date (end_date defaults to today start_date defaults to 4 days ago) porcessing date is the date of the report.
140+
141+
Each sync will download fresh copies of the reports
142+
Old files will be overwritten with new data
143+
144+
Reports ids can be found in the reports.csv file, this file is generated after making a request report api call with adming credentials.
145+
146+
for each report an instance is equivalent to a report's date so there is one instance per report per day

‎source-app-store/main.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#
2+
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
3+
#
4+
5+
from source_app_store.run import run
6+
7+
if __name__ == "__main__":
8+
run()

‎source-app-store/metadata.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
data:
2+
allowedHosts:
3+
registries:
4+
oss:
5+
enabled: true
6+
cloud:
7+
enabled: false
8+
remoteRegistries:
9+
pypi:
10+
enabled: true
11+
packageName: airbyte-source-app-store
12+
connectorBuildOptions:
13+
baseImage: docker.io/airbyte/python-connector-base:1.0.0@sha256:dd17e347fbda94f7c3abff539be298a65af2d7fc27a307d89297df1081a45c27
14+
connectorSubtype: api
15+
connectorType: source
16+
definitionId: 464a7cea-0317-485e-9a9c-bcd06155bfff
17+
dockerImageTag: 1.0.0
18+
dockerRepository: harbor.status.im/bi/airbyte/source-app-store
19+
githubIssueLabel: source-app-store
20+
icon: app-store.svg
21+
license: MIT
22+
name: Apple Store
23+
releaseDate: TODO
24+
supportLevel: community
25+
releaseStage: alpha
26+
documentationUrl: https://docs.airbyte.com/integrations/sources/source-app-store
27+
githubIssueLabel: source-app-store
28+
- language:python
29+
metadataSpecVersion: "1.0"

‎source-app-store/requirements.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
httplib2>=0.22.0
2+
oauth2client>=4.1.3
3+
setuptools>=6
4+
airbyte-cdk>=0.51.0
5+
requests>=2.28.1
6+
cryptography>=41.0.1
7+
PyJWT==2.8.0

‎source-app-store/sample_files/config-example.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"key_id": "key_id",
3+
"issuer_id": "issuer_id",
4+
"private_key": "private_key written in one line with \n as line break"
5+
}
6+

‎source-app-store/sample_files/configured_catalog.json‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
{
2+
"streams": [
3+
{
4+
"stream": {
5+
"name": "app_install_performance",
6+
"json_schema": {
7+
"$schema": "http://json-schema.org/draft-07/schema#",
8+
"type": "object"
9+
},
10+
"supported_sync_modes": [
11+
"full_refresh",
12+
"incremental"
13+
],
14+
"source_defined_cursor": true,
15+
"default_cursor_field": ["Date"]
16+
},
17+
"sync_mode": "incremental",
18+
"cursor_field": ["Date"],
19+
"destination_sync_mode": "append"
20+
},
21+
{
22+
"stream": {
23+
"name": "app_downloads_detailed",
24+
"json_schema": {
25+
"$schema": "http://json-schema.org/draft-07/schema#",
26+
"type": "object"
27+
},
28+
"supported_sync_modes": [
29+
"full_refresh",
30+
"incremental"
31+
],
32+
"source_defined_cursor": true,
33+
"default_cursor_field": ["Date"]
34+
},
35+
"sync_mode": "incremental",
36+
"cursor_field": ["Date"],
37+
"destination_sync_mode": "append"
38+
},
39+
{
40+
"stream": {
41+
"name": "app_installation_deletion_detailed",
42+
"json_schema": {
43+
"$schema": "http://json-schema.org/draft-07/schema#",
44+
"type": "object"
45+
},
46+
"supported_sync_modes": [
47+
"full_refresh",
48+
"incremental"
49+
],
50+
"source_defined_cursor": true,
51+
"default_cursor_field": ["Date"]
52+
},
53+
"sync_mode": "incremental",
54+
"cursor_field": ["Date"],
55+
"destination_sync_mode": "append"
56+
},
57+
{
58+
"stream": {
59+
"name": "app_sessions_detailed",
60+
"json_schema": {
61+
"$schema": "http://json-schema.org/draft-07/schema#",
62+
"type": "object"
63+
},
64+
"supported_sync_modes": [
65+
"full_refresh",
66+
"incremental"
67+
],
68+
"source_defined_cursor": true,
69+
"default_cursor_field": ["Date"]
70+
},
71+
"sync_mode": "incremental",
72+
"cursor_field": ["Date"],
73+
"destination_sync_mode": "append"
74+
},
75+
{
76+
"stream": {
77+
"name": "app_discovery_engagement_detailed",
78+
"json_schema": {
79+
"$schema": "http://json-schema.org/draft-07/schema#",
80+
"type": "object"
81+
},
82+
"supported_sync_modes": [
83+
"full_refresh",
84+
"incremental"
85+
],
86+
"source_defined_cursor": true,
87+
"default_cursor_field": ["Date"]
88+
},
89+
"sync_mode": "incremental",
90+
"cursor_field": ["Date"],
91+
"destination_sync_mode": "append"
92+
}
93+
]
94+
}
95+

‎source-app-store/setup.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
"""
2+
Setup file for the Google Play Source connector.
3+
"""
4+
5+
from setuptools import setup, find_packages
6+
7+
setup(
8+
name="source_app_store",
9+
description="Source implementation for Apple Store Data.",
10+
author="Claire",
11+
author_email="[email protected]",
12+
packages=find_packages(),
13+
install_requires=[
14+
"airbyte-cdk~=0.2",
15+
"google-api-python-client>=2.0.0",
16+
17+
],
18+
#package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
19+
package_data={
20+
"source_app_store": ["schemas/*.json", "*.yaml"],
21+
},
22+
entry_points={
23+
"console_scripts": [
24+
"source-app-store=source_app_store.run:run",
25+
],
26+
},
27+
)
28+

0 commit comments

Comments
 (0)