feat: add upload and download modules for external services

Add upload support for Figshare, Zenodo, Internet Archive, and triplestore.
Add download support for Figshare.

Author: arcangelo7

README.md

@@ -1,28 +1,95 @@
 # Piccione
 
-[![Run tests](https://github.com/arcangelo7/piccione/actions/workflows/tests.yml/badge.svg)](https://github.com/arcangelo7/piccione/actions/workflows/tests.yml)
+Pronounced *Py-ccione*.
+
+[![Run tests](https://github.com/opencitations/piccione/actions/workflows/tests.yml/badge.svg)](https://github.com/opencitations/piccione/actions/workflows/tests.yml)
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](https://opensource.org/licenses/ISC)
 
-A Python toolkit for uploading and downloading data to external repositories and cloud services..
+**PICCIONE** - Python Interface for Cloud Content Ingest and Outbound Network Export
+
+A Python toolkit for uploading and downloading data to external repositories and cloud services.
 
 ## Installation
 
 ```bash
 pip install piccione
 ```
 
-## Usage
+## Modules
+
+### Upload
+
+#### Figshare
+Upload files to Figshare.
+
+```bash
+python -m piccione.upload.on_figshare config.yaml
+```
+
+Configuration file format:
+```yaml
+TOKEN: your_figshare_token
+ARTICLE_ID: 12345678
+files_to_upload:
+  - /path/to/file1.zip
+  - /path/to/file2.zip
+```
+
+#### Zenodo
+Upload files to Zenodo.
+
+```bash
+python -m piccione.upload.on_zenodo config.yaml
+```
+
+Configuration file format:
+```yaml
+access_token: your_zenodo_token
+project_id: 12345678
+zenodo_url: https://zenodo.org
+files:
+  - /path/to/file1.zip
+  - /path/to/file2.zip
+```
+
+#### Internet Archive
+Upload files to the Internet Archive.
+
+```bash
+python -m piccione.upload.on_internet_archive config.yaml
+```
+
+Configuration file format:
+```yaml
+identifier: my-archive-item
+access_key: your_access_key
+secret_key: your_secret_key
+file_paths:
+  - /path/to/file1.zip
+metadata:
+  title: My Archive Item
+  description: Description of the item
+```
 
-```python
-from piccione import example_function
+#### Triplestore
+Execute SPARQL UPDATE queries on a triplestore.
 
-result = example_function()
-print(result)
+```bash
+python -m piccione.upload.on_triplestore http://localhost:8890/sparql /path/to/sparql/folder
+```
+
+### Download
+
+#### Figshare
+Download all files from a Figshare article.
+
+```bash
+python -m piccione.download.from_figshare 12345678 -o /output/directory
 ```
 
 ## Documentation
 
-Full documentation is available at: https://arcangelo7.github.io/piccione/
+Full documentation is available at: https://opencitations.github.io/piccione/
 
 ## Development
 
@@ -31,11 +98,8 @@ This project uses [UV](https://docs.astral.sh/uv/) for dependency management.
 ### Setup
 
 ```bash
-# Clone the repository
-git clone https://github.com/arcangelo7/piccione.git
+git clone https://github.com/opencitations/piccione.git
 cd piccione
-
-# Install dependencies
 uv sync --all-extras --dev
 ```
 

pyproject.toml

@@ -17,12 +17,19 @@ classifiers = [
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
 ]
-dependencies = []
+dependencies = [
+    "internetarchive>=5.7.1",
+    "pyyaml>=6.0.3",
+    "redis>=7.1.0",
+    "requests>=2.32.5",
+    "sparqlite>=1.0.0",
+    "tqdm>=4.67.1",
+]
 
 [project.urls]
-Homepage = "https://github.com/arcangelo7/piccione"
-Documentation = "https://arcangelo7.github.io/piccione/"
-Repository = "https://github.com/arcangelo7/piccione"
+Homepage = "https://github.com/opencitations/piccione"
+Documentation = "https://opencitations.github.io/piccione/"
+Repository = "https://github.com/opencitations/piccione"
 
 [dependency-groups]
 dev = [

src/piccione/__init__.py

@@ -1,12 +1,3 @@
-"""A Python toolkit for uploading and downloading data to external repositories and cloud services.."""
+"""PICCIONE - Python Interface for Cloud Content Ingest and Outbound Network Export."""
 
 __version__ = "0.1.0"
-
-
-def example_function() -> str:
-    """Return a greeting message.
-
-    Returns:
-        A greeting string.
-    """
-    return "Hello from piccione!"

src/piccione/download/__init__.py

@@ -0,0 +1,126 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025, Arcangelo Massari <[email protected]>
+#
+# Permission to use, copy, modify, and/or distribute this software for any purpose
+# with or without fee is hereby granted, provided that the above copyright notice
+# and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED 'AS IS' AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+# REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT,
+# OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE,
+# DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
+# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
+# SOFTWARE.
+
+"""
+Download files from a Figshare article using the Figshare API.
+
+This script downloads all files associated with a Figshare article ID.
+It uses the public Figshare API which works reliably unlike direct wget/curl
+on Figshare URLs.
+"""
+
+import argparse
+import hashlib
+import sys
+from pathlib import Path
+
+import requests
+from tqdm import tqdm
+
+BASE_URL = "https://api.figshare.com/v2"
+CHUNK_SIZE = 8192
+
+
+def get_article_metadata(article_id):
+    """Retrieve article metadata from Figshare API."""
+    url = f"{BASE_URL}/articles/{article_id}"
+    response = requests.get(url)
+    response.raise_for_status()
+    article_data = response.json()
+
+    # Figshare API has a default limit of 10 files. We need to fetch files separately with pagination.
+    files_url = f"{BASE_URL}/articles/{article_id}/files"
+    files_response = requests.get(files_url, params={"page_size": 1000})
+    files_response.raise_for_status()
+    article_data['files'] = files_response.json()
+
+    return article_data
+
+
+def download_file(download_url, output_path, expected_size, expected_md5=None):
+    """Download a file from URL with progress bar and optional MD5 verification."""
+    response = requests.get(download_url, stream=True)
+    response.raise_for_status()
+
+    md5_hash = hashlib.md5()
+
+    with open(output_path, 'wb') as f:
+        with tqdm(total=expected_size, unit='B', unit_scale=True, unit_divisor=1024) as pbar:
+            for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
+                f.write(chunk)
+                md5_hash.update(chunk)
+                pbar.update(len(chunk))
+
+    if expected_md5:
+        actual_md5 = md5_hash.hexdigest()
+        if actual_md5 != expected_md5:
+            raise ValueError(f"MD5 mismatch: expected {expected_md5}, got {actual_md5}")
+        print(f"  MD5 checksum verified: {actual_md5}")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Download files from a Figshare article"
+    )
+    parser.add_argument(
+        "article_id",
+        type=int,
+        help="Figshare article ID"
+    )
+    parser.add_argument(
+        "-o", "--output-dir",
+        type=Path,
+        default=Path("."),
+        help="Output directory for downloaded files (default: current directory)"
+    )
+
+    args = parser.parse_args()
+
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+
+    print(f"Fetching metadata for article {args.article_id}...")
+    metadata = get_article_metadata(args.article_id)
+
+    files = metadata.get("files", [])
+    if not files:
+        print("No files found in this article")
+        return 1
+
+    print(f"\nFound {len(files)} file(s) to download:")
+    for f in files:
+        size_mb = f['size'] / (1024 * 1024)
+        print(f"  - {f['name']} ({size_mb:.2f} MB)")
+
+    print(f"\nDownloading to: {args.output_dir.absolute()}\n")
+
+    for file_info in files:
+        filename = file_info['name']
+        download_url = file_info['download_url']
+        size = file_info['size']
+        md5 = file_info.get('supplied_md5')
+
+        output_path = args.output_dir / filename
+
+        print(f"Downloading {filename}...")
+        download_file(download_url, output_path, size, md5)
+        print(f"  Saved to {output_path}\n")
+
+    print("All files downloaded successfully")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/piccione/download/from_figshare.py

@@ -0,0 +1,55 @@
+from typing import Set
+
+import redis
+from redis.exceptions import ConnectionError as RedisConnectionError
+
+
+class CacheManager:
+    REDIS_KEY = "processed_files"
+
+    def __init__(
+        self,
+        redis_host: str = "localhost",
+        redis_port: int = 6379,
+        redis_db: int = 4,
+    ):
+        self._redis = None
+        self.redis_host = redis_host
+        self.redis_port = redis_port
+        self.redis_db = redis_db
+        self.processed_files: Set[str] = set()
+
+        self._init_cache()
+
+    def _init_redis(self) -> None:
+        """Initialize Redis connection."""
+        try:
+            self._redis = redis.Redis(
+                host=self.redis_host,
+                port=self.redis_port,
+                db=self.redis_db,
+                decode_responses=True,
+            )
+            self._redis.ping()
+        except RedisConnectionError:
+            raise RuntimeError("Redis is not available. Cache requires Redis.")
+
+    def _init_cache(self) -> None:
+        """Initialize cache from Redis."""
+        self._init_redis()
+        existing_redis_files = self._redis.smembers(self.REDIS_KEY)
+        self.processed_files.update(existing_redis_files)
+
+    def add(self, filename: str) -> None:
+        """Add a file to the cache."""
+        self.processed_files.add(filename)
+        self._redis.sadd(self.REDIS_KEY, filename)
+
+    def __contains__(self, filename: str) -> bool:
+        """Check if a file is in the cache."""
+        return filename in self.processed_files
+
+    def get_all(self) -> Set[str]:
+        """Return all files in the cache."""
+        self.processed_files.update(self._redis.smembers(self.REDIS_KEY))
+        return self.processed_files