docs: refine information, improve structure and add new pages (#908)

* docs: refine information, improve structure and add new pages

* review docs in depth

* check landing pages

* higher eval floor after #911

Author: adbar

.readthedocs.yaml

@@ -1,35 +1,19 @@
 # Read the Docs configuration file for Sphinx projects
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
 
-# Required
 version: 2
 
-# Set the OS, Python version and other tools you might need
 build:
   os: ubuntu-24.04
   tools:
     python: "3.13"
-    # You can also specify other tool versions:
-    # nodejs: "20"
-    # rust: "1.70"
-    # golang: "1.20"
 
-# Build documentation in the "docs/" directory with Sphinx
 sphinx:
   configuration: docs/conf.py
-  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
-  # builder: "dirhtml"
-  # Fail on all warnings to avoid broken references
-  # fail_on_warning: true
 
-# Optionally build your docs in additional formats such as PDF and ePub
-# formats:
-#    - pdf
-#    - epub
+formats:
+  - pdf
 
-# Optional but recommended, declare the Python requirements required
-# to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
 python:
   install:
     - method: pip

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-## How to contribute
+# How to contribute
 
 
 If you value this software or depend on it for your product,
@@ -8,15 +8,15 @@ Your support will help ensure the sustainability and growth of the project.
 There are many ways to contribute:
 
   * Sponsor the project: Show your appreciation [on GitHub](https://github.com/sponsors/adbar) or [ko-fi.com](https://ko-fi.com/adbarbaresi).
-  * Find bugs and submit bug reports: Help making Trafilatura an even more robust tool.
+  * Find bugs and submit bug reports: Help make Trafilatura an even more robust tool.
   * Write code: Fix bugs or add new features by writing [pull requests](https://docs.github.com/en/pull-requests) with a list of what you have done.
   * Improve the documentation: Write tutorials and guides, correct mistakes, or translate existing content.
   * Submit feature requests: Share your feedback and suggestions.
 
 
 Here are some important resources:
 
-  * [List of currently open issues](https://github.com/adbar/trafilatura/issues) (no pretension to exhaustivity!)
+  * [List of currently open issues](https://github.com/adbar/trafilatura/issues) (not exhaustive!)
   * [How to contribute to open source](https://opensource.guide/how-to-contribute/)
 
 A special thanks to all the [contributors](https://github.com/adbar/trafilatura/graphs/contributors) who have played a part in Trafilatura.

README.md

@@ -1,4 +1,4 @@
-# Trafilatura: Discover and Extract Text Data on the Web
+# Trafilatura: Discover Web Content, Extract Text and Metadata
 
 <br/>
 
@@ -13,81 +13,72 @@
 [![Downloads](https://static.pepy.tech/badge/trafilatura/month)](https://pepy.tech/project/trafilatura)
 [![Reference DOI: 10.18653/v1/2021.acl-demo.15](https://img.shields.io/badge/DOI-10.18653%2Fv1%2F2021.acl--demo.15-blue)](https://aclanthology.org/2021.acl-demo.15/)
 
-<br/>
-
-<img alt="Demo as GIF image" src="https://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-demo.gif" align="center" width="80%"/>
 
-<br/>
+``` python
+>>> from trafilatura import fetch_url, extract
+>>> downloaded = fetch_url("https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/")
+>>> extract(downloaded)
+'Erin Spiceland is a software engineer and ...'
+>>> extract(downloaded, output_format="json", with_metadata=True)
+'{"title": "Leader spotlight: Erin Spiceland", "author": "Jessica Rudder", "text": "Erin Spiceland is a software engineer and ..."}'
+```
 
 
 ## Introduction
 
-Trafilatura is a cutting-edge **Python package and command-line tool**
-designed to **gather text on the Web and simplify the process of turning
-raw HTML into structured, meaningful data**. It includes all necessary
+Trafilatura is a comprehensive **Python package and command-line tool**
+designed to **gather text from the Web and turn raw HTML into structured,
+meaningful data**. It includes all necessary
 discovery and text processing components to perform **web crawling,
 downloads, scraping, and extraction** of main texts, metadata and
 comments. It aims at staying **handy and modular**: no database is
 required, the output can be converted to commonly used formats.
 
-Going from HTML bulk to essential parts can alleviate many problems
-related to text quality, by **focusing on the actual content**,
-**avoiding the noise** caused by recurring elements like headers and footers
-and by **making sense of the data and metadata** with selected information.
+Going from raw HTML to essential parts, it **focuses on the actual
+content**, **avoids noise** caused by recurring elements (headers,
+footers, boilerplate), and **makes sense of the data and metadata**.
 The extractor strikes a balance between limiting noise (precision) and
 including all valid parts (recall). It is **robust and reasonably fast**.
 
 Trafilatura is [widely used](https://trafilatura.readthedocs.io/en/latest/used-by.html)
-and integrated into [thousands of projects](https://github.com/adbar/trafilatura/network/dependents)
-by companies like HuggingFace, IBM, and Microsoft Research as well as institutions like
-the Allen Institute, Stanford, the Tokyo Institute of Technology, and
-the University of Munich.
+across [thousands of projects](https://github.com/adbar/trafilatura/network/dependents),
+including by HuggingFace, IBM, Microsoft Research, NVIDIA, the Allen Institute for AI, Stanford, and the Internet Archive.
 
 
 ### Features
 
 - Advanced web crawling and text discovery:
-   - Support for sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS)
-   - Smart crawling and URL management (filtering and deduplication)
+  - Support for sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS)
+  - Smart crawling and URL management (filtering and deduplication)
 
 - Parallel processing of online and offline input:
-   - Live URLs, efficient and polite processing of download queues
-   - Previously downloaded HTML files and parsed HTML trees
+  - Live URLs, efficient and polite processing of download queues
+  - Previously downloaded HTML files and parsed HTML trees
 
 - Robust and configurable extraction of key elements:
-   - Main text (common patterns and generic algorithms like jusText and readability)
-   - Metadata (title, author, date, site name, categories and tags)
-   - Formatting and structure: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting
-   - Optional elements: comments, links, images, tables
+  - Main text (own rule-based extractor with jusText and readability-lxml as fallbacks)
+  - Metadata (title, author, date, site name, categories and tags)
+  - Formatting and structure: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting
+  - Optional elements: comments, links, images, tables
+  - Optional add-ons: language detection, speed optimizations
 
 - Multiple output formats:
-   - TXT and Markdown
-   - CSV
-   - JSON
-   - HTML, XML and [XML-TEI](https://tei-c.org/)
+  - TXT and Markdown
+  - CSV
+  - JSON
+  - HTML, XML and [XML-TEI](https://tei-c.org/)
 
-- Optional add-ons:
-   - Language detection on extracted content
-   - Speed optimizations
 
-- Actively maintained with support from the open-source community:
-   - Regular updates, feature additions, and optimizations
-   - Comprehensive documentation
-
-
-### Evaluation and alternatives
+### Evaluation
 
 Trafilatura consistently outperforms other open-source libraries in text
-extraction benchmarks, showcasing its efficiency and accuracy in
-extracting web content. The extractor tries to strike a balance between
-limiting noise and including all valid parts.
-
-For more information see the [benchmark section](https://trafilatura.readthedocs.io/en/latest/evaluation.html)
+extraction benchmarks. For more information see the
+[benchmark section](https://trafilatura.readthedocs.io/en/latest/evaluation.html)
 and the [evaluation readme](https://github.com/adbar/trafilatura/blob/master/tests/README.rst)
 to run the evaluation with the latest data and packages.
 
 
-#### Other evaluations:
+#### Other evaluations
 
 - Most efficient open-source library in *ScrapingHub*'s [article extraction benchmark](https://github.com/scrapinghub/article-extraction-benchmark)
 - Best overall tool according to [Bien choisir son outil d'extraction de contenu à partir du Web](https://hal.archives-ouvertes.fr/hal-02768510v3/document)
@@ -96,7 +87,7 @@ to run the evaluation with the latest data and packages.
   (Bevendorff et al. 2023)
 
 
-## Usage and documentation
+## Documentation
 
 [Getting started with Trafilatura](https://trafilatura.readthedocs.io/en/latest/quickstart.html)
 is straightforward. For more information and detailed guides, visit
@@ -108,12 +99,10 @@ is straightforward. For more information and detailed guides, visit
   [With Python](https://trafilatura.readthedocs.io/en/latest/usage-python.html),
   [With R](https://trafilatura.readthedocs.io/en/latest/usage-r.html)
 - [Core Python functions](https://trafilatura.readthedocs.io/en/latest/corefunctions.html)
-- Interactive Python Notebook: [Trafilatura Overview](docs/Trafilatura_Overview.ipynb)
+- Interactive Python Notebook: [Trafilatura Overview](docs/Trafilatura_Overview.ipynb) (in the repository)
 - [Tutorials and use cases](https://trafilatura.readthedocs.io/en/latest/tutorials.html)
 
-Youtube playlist with video tutorials in several languages:
-
-- [Web scraping tutorials and how-tos](https://www.youtube.com/watch?v=8GkiOM17t0Q&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci)
+See the [video tutorials playlist](https://www.youtube.com/watch?v=8GkiOM17t0Q&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci) (multiple languages).
 
 
 ## License
@@ -123,7 +112,7 @@ This package is distributed under the [Apache 2.0 license](https://www.apache.or
 Versions prior to v1.8.0 are under GPLv3+ license.
 
 
-### Contributing
+## Contributing
 
 Contributions of all kinds are welcome. Visit the [Contributing
 page](https://github.com/adbar/trafilatura/blob/master/CONTRIBUTING.md)
@@ -135,18 +124,21 @@ Many thanks to the
 who extended the docs or submitted bug reports, features and bugfixes!
 
 
+## Support
+
+**If you value this software or depend on it for your product, consider
+sponsoring it and contributing to its codebase.** Your support
+[on GitHub](https://github.com/sponsors/adbar) or [ko-fi.com](https://ko-fi.com/adbarbaresi)
+will help maintain and enhance this package.
+
+
 ## Context
 
 This work started as a PhD project at the crossroads of linguistics and
-NLP, this expertise has been instrumental in shaping Trafilatura over
+NLP. This expertise has been instrumental in shaping Trafilatura over
 the years. Initially launched to create text databases for research purposes
 at the Berlin-Brandenburg Academy of Sciences (DWDS and ZDL units),
-this package continues to be maintained but its future depends on community support.
-
-**If you value this software or depend on it for your product, consider
-sponsoring it and contributing to its codebase**. Your support
-[on GitHub](https://github.com/sponsors/adbar) or [ko-fi.com](https://ko-fi.com/adbarbaresi)
-will help maintain and enhance this popular package.
+this package continues to be maintained and its future depends on community support.
 
 *Trafilatura* is an Italian word for [wire
 drawing](https://en.wikipedia.org/wiki/Wire_drawing) symbolizing the
@@ -155,7 +147,7 @@ are formed.
 
 ### Author
 
-Reach out via ia the software repository or the [contact
+Reach out via the software repository or the [contact
 page](https://adrien.barbaresi.eu/) for inquiries, collaborations, or
 feedback. See also social networks for the latest updates.
 
@@ -198,10 +190,8 @@ acquisition. Here is how to cite it:
 Jointly developed plugins and additional packages also contribute to the
 field of web data extraction and analysis:
 
-<img alt="Software ecosystem" src="https://raw.githubusercontent.com/adbar/htmldate/master/docs/software-ecosystem.png" align="center" width="65%"/>
+<img alt="Diagram of trafilatura and related packages (htmldate, courlan, jusText)" src="https://raw.githubusercontent.com/adbar/htmldate/master/docs/software-ecosystem.png" align="center" width="65%"/>
 
 Corresponding posts can be found on [Bits of
 Language](https://adrien.barbaresi.eu/blog/tag/trafilatura.html).
 
-Impressive, you have reached the end of the page: Thank you for your
-interest!