You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/evaluation.rst
+44-11Lines changed: 44 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ The extraction focuses on the main content, which is usually the part displayed
21
21
External evaluations
22
22
--------------------
23
23
24
-
- Most efficient open-source library in *ScrapingHub*'s `article extraction benchmark <https://github.com/scrapinghub/article-extraction-benchmark>`_
24
+
- Most efficient open-source library in *ScrapingHub*'s `article extraction benchmark <https://github.com/scrapinghub/article-extraction-benchmark>`_ (older evaluation)
25
25
- Best overall tool according to `Bien choisir son outil d'extraction de contenu à partir du Web <https://hal.archives-ouvertes.fr/hal-02768510v3/document>`_ (Lejeune & Barbaresi 2020)
26
26
- Comparison on a small `sample of Polish news texts and forums <https://github.com/tsolewski/Text_extraction_comparison_PL>`_ (now integrated in the internal benchmark, Trafilatura has improved since)
27
27
- Best single tool by ROUGE-LSum Mean F1 Page Scores in `An Empirical Comparison of Web Content Extraction Algorithms <https://webis.de/downloads/publications/papers/bevendorff_2023b.pdf>`_ (Bevendorff et al. 2023)
@@ -34,6 +34,7 @@ Although a few corresponding Python packages are not actively maintained the fol
34
34
35
35
These packages keep the structure intact but do not focus on main text extraction:
36
36
37
+
- `BeautifulSoup <https://www.crummy.com/software/BeautifulSoup/>`_ is a general-purpose HTML parser, used here to grab all the text in the document
37
38
- `html2text <https://github.com/Alir3z4/html2text>`_ converts HTML pages to Markup language
38
39
- `html_text <https://github.com/TeamHG-Memex/html-text>`_ converts HTML code to plain text
39
40
- `inscriptis <https://github.com/weblyzard/inscriptis>`_ converts HTML to text with a particular emphasis on nested tables
@@ -44,10 +45,12 @@ These packages focus on main text extraction:
44
45
- *dragnet* is not maintained anymore, it is provided for reference only (in older evaluations)
45
46
- `goose3 <https://github.com/goose3/goose3>`_ can extract information for embedded content but doesn't preserve markup
46
47
- `jusText <https://github.com/miso-belica/jusText>`_ is designed to preserve mainly text containing full sentences along with some markup, it has been explicitly developed to create linguistic resources
47
-
- `newspaper3k <https://github.com/codelucas/newspaper>`_ is mostly geared towards newspaper texts, provides additional functions but no structured text or comment extraction
48
+
- `magic-html <https://github.com/opendatalab/magic-html>`_ extracts the main content of a page as HTML, geared towards LLM training data
49
+
- `newspaper4k <https://github.com/AndyTheFactory/newspaper4k>`_ (the maintained successor of *newspaper3k*) is mostly geared towards newspaper texts, provides additional functions but no structured text or comment extraction
48
50
- `news-please <https://github.com/fhamborg/news-please>`_ is a news crawler that extracts structured information
49
51
- `readability-lxml <https://github.com/buriy/python-readability>`_ cleans the page and preserves some markup
50
-
- `readabilipy <https://github.com/alan-turing-institute/ReadabiliPy>`_ contains a Python wrapper for Mozilla's Node.js package, as well as article extraction routines written in pure Python
52
+
- `resiliparse <https://github.com/chatnoir-eu/chatnoir-resiliparse>`_ converts HTML to plain text with a focus on speed and robustness, run here in main-content mode
53
+
- *readabilipy* is not part of the comparison anymore, it is provided for reference only (in older evaluations)
51
54
- `trafilatura <https://github.com/adbar/trafilatura>`_ is the library documented here, several options are tested regarding main text extraction only, without metadata or comments
52
55
53
56
The tools are compared to the raw page source and to a meaningful baseline consisting of extracting the raw text contained in the JSON article element or in a combination of paragraph, code and quote elements.
@@ -60,9 +63,9 @@ Description
60
63
61
64
**Evaluation**: Decisive document segments are singled out which are not statistically representative but very significant in the perspective of working with the texts, most notably left/right columns, additional header, author or footer information such as imprints or addresses, as well as affiliated and social network links, in short boilerplate. Raw text segments are expected which is also a way to evaluate the quality of HTML extraction in itself.
62
65
63
-
**Time**: The execution time is provided as an indication. As the baseline extraction is simple and fast, it is used for the benchmark. Certain packages are noticeably slower than the rest: *goose3* and *newspaper*, while *news-please*'s execution time isn't comparable because of operations unrelated to text extraction. ReadabiliPy is very slow for unclear reasons.
66
+
**Time**: The execution time is provided as an indication. As the baseline extraction is simple and fast, it is used for the benchmark. Certain packages are noticeably slower than the rest: *goose3* and *newspaper*, while *news-please*'s execution time isn't comparable because of operations unrelated to text extraction and is by far the slowest.
64
67
65
-
**Errors**: The *boilerpy3*, *newspaper3k*, and *readabilipy* modules do not work without errors on every HTML file in the test set, probably because of malformed HTML, encoding or parsing bugs. These errors are ignored in order to complete the benchmark.
68
+
**Errors**: The *boilerpy3*and *newspaper4k* modules do not work without errors on every HTML file in the test set, probably because of malformed HTML, encoding or parsing bugs. These errors are ignored in order to complete the benchmark.
66
69
67
70
**Results**: The baseline beats a few systems, showing its interest. *justext* is highly configurable and tweaking its configuration (as it is done here) can lead to better performance than its generic settings. *goose3* is the most precise algorithm, albeit at a significant cost in terms of recall. The packages focusing on raw text extraction *html_text* and *inscriptis* are roughly comparable and achieve the best recall as they try to extract all the text. Rule-based approaches such as *trafilatura*'s obtain balanced results despite a lack of precision. Combined with an algorithmic approach they perform significantly better than the other tested solutions. Trafilatura consistently outperforms other open-source libraries, showcasing its efficiency and accuracy in extracting web content.
68
71
@@ -71,9 +74,44 @@ Description
71
74
The evaluation script is available on the project repository: `tests/README.rst <https://github.com/adbar/trafilatura/blob/master/tests/>`_. To reproduce the tests just clone the repository, install all necessary packages and run the evaluation script with the data provided in the *tests* directory.
Each package receives the raw HTML bytes and handles character encoding itself; packages evaluated on string input (*boilerpy3*, *html2text*, *html_text*, *inscriptis*, *magic-html*, *news-please*, *readability-lxml*) get it pre-decoded by the benchmark, with the conversion counted in their execution time.
Copy file name to clipboardExpand all lines: tests/README.rst
+20-8Lines changed: 20 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,21 +28,33 @@ Running the code
28
28
The results and a list of comparable benchmarks are available on the `evaluation page of the docs <https://trafilatura.readthedocs.io/en/latest/evaluation.html>`_.
29
29
30
30
31
-
Evaluation
32
-
----------
31
+
Quality gate
32
+
------------
33
33
34
34
The following allows for comparing changes made to Trafilatura, for example in a new version or pull request:
35
35
36
-
1. Install Trafilatura
37
-
2. Run the script ``comparison_small.py``
36
+
1. Install Trafilatura from the working tree: ``pip install -e ".[all]"`` (from the repository root; the ``all`` extra matches the environment of the CI gate, plain ``pip install -e .`` can score slightly differently on non-UTF-8 pages)
37
+
2. Run ``python tests/eval_gate.py``
38
+
39
+
``eval_gate.py`` scores the whole corpus with Trafilatura alone and compares the F1-scores with the floors pinned in ``eval_baseline.json``, exiting non-zero on a regression. It needs no competitor library and is also run in CI.
40
+
41
+
After editing the annotations or an HTML input, re-pin the corpus fingerprint with ``python tests/eval_gate.py --update``. A re-pin never lowers the baseline on its own: an F1 below a pinned floor keeps the floor and exits non-zero, and accepting a lower bar takes an explicit ``--allow-regression``.
38
42
43
+
Note for Windows: the corpus fingerprint requires the HTML inputs exactly as committed. On a clone made before the ``.gitattributes`` rules were added, run ``git add --renormalize .`` and reset (or re-clone) so line endings match the repository.
39
44
40
-
A comparison with similar software is run periodically. As the packages tend to evolve the script may not always be up-to-date and all packages may not be available. If that happens, commenting out the corresponding sections is the most efficient solution. Fixes to the file can be submitted as pull requests.
45
+
46
+
Comparison with other software
47
+
------------------------------
48
+
49
+
``evaluate.py`` additionally runs other extractors. Each competitor library is imported by the algorithm that uses it, not at module level, so ``pip install -e ".[eval]"`` is enough to get started and an algorithm whose library is missing or does not import is reported and dropped from the comparison rather than stopping the run. ``--small`` needs no competitor at all.
41
50
42
51
Note: As numerous packages are installed it is recommended to create a virtual environment, for example with ``pyenv`` or ``venv``.
1. Install the evaluation dependencies from the repository root: ``pip install -e ".[eval]"`` (``magic-html`` requires Python 3.12+ and is skipped on older versions)
54
+
2. For ``news-please``, download the NLTK tokenizer data once: ``python -m nltk.downloader punkt punkt_tab`` (otherwise it attempts a network download per document)
55
+
3. Run the script ``evaluate.py``
56
+
57
+
The published results record the version of each package next to its name, since the packages evolve and their output changes with them.
46
58
47
59
Options:
48
60
@@ -52,7 +64,7 @@ Options:
52
64
53
65
``python3 evaluate.py --help``: Display all algorithms and further options.
54
66
55
-
More comprehensive evaluations are available, mostly focusing on English and/or a particular text type. With minimal adaptations, the evaluation can support the use gold standard files in JSON format.
67
+
More comprehensive evaluations are available, mostly focusing on English and/or a particular text type. The evaluation only supports the handcrafted with/without segment format described above; an external annotation file can be passed with ``--testfile`` as long as its HTML documents live in ``tests/cache`` or ``tests/eval``.
0 commit comments