Reddit links not fetching

[murtock]

I get "Failed to fetch link content" for all Reddit links. When i Download Full Page Archive it works.
All other Links from different sites also work.

Anyone got a similar problem?

[MohamedBassem]

Can you share logs from the container when fetching a reddit?

[murtock]

2025-02-17T09:29:04.318Z info: [search][314] Attempting to index bookmark with id pxfj9ugcdvihsq7zrw6wklma ... 2025-02-17T09:29:04.365Z info: [webhook][315] Starting a webhook job for bookmark with id "pxfj9ugcdvihsq7zrw6wklma" 2025-02-17T09:29:04.365Z info: [webhook][315] Completed successfully 2025-02-17T09:29:04.436Z info: [Crawler][313] Will crawl "https://www.reddit.com/r/AskReddit/comments/1ir8idw/americans_what_is_your_opinion_on_canadians/" for link with id "pxfj9ugcdvihsq7zrw6wklma" 2025-02-17T09:29:04.436Z info: [Crawler][313] Attempting to determine the content-type for the url https://www.reddit.com/r/AskReddit/comments/1ir8idw/americans_what_is_your_opinion_on_canadians/ 2025-02-17T09:29:04.610Z info: [search][314] Completed successfully 2025-02-17T09:29:04.688Z info: [Crawler][313] Content-type for the url https://www.reddit.com/r/AskReddit/comments/1ir8idw/americans_what_is_your_opinion_on_canadians/ is "text/html; charset=utf-8" 2025-02-17T09:29:08.362Z info: [Crawler][313] Successfully navigated to "https://www.reddit.com/r/AskReddit/comments/1ir8idw/americans_what_is_your_opinion_on_canadians/". Waiting for the page to load ... 2025-02-17T09:29:13.362Z info: [Crawler][313] Finished waiting for the page to load. 2025-02-17T09:31:04.433Z error: [Crawler][313] Crawling job failed: Error: Timed-out after 120 secs Error: Timed-out after 120 secs at Timeout._onTimeout (/app/apps/workers/utils.ts:2:1025) at listOnTimeout (node:internal/timers:594:17) at process.processTimers (node:internal/timers:529:7)

[MohamedBassem]

What machine are you currently running hoarder on? It seems that the crawler is unable to load the page content, which usually happens if the chrome container is running low on resources.

[murtock]

Running it in Portainer on Truenas. I only have 16GB RAM at the moment so maybe this is the issue? I will upgrade the system soon, so maybe then it will work.
As for the issue above, this time Hoarder managed to fetch everything on the 3rd try. Most of the times it wont work though.

[MohamedBassem]

@murtock 16GB is waay more than what hoarder needs. Can you try restarting the chrome container and see if it makes things any better?

[murtock]

Not making it better no.

[murtock]

These are the logs in the chrome container:

[0217/094644.610340:WARNING:bluez_dbus_manager.cc(248)] Floss manager not present, cannot set Floss enable/disable. [0217/094644.624470:WARNING:dns_config_service_linux.cc(427)] Failed to read DnsConfig. [0217/094645.078095:WARNING:sandbox_linux.cc(418)] InitializeSandbox() called with multiple threads in process gpu-process. [0217/094648.104267:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/094848.157433:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/095048.221541:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/095248.414671:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/095448.557844:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/095648.836456:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/095848.736808:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100048.818043:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100144.696595:INFO:policy_logger.cc(145)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping mandatory platform policies because no policy file was found at: /etc/chromium/policies/managed [0217/100144.696686:INFO:policy_logger.cc(145)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping recommended platform policies because no policy file was found at: /etc/chromium/policies/recommended [0217/100248.951897:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100449.125231:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100649.155650:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100849.362667:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/100854.834021:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory [0217/100854.834161:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory [0217/100854.834545:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory [0217/100854.836540:WARNING:dns_config_service_linux.cc(427)] Failed to read DnsConfig. [0217/100854.843008:INFO:policy_logger.cc(145)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping mandatory platform policies because no policy file was found at: /etc/chromium/policies/managed [0217/100854.843226:INFO:policy_logger.cc(145)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping recommended platform policies because no policy file was found at: /etc/chromium/policies/recommended DevTools listening on ws://0.0.0.0:9222/devtools/browser/3898945c-d123-4066-b4ca-5c83bddc6773 [0217/100854.845466:WARNING:bluez_dbus_manager.cc(248)] Floss manager not present, cannot set Floss enable/disable. [0217/100854.871346:WARNING:sandbox_linux.cc(418)] InitializeSandbox() called with multiple threads in process gpu-process. [0217/100854.927415:WARNING:dns_config_service_linux.cc(427)] Failed to read DnsConfig. [0217/100904.898846:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition. [0217/101105.106112:WARNING:runtime_features.cc(728)] AttributionReportingCrossAppWeb cannot be enabled in this configuration. Use --enable-features=ConversionMeasurement,AttributionReportingCrossAppWeb in addition.

[rizkyarlin]

This one have bothering me for quite some time. I save a lot of reddit content and I have to manually edit the title and description every time.

I don't see any solutions other than using Reddit API Key.

How about we create some kind of programmable pipelines in the admin area? The trigger will be specified URLs pattern.

[rizkyarlin]

I was in the 0.21 version. Now that I upgraded to 0.23, I can just connect it to my n8n via webhook.

[allout58]

Can you give more details on what you did? It's annoying that reddit just doesn't work ☹️

[Xurbax]

Yes please, let us know exactly what you did, rizkyarlin!
For the time being, I have taken to changing any reddit link to "old.reddit.com" before saving it - this give me decent results.

[akohlsmith]

manually changing to old.reddit.com still archives a "blocked" page for me, which is better than the http 403 I was getting without it

[lameslime]

My problem is a bit different, it archives successfully, but it only shown reddit cookies prompt.

[PoloDePoli]

I have the same problem and as far as I can gather what is needed to solve the issue is to login into reddit using my account.
However the chrome version used by Karakeep is headless so I have no idea how to do the actual logging in to resolve the issue

[kriskras69]

I use the singlefile extension and it grabs reddit just fine for me, worth the try?
It also stores a full copy of the website.
Respect!