cURL - Downloading a site

	Bottom Previous Topic Next Topic
Register To Post

Raziel

Posted on: 2022/8/25 15:22 #1

Home away from home

Can anyone give any hints on how i could download a specific site with cURL?

Could it be a leech/DDoS protection?
I know it works with other sites, so maybe nothing can be done about it as it's site related?

The site is https://www.accuweather.com
cURL version is
Quote:

curl 7.79.1 (powerpc-unknown-amigaos) libcurl/7.79.1 OpenSSL/1.1.1l zlib/1.2.11 librtmp/2.3
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy libz NTLM SSL TLS-SRP


cURL https://www.accuweather.com -o ram:test.html

is just sitting there counting seconds...

Thank you

People are dying.
Entire ecosystems are collapsing.
We are in the beginning of a mass extinction.
And all you can talk about is money and fairytales of eternal economic growth.
How dare you!
– Greta Thunberg

Petrol

Re: cURL - Downloading a site

Posted on: 2022/8/25 16:47 #2

Just popping in

Hi,

Doesn't work with online tools also.
It works with the url : https://api.accuweather.com/ but I don't know if it's what you're looking for.

Regards,

Raziel

Re: cURL - Downloading a site

Posted on: 2022/8/25 16:51 #3

Home away from home

@Petrol

Bad Request - Invalid Header

HTTP Error 400. The request has an invalid header name.

edit:

Ah i see, that's the dev center for AccuWeather...nope, unfortunately not what i'm looking for

Edited by Raziel on 2022/8/25 17:32:29

msteed

Re: cURL - Downloading a site

Posted on: 2022/8/26 7:02 #4

Just popping in

@Raziel

Users on the IBrowse mailing list have noted that cURL and Wget both use old versions of SSL that no longer work with many websites, so perhaps that's the problem. Maybe try the verbose (-v) option and see if that provides any clues.

Raziel

Re: cURL - Downloading a site

Posted on: 2022/8/26 7:36 #5

Home away from home

@msteed

Quote:

APPDIR:cURL -v https://accuweather.com -o test.html
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2.16.104.230:443...
* Connected to accuweather.com (2.16.104.230) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
* CApath: /DEVS/AmiSSL/Certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [35 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3935 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=Pennsylvania; L=State College; O=Accuweather, Inc.; CN=*.accuweather.com
* start date: Aug 18 00:00:00 2022 GMT
* expire date: Sep 4 23:59:59 2023 GMT
* subjectAltName: host "accuweather.com" matched cert's "accuweather.com"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
} [5 bytes data]
> GET / HTTP/1.1
> Host: accuweather.com
> User-Agent: curl/7.79.1
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing

Does that tell anyone anything?

Register To Post
	Top Previous Topic Next Topic

Currently Active Users Viewing This Thread: 1 ( 0 members and 1 Anonymous Users )