I try to download a file with wget and curl and it is rejected with a 403 error (forbidden).
I can view the file using the web browser on the same machine.
I try again with my browser’s user agent, obtained by http://www.whatsmyuseragent.com. I do this:
wget -U 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' http://...
and
curl -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' http://...
but it is still forbidden. What other reasons might there be for the 403, and what ways can I alter the wget and curl commands to overcome them?
(this is not about being able to get the file — I know I can just save it from my browser; it’s about understanding why the command-line tools work differently)
update
Thanks to all the excellent answers given to this question. The specific problem I had encountered was that the server was checking the referrer. By adding this to the command-line I could get the file using curl and wget.
The server that checked the referrer bounced through a 302 to another location that performed no checks at all, so a curl or wget of that site worked cleanly.
If anyone is interested, this came about because I was reading this page to learn about embedded CSS and was trying to look at the site’s css for an example. The actual URL I was getting trouble with was this and the curl I ended up with is
curl -L -H 'Referer: http://css-tricks.com/forums/topic/font-face-in-base64-is-cross-browser-compatible/' http://cloud.typography.com/610186/691184/css/fonts.css
and the wget is
wget --referer='http://css-tricks.com/forums/topic/font-face-in-base64-is-cross-browser-compatible/' http://cloud.typography.com/610186/691184/css/fonts.css
Very interesting.
I am using the following command:
wget —no-check-certificate «https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet»
I get 403 Forbidden error (it is the expected output), but there is also an accompanying server response as the reason which I am not able to get. I tried with various options — ignoring content length, giving a private PEM certificate, setting user-agent, etc. but still not able to get that output.
Using curl however, the output comes as expected.
curl «https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet»
Wget Output:
--2013-06-07 18:37:13-- https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet
Resolving www.googleapis.com (www.googleapis.com)... 74.125.135.95, 2404:6800:4001:c01::5f
Connecting to www.googleapis.com (www.googleapis.com)|74.125.135.95|:443... connected.
WARNING: The certificate of `www.googleapis.com' is not trusted.
WARNING: The certificate of `www.googleapis.com' hasn't got a known issuer.
HTTP request sent, awaiting response... 403 Forbidden
2013-06-07 18:37:14 ERROR 403: Forbidden.
Curl Output:
{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "dailyLimitExceededUnreg",
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
"extendedHelp": "https://code.google.com/apis/console"
}
],
"code": 403,
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
}
}
The problem is that many of my machines have only wget installed, so this is being an annoying issue to me. I’m almost sure I missed some option which makes wget show the output. If you have any ideas, I’d be happy to try it out.
Thanks.
So, today I was trying to download an entire C programming tutorial from a website, it was splitted in several different html files and I wanted to have it all so I could read it while offline.
My first thought was about using wget to automatically get it all with the following parameters:
#wget -r -np http://www.xxx.com/docs/stuff/yeah/
The output:
–2009-04-08 02:34:59– http://www.xxx.com/docs/stuff/yeah/
Resolving http://www.xxx.com… 75.126.69.23
Connecting to http://www.xxx.com|75.126.69.23|:80… connected.
HTTP request sent, awaiting response… 403 Forbidden
2009-04-08 02:35:00 ERROR 403: Forbidden.
So, you might ask yourself “but then how the hell my browser got the html files without any error?”.
The webserver use a kind of security configuration where they will refuse any “user agent” which is not related to a browser. For example, when you use wget to download the html, the webserver will answer you with “ERROR403: Forbidden”, because it is not a valid browser, it is Wget. I just don’t know yet how it works at the server side, hopefully I will be writing more about it in the next posts.
So now we can try the following parameters in order to break it:
#wget -U firefox -r -np http://www.xxx.com/docs/stuff/yeah/
Where “-U” means “user agent”
It works flawlessly! 😀
Please share with us your experiences about it.
References:
http://www.checkupdown.com/status/E403.html
http://www.gnu.org/software/wget/manual/
You have two unrelated problems.
«403 Forbidden» is a web server error. You have to fix your URL.
When you ran wget https://releases.mattermost.com/X.X.X/mattermost-X.X.X-linux-amd64.tar.gz, you got a 403: Forbidden error. This is an error from the web server and is unrelated to the user account used to download the file.
When you attempt to load a page or download a file and you specify an incorrect URL, a 404: Not Found errors is the most common result. But a 403 error can occur sometimes in this situation, and that appears to be what is happening here. It looks, from both the command you ran and wget‘s output, like you forgot to put the actual version numbers in the URL.
On my system, I am able to get exactly the same error using the URL with X.X.X instead of an actual version number:
ek@Io:~/Downloads$ wget https://releases.mattermost.com/X.X.X/mattermost-X.X.X-linux-amd64.tar.gz
--2017-01-17 03:15:30-- https://releases.mattermost.com/X.X.X/mattermost-X.X.X-linux-amd64.tar.gz
Resolving releases.mattermost.com (releases.mattermost.com)... 54.192.147.30, 54.192.147.18, 54.192.147.46, ...
Connecting to releases.mattermost.com (releases.mattermost.com)|54.192.147.30|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-01-17 03:15:32 ERROR 403: Forbidden.
And if I use the correct URL, it works:
ek@Io:~/Downloads$ wget https://releases.mattermost.com/3.6.0/mattermost-team-3.6.0-linux-amd64.tar.gz
--2017-01-17 03:09:50-- https://releases.mattermost.com/3.6.0/mattermost-team-3.6.0-linux-amd64.tar.gz
Resolving releases.mattermost.com (releases.mattermost.com)... 54.192.147.156, 54.192.147.30, 54.192.147.160, ...
Connecting to releases.mattermost.com (releases.mattermost.com)|54.192.147.156|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27221349 (26M) [application/x-gzip]
Saving to: ‘mattermost-team-3.6.0-linux-amd64.tar.gz’
mattermost-team-3.6.0-linux- 100%[==============================================>] 25.96M 106KB/s in 2m 6s
2017-01-17 03:11:58 (212 KB/s) - ‘mattermost-team-3.6.0-linux-amd64.tar.gz’ saved [27221349/27221349]
(Both the working and non-working URLs are from your post. I don’t actually know exactly what file you need.)
The mattermost user isn’t a sudoer.
Not all users may run programs as root (or at all) with the sudo command. The file /etc/sudoers, or a file in /etc/sudoers.d, must be configured to allow them to do so. Most commonly, this is achieved by putting the user in a group that one of those files permits to run commands.
In particular, to make a user an administrator on an Ubuntu system so that they are permitted to run any command as root using sudo, add the user to the sudo group:
sudo usermod -aG sudo mattermost
Of course, you must run that as a user who can run commands as root with sudo already. (You were able to run sudo passwd mattermost, so I know you have at least one such user.)
Keep in mind:
- You must only do this if you really want that user to have full power to do anything. They will have the same power as you. Once you’ve decided you want that, though, you should go ahead and simply add the user to the
sudoersgroup. There is usually no need to actually edit any configuration files forsudo— they’re already set up to recognize thesudogroup as conferring administrative power. - If for some reason you instead choose to edit
/etc/sudoersor create/edit a file in/etc/sudoers.d, it is important that you use thevisudoutility to do it, because it performs a syntax check to make sure your changes will be understood bysudo. Ifsudosees incorrect syntax in a sudoers file, then it will completely refuse to work. For servers you only have remote access to, this can lock you out of the system.
Finally, it’s very rare that you actually have to manually download a file with wget as root. In this case, you didn’t — the error was an incorrect URL and the «forbidden» message was from the web server. Therefore:
- If this is the only reason mattermost needed to use
sudo, you probably shouldn’t give that user account these abilities, because there’s no need. - Even if mattermost really does need to be an administrator, they should perform actions as themselves (without
sudo) rather than as root (withsudo) except when there is a clear reason to do the latter.
I need to download suricata’s latest rules from this url:
http://rules.emergingthreats.net/blockrules/
I’m trying to compose my own rules based on different emerging rules and snort’s rules too. Trying to replace alert with drop.
I can’t even download all rules, because apache 2.2.22 of rules.emergingthreats.net configured somehow against wget.
My command looks like:
# wget -m --user-agent="Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3" "https://rules.emergingthreats.net/"
I’m trying to mirror whole the site.
--2014-07-22 06:18:49-- https://rules.emergingthreats.net/
Resolving rules.emergingthreats.net (rules.emergingthreats.net)... 204.12.217.19, 96.43.137.99
Connecting to rules.emergingthreats.net (rules.emergingthreats.net)|204.12.217.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘rules.emergingthreats.net/index.html’
[ <=> ] 1,951 --.-K/s in 0s
Last-modified header missing -- time-stamps turned off.
2014-07-22 06:18:50 (23.3 MB/s) - ‘rules.emergingthreats.net/index.html’ saved [1951]
Thereafter, all inner directories give me 403 Forbidden:
--2014-07-22 06:18:51-- https://rules.emergingthreats.net/blockrules/
Reusing existing connection to rules.emergingthreats.net:443.
HTTP request sent, awaiting response... 403 Forbidden
2014-07-22 06:18:51 ERROR 403: Forbidden.
I’m successfully browse the site(directory), I can download any file with my browser — chromium ( Ubuntu ).
Why does wget fail to fetch the data?