No live upstreams while connecting to upstream nginx ошибка

abnormal

upstream server temporarily disabled while connecting to upstream

no live upstreams while connecting to upstream

Max_fails and fail_timeout

Max_fails defaults to 1 and 1,fail_timeout defaults to 10 seconds.

Nginx can set the number of failed attempts and failure time of nodes by setting max _ failures (maximum number of failed attempts) and fail_timeout (failure time, after reaching the maximum number of failed attempts, the nodes are set to fail within the time range of fail_timeout, and the nodes do not recover within this time unless all nodes fail) . When the maximum number of attempts is exceeded or the failure time does not exceed the configured failure time, nginx will set the node state to a failure state, and nginx will not connect the backend until the failure time is exceeded or all nodes fail, and then the node is reset to valid and re-detected.

upstream backend {
    server backend1.example.com weight=5;
    server 127.0.0.1:8080       max_fails=3 fail_timeout=30s;
    server unix:/tmp/backend3;

    server backup1.example.com  backup;
}

Fail criteria

such as

connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "POST /demo HTTP/1.1", subrequest: "/capture/getstatus", upstream: "http://192.168.99.100:8080/api/demo/

such as

upstream timed out (110: Connection timed out) while reading response header from upstream

Nginx judges the failed node status by default based on the connect refuse and time out status, and does not judge failure based on HTTP error status. As long as HTTP can return status indicating that the node can still connect normally, Nginx judges whether it is still alive. Unless proxy_next_upstream instruction is added to set 404, 502, 503, 504, 500, time out and other errors to be transferred to standby machine for processing, fails will be accumulated during next_upstream. If standby machine processing is still wrong, error information will be directly returned (but 404 will not record the number of errors. If the error state is not configured or recorded), overview, nginx records the number of errors only in the six states of timeout, connect refuse, 502, 500, 503, 504. timeout and connect refuse are always recorded error states. However, nginx will only record these four HTTP errors into fails after configuring proxy_next_upstream. When fails is greater than or equal to max_fails, the node fails.

Detection mechanism

If all nodes are detected to be invalid and the standby machine is also invalid, then nginx will resume to be valid for all nodes and try to detect valid nodes again. If valid nodes are detected, the correct node contents will be returned. If all the nodes are still wrong, the detection will continue. If there is no correct information, the default return status for node failure is 502, but the next time the node is visited, the correct node will continue to be detected until the correct one is found.

Laboratory log

upstream test_server{
        server 192.168.99.100:80801;
        server 192.168.99.100:80802;
        server 192.168.99.100:80803;
    }
##for capture
location /api/test/demo{
            proxy_pass http://test_server/api/demo;
}    
location /api/demo{
            default_type application/json;
            content_by_lua_file conf/lua/demo.lua;
}

lua

local cjson = require "cjson.safe"
testres = ngx.location.capture("/api/test/demo",{
    method= ngx.HTTP_POST,
    body = "arg1=xxxx&arg2=xxxxx"
})
ngx.log(ngx.ERR,"status"..testres.status)
local testbody = cjson.decode(testres.body)
ngx.log(ngx.ERR,testbody==nil)

Request 192.168.99.100:8080/api/demo , lua in it will initiate a capture, Request /api/test/demo

Request once

2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80801/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80801/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80802/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80802/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80803/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [warn] 5#5: *1 upstream server temporarily disabled while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://192.168.99.100:80803/api/demo", host: "192.168.99.100:8080"
2017/02/09 14:48:57 [error] 5#5: *1 [lua] demo.lua:44: status502 while sending to client, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", host: "192.168.99.100:8080"

If each request for upstream fails, capture’s subrequest returns 502, and the status code returned for client depends on the lua script

Ask again

2017/02/09 15:09:34 [error] 6#6: *11 no live upstreams while connecting to upstream, client: 192.168.99.1, server: , request: "POST /api/demo HTTP/1.1", subrequest: "/api/test/demo", upstream: "http://test_server/api/demo", host: "192.168.99.100:8080"

No live upstreams while connecting to upstreams when server below the upstreams are all hung up.

doc

• ngx_http_upstream_module

• In-depth Analysis of health_check Mechanism in nginx

• Original fault-tolerant mechanism of nginx upstream-Hu Zhiguang

• An analysis of “no live upstreams while connecting to upstreams” in online nginx.