No healthy upstream ошибка что значит

get request errors: «no healthy upstream» #9050

Title: get request errors: «no healthy upstream»

Dynamic configuration discovery through control panel.
In step:
2019-11-15 19:00:43: update cluster timeout and change config version.
2019-11-15 21:17:03: get A lot of request errors, grpc-status: 14, grpc-message: no healthy upstream
Deploying 21 envoy nodes, two of them had this error.
Envoy restart or reload node returns to normal.
Envoy version: 1.11.2

[2019-11-15 19:00:43.206][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream conne
ct error or disconnect/reset before headers. reset reason: connection termination
[2019-11-15 19:03:03.399][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream conne
ct error or disconnect/reset before headers. reset reason: connection termination
[2019-11-15 19:08:36.138][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-11-15 19:08:36.620][23415][info][upstream] [source/server/lds_api.cc:60] lds: add/update listener ‘grpc-listener’
[2019-11-15 19:08:36.621][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster app.xxx starting warming
[2019-11-15 19:08:36.625][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster app.xxx starting warming
[2019-11-15 19:08:36.626][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster app.xxx starting warming
[2019-11-15 19:08:36.626][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster app.xxx starting warming
[2019-11-15 19:08:36.627][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:495] add/update cluster app.xxx starting warming
[2019-11-15 19:08:36.627][23415][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 19:08:36.627][23415][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 19:08:36.627][23415][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 19:08:36.627][23415][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 19:08:36.627][23415][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 19:08:52.511][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2019-11-15 21:01:22.203][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-11-15 21:17:03.937][23415][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-11-15 21:17:03.937][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:507] warming cluster app.xxx complete
[2019-11-15 21:17:03.938][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:507] warming cluster app.xxx complete
[2019-11-15 21:17:03.938][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:507] warming cluster app.xxx complete
[2019-11-15 21:17:03.938][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:507] warming cluster app.xxx complete
[2019-11-15 21:17:03.939][23415][info][upstream] [source/common/upstream/cluster_manager_impl.cc:507] warming cluster app.xxx complete
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:238] initializing epoch 7 (hot restart version=11.104)
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:240] statically linked extensions:
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:242] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:245] filters.http: envoy.buffer,envoy.cors,envoy.csrf,envoy.ext_authz,envoy.fault,envoy.filters.http.dynamic_forward_proxy,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.original_src,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:248] filters.listener: envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2019-11-15 21:41:03.451][3324][info][main] [source/server/server.cc:251] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-11-15 21:41:03.452][3324][info][main] [source/server/server.cc:253] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2019-11-15 21:41:03.452][3324][info][main] [source/server/server.cc:255] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.tracers.opencensus,envoy.zipkin
[2019-11-15 21:41:03.452][3324][info][main] [source/server/server.cc:258] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2019-11-15 21:41:03.452][3324][info][main] [source/server/server.cc:261] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2019-11-15 21:41:03.452][3324][info][main] [source/server/server.cc:267] buffer implementation: old (libevent)
[2019-11-15 21:41:03.458][23415][warning][main] [source/server/server.cc:574] shutting down admin due to child startup
[2019-11-15 21:41:03.458][23415][warning][main] [source/server/server.cc:580] terminating parent process
[2019-11-15 21:41:03.459][3324][info][main] [source/server/server.cc:322] admin address: 0.0.0.0:9901
[2019-11-15 21:41:03.460][3324][info][main] [source/server/server.cc:432] runtime: layers:

• name: base
  static_layer:
  <>
• name: admin
  admin_layer:
  <>
  [2019-11-15 21:41:03.460][3324][warning][runtime] [source/common/runtime/runtime_impl.cc:497] Skipping unsupported runtime layer: name: «base»
  static_layer <
  >

[2019-11-15 21:41:03.460][3324][info][config] [source/server/configuration_impl.cc:61] loading 0 static secret(s)
[2019-11-15 21:41:03.460][3324][info][config] [source/server/configuration_impl.cc:67] loading 2 cluster(s)
[2019-11-15 21:41:03.462][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:124] cm init: initializing secondary clusters
[2019-11-15 21:41:03.463][3324][info][config] [source/server/configuration_impl.cc:71] loading 0 listener(s)
[2019-11-15 21:41:03.463][3324][info][config] [source/server/configuration_impl.cc:96] loading tracing configuration
[2019-11-15 21:41:03.463][3324][info][config] [source/server/configuration_impl.cc:116] loading stats sink configuration
[2019-11-15 21:41:03.463][3324][info][main] [source/server/server.cc:516] starting main dispatch loop
[2019-11-15 21:41:03.468][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:144] cm init: initializing cds
[2019-11-15 21:41:03.469][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:489] add/update cluster app.xxx during init
[2019-11-15 21:41:03.470][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:489] add/update cluster app.xxx during init
[2019-11-15 21:41:03.471][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:489] add/update cluster app.xxx during init
[2019-11-15 21:41:03.471][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:489] add/update cluster app.xxx during init
[2019-11-15 21:41:03.472][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:489] add/update cluster app.xxx during init
[2019-11-15 21:41:03.472][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:124] cm init: initializing secondary clusters
[2019-11-15 21:41:03.475][3324][info][upstream] [source/common/upstream/cluster_manager_impl.cc:148] cm init: all clusters initialized
[2019-11-15 21:41:03.475][3324][info][main] [source/server/server.cc:500] all clusters initialized. initializing init manager
[2019-11-15 21:41:03.479][3324][info][upstream] [source/server/lds_api.cc:60] lds: add/update listener ‘grpc-listener’
[2019-11-15 21:41:03.480][3324][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 21:41:03.480][3324][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 21:41:03.480][3324][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 21:41:03.481][3324][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 21:41:03.481][3324][warning][misc] [source/common/protobuf/utility.cc:199] Using deprecated option ‘envoy.api.v2.route.CorsPolicy.allow_origin_regex’ from file route.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-11-15 21:41:03.481][3324][info][config] [source/server/listener_manager_impl.cc:761] all dependencies initialized. starting workers

Normal node:

Abnormal node:

The text was updated successfully, but these errors were encountered:

Источник

2 способа навсегда исправить ошибку «Нет работоспособного восходящего потока»

Получение сообщения об ошибке «Нет работоспособного восходящего потока» никогда не бывает приятным. Все, что вам нужно сделать, это сесть, расслабиться и получать удовольствие от использования вашего любимого веб-сайта.

Однако ваши планы могут быть разрушены, когда вы получите это сообщение. Это проблема, связанная с сервером, и чтение этой статьи даст вам больше информации об альтернативе Windows вашему существующему серверу.

Если вы задавались вопросом, как исправить эту ошибку, вам повезло. Ниже приведены несколько общих решений, которые помогут вам навсегда исправить эту ошибку.

Что означает отсутствие здорового восходящего потока?

Восходящий поток определяется в разработке программного обеспечения как действие по отправке исправления или пакета администратору для включения в кодовую базу этого программного обеспечения.

Даже если вы столкнулись с этим сообщением об ошибке на своем компьютере, вы можете даже не знать об этом.

Ошибка «No Healthy Upstream» начинается как программная ошибка, которая препятствует работе определенного приложения.

Как я могу исправить ошибку отсутствия работоспособного восходящего потока?

1. Очистите кеш в браузере вашего компьютера

• В браузере нажмите CTRL + SHIFT + DEL .

• Отметьте только кешированные изображения и файлы и нажмите очистить данные.

Для более глубокой очистки браузера используйте CCleaner. Он сканирует ваш браузер, делит ваши данные на более конкретные категории и дает вам обзор всего, что можно безопасно удалить.

2. Перезагрузите компьютер

• Щелкните значок «Пуск».
• Нажмите на значок питания.

• Нажмите «Перезагрузить».

Нет работоспособной ошибки восходящего потока в vCenter

Если нет исправной ошибки восходящего потока, vCenter еще не приспособлен для использования. Поэтому лучше подождать несколько минут, прежде чем получить доступ к vCenter из браузера вашего компьютера.

Недостаточно подготовленный vCenter виноват в большинстве ошибок, связанных с неработоспособным восходящим потоком. Возможно, возникла проблема, которая препятствует правильной работе служб vCenter.

Вы можете исправить эту ошибку в Center следующим образом:

• Выключение Vcenter
• Обновление аппаратной версии виртуальной машины.
• Редактирование настроек Центра. Вы можете сделать это, нажав кнопку параметров виртуальной машины, щелкнув Общие параметры и выбрав ОС VMware Photon.

Убедитесь, что виртуальные машины V7 имеют соответствующую память и вычислительную мощность. Например, vCenter 7 потребляет много ресурсов ЦП и памяти.

Нет здоровой ошибки восходящего потока eBay

В 1995 году американский предприниматель Пьер Омидьяр основал eBay, глобальную онлайн-аукционную и торговую платформу. Ни для кого не секрет, что eBay была одной из первых организаций, запустивших онлайн-рынок, объединяющий потребителей и продавцов.

Малые предприятия и отдельные продавцы одинаково пользуются услугами этого глобального центра электронной коммерции.

Тем не менее, несколько пользователей столкнулись с ошибкой неработоспособного исходящего потока на eBay.

Эта ошибка на компьютере чаще всего вызвана техническими проблемами на eBay и может быть устранена только ими.

Нет здоровой ошибки восходящего потока Spotify

Приложения для потоковой передачи музыки на мобильные устройства стали чрезвычайно интенсивными, но Spotify был на вершине кучи с 2008 года.

Однако даже у Spotify иногда могут возникнуть проблемы.

Если в Spotify нет здоровой ошибки восходящего потока, вот как это исправить.

• Новая поисковая система или сеанс инкогнито/приватный браузер.
• Проверьте, установлена ​​ли последняя версия браузера на вашем компьютере.
• Перезагрузите маршрутизатор.
• Попробуйте другую сеть. Не стесняйтесь связаться с поставщиком услуг предыдущей сети, если он не загружается при вашем новом соединении.
• Услуги могут быть ограничены в общедоступных или совместно используемых сетях (например, в школе или на работе). Вы можете получить дополнительную информацию о сети, связавшись с ответственными за нее людьми.
• Файл вашего хоста также может нуждаться в очистке.

Что такое тайм-аут восходящего потока?

Восходящий поток — это термин компьютерной сети, который относится к передаче данных с локального компьютера или клиента на удаленный хост или сервер.

Вы получаете тайм-аут восходящего потока, когда передача занимает слишком много времени, и система интерпретирует запрос как неудачный.

Мы надеемся, что вы нашли некоторое просветление с этой частью. Пожалуйста, поделитесь своими мыслями в разделе комментариев ниже.

Источник

From my experience, the «no healthy upstream» error can have different causes. Usually, Istio has received ingress traffic that should be forwarded (the client request, or Istio downstream), but the destination is unavailable (istio upstream / kubernetes service). This results in a HTTP 503 «no healthy upstream» error.

1.) Broken Virtualservice definitions
If you have a destination in your VirtualService context where the traffic should be routed, ensure this destination exists (in terms of the hostname is correct, or the service is available from this namespace)

2.) ImagePullBack / Terminating / Service is not available

Ensure your destination is available in general. Sometimes no pod is available, so no upstream will be available too.

3.) ServiceEntry — same destination in 2 lists, but lists with different DNS Rules

Check your namespace for ServiceEntry objects with:

kubectl -n <namespace> get serviceentry

If the result has more than one entry (multiple lines in one ServiceEntry object), check if a destination address (e.g. foo.com) is available in various lines.
If the same destination address (e.g. foo.com) is available in various lines, ensure that the column «DNS» does not have different resolution settings (e.g. one line uses DNS, the other line has NONE). If yes, this is an indicator that you try to apply different DNS settings to the same destination address.

A solution is:

a) to unify the DNS setting, setting all lines to NONE or DNS, but not to mix it up.

b) Ensure the destination (foo.com) is available in one line, and a collision of different DNS rules does not appear.

a) involves restarting istio-ingressgateway pods (data plane) to make it work.

b) Involves no restart of istio data or istio control plane.

Basically: It helps to check the status between Control Plane (istiod) and DatapPlane (istio-ingressgateway) with

istioctl proxy-status

The output of istioctl proxy-status should ensure that the columns say «SYNC» this ensures that the control plane and Data Plane are synced. If not, you can restart the istio-ingressgateway deployment or the istiod daemonset, to force «fresh» processes.

Further, it helped to run

istioctl analyze -A

to ensure that targets are checked in the VirtualService context and do exist. If a virtual service definition exists with routing definitions whose destination is unavailable, istioctl analyze -A can detect these unavailable destinations.

Furthermore, reading the logfiles of the istiod container helps. The istiod error messages often indicate the context of the error in the routing (which namespace and service or istio setting). You can use the default way with

kubectl -n istio-system logs <nameOfIstioDPod>

Referenes:

• https://istio.io/latest/docs/reference/config/networking/service-entry/
• https://istio.io/latest/docs/reference/config/networking/virtual-service/
• https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/

Istio service mesh offers a multitude of solutions at network level 7 (L7) to define traffic routing, security, and application monitoring in a cloud environment. However, given the complexity of cloud-based networks, the host of devices involved, and the difficulty of visualizing effective changes made by Istio, it’s hard to debug the unpopular «no healthy upstream» error messages that often show up in Envoy logs.

This article attempts some pain relief in the form of quick guidance on how to respond to emergency calls demanding a resolution to «no healthy upstream» error messages and related errors such as «Applications in the Mesh are not available» or «Istio is broken.»

In my experience, 90% of these issues are caused by configuration problems in either the network or Istio. This article shows some troubleshooting tools you can use to identify such problems quickly, in the context of two recent cases that a Red Hat customer escalated to us.

It’s important to understand a few aspects of this customer’s architecture. The customer is running services in separate Red Hat OpenShift clusters, some of which are in the customer’s own on-premises infrastructure, while others span several countries in the EU region. Each OpenShift cluster has its own instance of a Red Hat OpenShift Service Mesh, Red Hat’s productized Istio service.

Kubernetes services make both intramesh and intermesh requests. But a service in this customer’s configuration always makes a local call. Integration and routing between services in the different clusters are performed by the mesh via a set of VirtualService, DestinationRule, and ServiceEntry resources that redirect the local call to a remote service.

A duplicate service

In our first real-life example, the customer complained that the service mesh somehow was causing cluster-to-cluster communications to fail, and reported the «no healthy upstream» message.

To identify a problem related to Istio configuration, I always use Istio’s Kiali console to visualize the network state and pinpoint where issues are occurring. Kiali allows you to «play back» network behavior, a nice feature that is very helpful if you’re dealing with a problem that is not occurring right now. Whether or not I discover the problematic service, I turn next to checking the logs of the Envoy proxy via either Kiali or OpenShift (using an oc logs <pod_name> -c istio-proxy commmand). The aim in both cases is to find the service for which the «no healthy upstream» error appears.

In this case, Kiali showed that 95% of the traffic to the destination service destination.mynamespace.svc.cluster.local was failing. My next resource was the istioctl command, which can provide a quick view of the state of the Envoy proxy and whether its configuration was updated correctly by Istio:

$ istioctl proxy-status
NAME CDS LDS EDS RDS PILOT VERSION
...
service-source-v1-74f955bd84-9lmnf.mynamespace SYNCED SYNCED SYNCED SYNCED istiod-86798869b8-bqw7c 1.5.0
...

Getting confirmation from the output that the mesh managed to keep all relevant service Envoy proxies up to date, I then checked the cluster names configured on the Envoy proxy of the client service pod. I focused only on the clusters related to the outbound service host for which logs showed the «no healthy upstream» message:

$ istioctl proxy-config cluster -i istio-system service-source-v1-74f955bd84- 9lmnf.mynamespace --fqdn service-destination.mynamespace.svc.cluster.local -o json | jq -r .[].name
outbound|80||service-destination.mynamespace.svc.cluster.local
inbound|80|9180-tcp|service-destination.mynamespace.svc.cluster.local   outbound|80|v1|service-destination.mynamespace.svc.cluster.local
outbound|80|v2|service-destination.mynamespace.svc.cluster.local

In the output, I noticed that the Istio configuration had defined two services (v1 and v2) for the cluster in question: outbound|80|v2|service-destination.mynamespace.svc.cluster.local. I then checked for the available endpoints for the v2 service:

$ istioctl proxy-config endpoints service-source-v1-74f955bd84-9lmnf.mynamespace --cluster "outbound|80|v2|service-destination.mynamespace.svc.cluster.local"   ENDPOINT STATUS OUTLIER CHECK CLUSTER
172.17.0.28:9180 HEALTHY OK outbound|80|v2|service destination.mynamespace.svc.cluster.local
172.17.0.29:9180 HEALTHY OK outbound|80|v2|service destination.mynamespace.svc.cluster.local

Then I proceeded to check the endpoints for v1. However, for the v1 service for outbound|80|v2|service destination.mynamespace, the mesh has no endpoints, and therefore no pods:

$ istioctl proxy-config endpoints teachstore-course-v1-74f965bd84-8lmnf.development 2
cluster "outbound|80|v`|service-destination.mynamespace.svc.cluster.local"   ENDPOINT STATUS OUTLIER CHECK CLUSTER

This misconfiguration caused the «no healthy upstream» errors.

Checking the VirtualService for the destination, I noticed that 5% of the traffic is routed to v2, which agrees with what I saw also in Kiali, while 95% is routed to v1, which also explains why the customer saw 95% failures with the «no healthy upstream» message.

All the customer needed to do to fix the problem was to deploy service v1 or update the VirtualService to distribute all requests to v2.

Duplicate Envoy clusters

In a follow-up escalation, the «no healthy upstream» issue came up again. We followed the same troubleshooting approach as in the previous example, but in this case there was no VirtualService.

We saw multiple ServiceEntry definitions for multiple country destinations. We found it puzzling that all country destinations, apart from the one reported, had requests directed correctly. The following check verified that all clusters were reported as healthy except service-destination.remote-namespace.ocp4.customdomain.com:

$ oc exec istio-egressgateway-6567f7d756-4gvh8 -- curl localhost:15000/clusters |egrep 'health|remote-service-destination.remote-namespace.ocp4.customdomain.com'

In this case I looked for a log entry like:

outbound|443||remote-service-destination.remote namespace.ocp4.customdomain.com::10.128.2.21:443::health_flags::/failed_active_hc

Having verified the cluster as unhealthy, I turned to Istiod to ensure that no errors were being reported against this cluster. The Istiod pod reported:

2022-05-13T12:27:51.262316Z info ads Push finished: 5.709679819s {   "ProxyStatus": {
"pilot_duplicate_envoy_clusters": {
"outbound|443||remote-service-destination.remote-namespace.ocp4.customdomain.com": {
"proxy": "e2e-871-remote-namespace-c58f7f7f6-vljr6.e2e-871-remote namespace",
"message": "Duplicate cluster outbound|53||remote-service destination.remote-namespace.ocp4.customdomain.com found while pushing CDS"   }
},

This output indicates that Istiod was trying to apply a cluster configuration for which there was a duplicate. This information prompted me to check the applied ServiceEntry resources, which quickly revealed that there had been duplicate definitions for remote-service-destination.remote-namespace.ocp4.customdomain.com.

Istio administrative tools reveal the source of errors

A «no healthy upstream» error can be caused by Istio, a misconfigured network device, or actual network outages. Thus, it is difficult for the mesh operations team to pinpoint the cause or even predict its occurrence, because DevOps teams may unwittingly apply an incorrect configuration. This article provided guidance on how to establish the cause of such errors, determining whether they are or are not due to Istio configuration.

Even greater benefits can be realized when, as in the case of the customer in this example, the operations team applies observability monitoring and alerts against such occurrences, so that the team can be aware in advance of the issue and inform the relevant teams before an escalation occurs.

Last updated:
January 4, 2023