Selaa lähdekoodia

follow_with_liveness: don't mislabel a slow boot as 'engine down'

When the log-follow stream ended, the show_logs branch decided death from the
one-shot API probe (readiness::alive). The API port stays closed for the whole
multi-minute weight load, so a false answer there is indistinguishable from
'still starting' — this fired twice today (both GLM serves) reporting 'log
stream ended and the engine is down — containers died mid-follow' while the
engine was alive and loading.

A real death is a container that has EXITED. Now mirrors wait_engine: on log-
stream end, run containers_alive (an exited container → fail + close the run;
a still-Up container → warn 'ssh drop or still starting', return Ok). The
separate watch() branch already handled startup correctly (misses counted only
after first readiness).
Alexander Korolev 6 päivää sitten
vanhempi
sitoutus
bde9a8c427
1 muutettua tiedostoa jossa 14 lisäystä ja 3 poistoa
  1. 14 3
      src/docker/logs.rs

+ 14 - 3
src/docker/logs.rs

@@ -107,11 +107,22 @@ pub async fn follow_with_liveness(
     tokio::select! {
         r = show_logs(config, cluster, recipe_name, true, node_name, dry_run) => {
             r?;
-            if !crate::readiness::alive(cluster, port).await {
+            // The log stream ended. Do NOT decide death from the one-shot API
+            // probe (`alive`): the API port is closed for the whole (multi-
+            // minute) weight load, so a false answer here is indistinguishable
+            // from "still starting" — that mislabelled a slow boot as "engine
+            // is down — containers died". A real death is a container that has
+            // EXITED; a still-Up container means the engine is loading or the
+            // ssh follow simply dropped. Mirror `wait_engine`'s check.
+            if let Err(e) =
+                crate::readiness::containers_alive(config, cluster, recipe_name).await
+            {
                 let _ = crate::runlock::mark_stop(recipe_name, &config.recipe_sources);
-                anyhow::bail!("log stream ended and the engine is down — containers died mid-follow");
+                anyhow::bail!("log stream ended and a container exited — {e}");
             }
-            tracing::warn!("log stream ended but the engine is still up (ssh drop?)");
+            tracing::warn!(
+                "log stream ended but the containers are still up (ssh drop, or the engine is still starting)"
+            );
             Ok(())
         }
         r = watch => {