We spent a day building AutoPilot Agentic — a Google ADK agent crew running on Gemini 3.5 over Vertex AI, deployed to Cloud Run — as an entry for Google's All Things Agentic hackathon. Most of it went the way you would hope. Two failures ate the bulk of the time, and they turned out to share a shape worth writing down:
Both were 404s that read as permission problems. Neither had anything to do with permissions.
That shape is expensive because of what it does to your next hour. A 404 saying "not found or your project does not have access to it" sends you to audit IAM, check entitlements, re-read quota pages, and open a support tab. None of that is where the answer is, and the whole time the thing you are looking for is working perfectly somewhere you did not look.
1. Gemini 3.x answers on the global endpoint, not a regional one
The first call we made to gemini-3.5-flash through Vertex AI came back like this:
404 Publisher model projects/<project>/locations/us-central1/publishers/google/models/gemini-3.5-flash
was not found or your project does not have access to it.Read that sentence as a developer who has just been handed a new model family. "Your project does not have access to it" is doing a lot of work there. It is a plausible, complete explanation: the model is new, maybe it is allowlisted, maybe our billing account is not enrolled, maybe there is a preview form we did not fill in.
Meanwhile gcloud ai model-garden models list showed the model sitting right there in the catalog for that project — which, if anything, made it more confusing. The catalog says we have it; the endpoint says we do not.
The actual answer is that the Gemini 3.x family is served from the global endpoint. The same call, same project, same credentials, with location="global" instead of us-central1, answered on the first attempt:
client = genai.Client(vertexai=True, project=PROJECT, location="global")
client.models.generate_content(model="gemini-3.5-flash", contents="…") # 200Two things generalise from this, and the second matters more than the first.
Catalog presence is not endpoint reachability. A model appearing in Model Garden for your project tells you it exists in the catalog. It does not tell you which endpoints serve it. Only a call to a specific endpoint answers that question, and the two facts are independent enough that trusting the first will cost you the afternoon.
So do not hardcode the model — probe for it. We stopped treating the model name as a constant and made it a resolved value: a ladder of candidates, probed once at startup, first one that actually answers wins.
def resolve_model(ladder, location, probe):
tried = []
for candidate in ladder:
tried.append(candidate)
try:
if probe(candidate):
return Resolution(model=candidate, location=location,
probed=tuple(tried),
fallback_used=candidate != ladder[0])
except Exception:
continue # a 404 here is information, not a crash
raise RuntimeError(
f"no model in {list(ladder)} answered at location {location!r}; "
"the Gemini 3.x family is served from the global endpoint — check "
"the location before assuming an entitlement problem")Note the error message. When the ladder is exhausted, the exception names the endpoint as the prime suspect, because that is the thing the next person will not think of. An error message is a place to leave a note for whoever hits this at 2am — and that person is usually you.
The resolved model, the endpoint it answered on, and whether a fallback rung was used are all reported on the service's health endpoint. That distinction — the model this instance resolved versus the model it was configured with — turns out to matter, because they are not always the same and only one of them is a fact about the running process.
2. /healthz never reaches a Cloud Run container
The second one is stranger, and we only believed it after building a control.
Our service exposed GET /healthz. The deploy pipeline polled it after each release and refused to go green until it answered. It never answered. Every request returned Google's own branded 404 page — not our application's JSON 404, the HTML one with the Google logo.
The obvious readings are all wrong. The service was healthy. In the same minute, on the same revision:
GET / → 200 (the console HTML)
GET /api/runs → 200 (real JSON, Firestore-backed)
GET /openapi.json → 200 — and its paths block LISTS /healthz
GET /healthz → 404 Google's HTML error pageSo the route existed, the framework knew about it, the app was serving everything else, and the health check said dead.
The measurement that settles it is in the request log. On a *.run.app hostname, a request to /healthz appears in no Cloud Run request log at all. It never reaches the container.
That alone is not evidence — an absence in a log can just as easily mean logging is off, and an unmeasured zero is not a zero. So before believing it we needed a known-positive: something that should produce a 404 from the app, to prove the instrument could see one. GET /zzz on the same service, in the same minute, did log a 404 from the application. Logging was on. The instrument worked. The absence of /healthz was therefore a real absence.
(A related detail that made this harder to spot from the outside: the Google frontend substitutes its own error page for a 404, so /zzz and /healthz looked identical from curl while being completely different underneath — one served by our app, one never delivered to it.)
The fix is a rename. Our health endpoint is now /api/health, under a path prefix we own. /healthz is kept as an alias, which still works when you probe the container directly — locally, or in Docker — and that is exactly the trap: it works everywhere you test it and fails only on the hostname your users and your deploy gate actually use.
The practical rule: do not name a Cloud Run health endpoint /healthz. And if a health check 404s while the service seems fine, curl the root and one other real route before you go looking for the outage. Two 200s and a 404 on the health path is this bug, not a broken service.
The habit both of these argue for
Record the status code, not the conclusion.
A note that says "404 on the regional endpoint, 2026-08-30" stays true forever and can be re-tested by anyone. A note that says "our project doesn't have access to Gemini 3" was never true, is now permanently misleading, and — this is the expensive part — nobody ever re-checks a wall. A conclusion written into a README or a ticket becomes a fact that the team routes around for months. The measurement it came from would have been falsified in thirty seconds.
We have watched this exact failure play out before in our own docs: a capability recorded as impossible, generalised from a single denial, that turned out to be one configuration line away the whole time. The status code is cheap to write down and it does not rot.
What we were building
AutoPilot Agentic takes the next ticket off a delivery board, reads the repository, writes the change on its own branch, and opens one pull request per ticket — then stops at a gate it cannot open. Four ADK agents (triage → plan → implement → review) do the thinking; the guard rails, the approval gate and every write live in ordinary Python the agents cannot reach. The model proposes; the orchestrator disposes.
You can watch a run — including the guard refusing a change the reviewing agent had already approved — in the three-minute demo, or read the writeup on Devpost.
If you are building agents that touch real repositories and want a second pair of eyes on the guard rails, talk to us — it is most of what we do.