Google container builder task cannot find gcloud

[varungbt]

I am using the vsts google container builder task inside my build pipeline.
I am running this task on a Hosted Linux preview vsts agent.

The container builder task fails to find gcloud on the agent machine.

 ##[error]Failed which: Not found gcloud: null

I also tried installing gcloud on the agent using the installer and adding gcloud to the PATH

curl https://sdk.cloud.google.com | bash
source /root/google-cloud-sdk/path.bash.inc

Container builder tasks fails to find gcloud.

[ILMTitan]

VSTS does not maintain the path environment variable between task executions. You have to tell VSTS to carry the path forward. The Google Cloud SDK-Tool Installer Task does this for you.

If you want to use your own script, you have to output to the console a command to prepend the path:
From vsts-task-tool-lib/tool.ts

console.log('##vso[task.prependpath]' + toolPath);

[varungbt]

@ILMTitan Thanks for your reply. I used the Google SDK Tool installer task as the first step and later used the container builder task as the second step. In this case too gcloud could not be found.

[ILMTitan]

These were two steps in the same build on the same build agent? Could you show me the build logs for, at a minimum, the Tool Installer task?

[varungbt]

@ILMTitan you are right the usage of the goolge sdk tool installer task prepends the path to the vso[task.prependpath] variable

Logs from Google cloud sdk installer task

 Prepending PATH environment variable with directory: /opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin
 ##[debug]new Path: /opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go1.10/bin:/opt/hostedtoolcache/Ruby/2.5.1/x64/bin
 ##[debug]Processed: ##vso[task.prependpath]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin
20

When I try to use the installed gcloud in step2 container builder task

Logs from Google container builder task

2018-07-10T19:19:57.2940606Z c847ff37-d156-4f4d-861d-52dbb0872b11 exists true
2018-07-10T19:19:57.3038872Z ##[debug]c847ff37-d156-4f4d-861d-52dbb0872b11 exists true
2018-07-10T19:19:57.3132435Z ##[debug]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud arg: container builds submit --quiet --format=json
2018-07-10T19:19:57.3228475Z ##[debug]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud arg: ["--project=\"jord-dev\"","--credential-file-override=\"/opt/vsts/work/1/s/tempKeyFile.json\""]
2018-07-10T19:19:57.3321169Z ##[debug]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud arg: "/opt/vsts/work/1/s/vsts-agent"
2018-07-10T19:19:57.3416560Z ##[debug]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud arg: --config="/opt/vsts/work/1/s/vsts-agent/cloudbuild.yml"
2018-07-10T19:19:57.3521250Z ##[debug]exec tool: /opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud
2018-07-10T19:19:57.3624188Z ##[debug]Arguments:
2018-07-10T19:19:57.3730397Z ##[debug]   container
2018-07-10T19:19:57.3832405Z ##[debug]   builds
2018-07-10T19:19:57.3933346Z ##[debug]   submit
2018-07-10T19:19:57.4035188Z ##[debug]   --quiet
2018-07-10T19:19:57.4138290Z ##[debug]   --format=json
2018-07-10T19:19:57.4242412Z ##[debug]   --project="someproject"
2018-07-10T19:19:57.4345825Z ##[debug]   --credential-file-override="/opt/vsts/work/1/s/tempKeyFile.json"
2018-07-10T19:19:57.4448792Z ##[debug]   "/opt/vsts/work/1/s/vsts-agent"
2018-07-10T19:19:57.4551123Z ##[debug]   --config="/opt/vsts/work/1/s/vsts-agent/cloudbuild.yml"
2018-07-10T19:19:57.4624610Z [command]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud container builds submit --quiet --format=json --project="someproject" --credential-file-override="/opt/vsts/work/1/s/tempKeyFile.json" "/opt/vsts/work/1/s/vsts-agent" --config="/opt/vsts/work/1/s/vsts-agent/cloudbuild.yml"
2018-07-10T19:19:57.4725658Z ##[debug]rc:1
2018-07-10T19:19:57.4826156Z ##[debug]success:false
2018-07-10T19:19:57.4930767Z ##[debug]task result: Failed
2018-07-10T19:19:57.5129836Z ##[error]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud failed with return code: 1
2018-07-10T19:19:57.5244161Z ##[debug]Processed: ##vso[task.issue type=error;]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud failed with return code: 1
2018-07-10T19:19:57.5403323Z ##[debug]Processed: ##vso[task.complete result=Failed;]/opt/hostedtoolcache/google-cloud-sdk/208.0.0/x64/google-cloud-sdk/bin/gcloud failed with return code: 1
2018-07-10T19:19:57.5551221Z ##[section]Finishing: Container build for vsts-agent

I do not clearly understand why the gcloud command failed.

I do not face this issue when I run this pipeline of two steps on a Hosted VS2017 agent.

I notice this issue of gcloud command failing when I use linux preview agent.

May be I am missing out on something.

[varungbt]

@ILMTitan any pointers on what could be wrong in my settings ? thanks

[ILMTitan]

That log shows the path to gcloud was found, and that a reasonable command was executed. It looks like any error output from gcloud is being swallowed. Are any builds showing up in your Cloud Build Console?

[varungbt]

@ILMTitan gcloud is found in the PATH when I use the google sdk installer however the container builder task fails with no information on the error. There are no builds reaching the Google Cloud builder service. I have checked the logs on GCP and I do not see any traces of the build request. I suppose the build is failing inside vsts even before reaching GCP.

As I mentioned earlier the same set of vsts tasks work perfectly fine on Hosted VS2017 agent but they fail on Hosted Linux preview agent.

[varungbt]

@ILMTitan I am trying to debug locally by running a vsts agent and registering the agent inside a queue.
I still cannot figure why the gcloud command fails.

I tried to manually exec into the running docker vsts agent
and run

/var/vsts/_tool/google-cloud-sdk/208.0.2/x64/google-cloud-sdk/bin/gcloud container builds submit --quiet --format=json --project="myproject" --credential-file-override="/var/vsts/1/s/myserviceaccountkey.json" "/var/vsts/1/s/vsts-agent" --config="/var/vsts/1/s/vsts-agent/cloudbuild.yml"

When I run this command manually inside a running container, the build is getting submitted to google cloud container builder service.

The same command when it runs from a vsts build pipeline it fails with
##[error]/var/vsts/_tool/google-cloud-sdk/208.0.2/x64/google-cloud-sdk/bin/gcloud failed with return code: 1

I also tried to update the vsts service account key with the same key as myserviceaccountkey.json in the command above.

Is there a way to debug and see why it fails only from a vsts build pipeline ?

[varungbt]

@ILMTitan any updates ? thanks

[ILMTitan]

@VarunkumarManohar What happens if you run the same command using the Gcloud Command Task?

container builds submit --quiet --format=json "/var/vsts/1/s/vsts-agent" --config="/var/vsts/1/s/vsts-agent/cloudbuild.yml"

[varungbt]

@ILMTitan When I try to use the gcloud command task I on hosted linux agent I see:
ERROR: (gcloud.container.builds.submit) Failed to load credential file: ["/home/vsts/work/1/s/tempKeyFile.json"]. The parameter passed to the from_stream() method should point to a file.

This however works fine on windows agent

[varungbt]

Allow scripts to access the OAuth token was set to true