Running with gitlab-runner 17.11.0 (0f67ff19)  on gitlab-runner-66336c5289e5e542-docker ncJ2-Cr2P, system ID: s_43282bf41924 section_start:1767787507:prepare_executor Preparing the "docker" executor Using Docker executor with image registry.gitlab.com/opencraft/dev/grove/tools-container:0.36.1 ... Starting service docker:28-dind... Pulling docker image docker:28-dind ... Using docker image sha256:6a68f64cf32d98b09a11c208de78f59f17c0a6fff33c13f11acac853d6aad5ae for docker:28-dind with digest docker@sha256:2a232a42256f70d78e3cc5d2b5d6b3276710a0de0596c145f627ecfae90282ac ... Waiting for services to be up and running (timeout 30 seconds)... Authenticating with credentials from job payload (GitLab Registry) Pulling docker image registry.gitlab.com/opencraft/dev/grove/tools-container:0.36.1 ... Using docker image sha256:646fe95ffa88a1e9c14a40f6a852d40e89ce873360b1ce823aa42db6f9a447ba for registry.gitlab.com/opencraft/dev/grove/tools-container:0.36.1 with digest registry.gitlab.com/opencraft/dev/grove/tools-container@sha256:4efd5c7c74e2eab7e433d412c8dfbcb8c750f263450a2ec0a2e2257379be0b0f ... section_end:1767787527:prepare_executor section_start:1767787527:prepare_script Preparing environment Running on runner-ncj2-cr2p-project-51564656-concurrent-1 via gitlab-runner-66336c5289e5e542... section_end:1767787528:prepare_script section_start:1767787528:get_sources Getting source from Git repository Fetching changes with git depth set to 50... Reinitialized existing Git repository in /builds/opencraft/ops/axim-pr-sandboxes/.git/ Created fresh repository. Checking out b63fc39f as detached HEAD (ref is main)... Removing .terraform/ Removing instances/pr-37795-139931/.venv/ Removing instances/pr-37795-139931/env/ Removing instances/pr-37795-139931/grove-preserved-config.yml.previous Updating/initializing submodules recursively with git depth set to 50... Submodule 'grove' (https://gitlab.com/opencraft/dev/grove.git) registered for path 'grove' Synchronizing submodule url for 'grove' Entering 'grove' Removing provider-digitalocean/.terraform.lock.hcl Removing tools-container/grove-cli/grove/__pycache__/ Entering 'grove' HEAD is now at 4c92eaa temp: remove caching Updated submodules Synchronizing submodule url for 'grove' Entering 'grove' Entering 'grove' section_end:1767787531:get_sources section_start:1767787531:restore_cache Restoring cache Checking cache for tf-b63fc39f-protected... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. WARNING: Cache file does not exist  Failed to extract cache section_end:1767787531:restore_cache section_start:1767787531:step_script Executing "step_script" stage of the job script Using docker image sha256:646fe95ffa88a1e9c14a40f6a852d40e89ce873360b1ce823aa42db6f9a447ba for registry.gitlab.com/opencraft/dev/grove/tools-container:0.36.1 with digest registry.gitlab.com/opencraft/dev/grove/tools-container@sha256:4efd5c7c74e2eab7e433d412c8dfbcb8c750f263450a2ec0a2e2257379be0b0f ... $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE opencraft/ops/axim-pr-sandboxes:axim-pr-sandboxes gitlab agent:1077905 $ kubectl config use-context ${CI_PROJECT_PATH}:${K8S_CLUSTER_AGENT_CONTEXT} Switched to context "opencraft/ops/axim-pr-sandboxes:axim-pr-sandboxes". $ $SCRIPTS_DIR/tf.sh init Initializing the backend...  Successfully configured the backend "http"! OpenTofu will automatically use this backend unless the backend configuration changes. Initializing modules... - edx_instances in ../provider-modules/instance - harmony in ../provider-modules/harmony - instance_edxapp_bucket in spaces - k8s_cluster in k8s-cluster - k8s_gitlab_connector in ../provider-modules/k8s-gitlab-connector - k8s_gitlab_container_registry in ../provider-modules/k8s-gitlab-container-registry - k8s_monitoring in ../provider-modules/k8s-monitoring - mongodb in database - mysql in database - plugin in ../provider-plugin - tutor_env in spaces - velero_backups in spaces Initializing provider plugins... - Finding latest version of hashicorp/external... - Finding latest version of hashicorp/random... - Finding latest version of loafoe/htpasswd... - Finding latest version of hashicorp/null... - Finding gavinbunney/kubectl versions matching "~> 1.18"... - Finding latest version of hashicorp/tls... - Finding digitalocean/digitalocean versions matching "~> 2.0"... - Finding hashicorp/helm versions matching "~> 2.17"... - Finding latest version of hashicorp/kubernetes... - Finding gitlabhq/gitlab versions matching "~> 17.6"... - Installing hashicorp/external v2.3.5... - Installed hashicorp/external v2.3.5 (signed, key ID 0C0AF313E5FD9F80) - Installing hashicorp/null v3.2.4... - Installed hashicorp/null v3.2.4 (signed, key ID 0C0AF313E5FD9F80) - Installing hashicorp/tls v4.1.0... - Installed hashicorp/tls v4.1.0 (signed, key ID 0C0AF313E5FD9F80) - Installing digitalocean/digitalocean v2.72.0... - Installed digitalocean/digitalocean v2.72.0 (signed, key ID F82037E524B9C0E8) - Installing hashicorp/helm v2.17.0... - Installed hashicorp/helm v2.17.0 (signed, key ID 0C0AF313E5FD9F80) - Installing hashicorp/kubernetes v3.0.1... - Installed hashicorp/kubernetes v3.0.1 (signed, key ID 0C0AF313E5FD9F80) - Installing hashicorp/random v3.7.2... - Installed hashicorp/random v3.7.2 (signed, key ID 0C0AF313E5FD9F80) - Installing loafoe/htpasswd v1.5.0... - Installed loafoe/htpasswd v1.5.0 (signed, key ID C0E4EB79E9E6A23D) - Installing gavinbunney/kubectl v1.19.0... - Installed gavinbunney/kubectl v1.19.0 (signed, key ID 1E1CE42504F5FBB2) - Installing gitlabhq/gitlab v17.11.0... - Installed gitlabhq/gitlab v17.11.0 (signed, key ID 0D47B7AB85F63F65) Providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://opentofu.org/docs/cli/plugins/signing/ OpenTofu has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that OpenTofu can guarantee to make the same selections by default when you run "tofu init" in the future. OpenTofu has been successfully initialized! $ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your credentials are stored unencrypted in '/home/grove/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded $ printf "debug = true\n[worker.oci]\n max-parallelism = $DOCKER_BUILDKIT_PARALLELISM\n[registry.\"docker.io\"]\n mirrors = [\"${CUSTOM_DOCKER_REGISTRY:-https://registry-1.docker.io}\"]" > ~/buildkitd.toml $ docker buildx create --use --config ~/buildkitd.toml great_tharp $ docker buildx install $ grove tutor buildimage $INSTANCE_NAME $IMAGE_NAME --edx-platform-repository="$EDX_PLATFORM_REPOSITORY" --edx-platform-version="$EDX_PLATFORM_VERSION" Loading outputs from Terraform bash /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/scripts/tf.sh output -json bash /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/scripts/sync_tutor_env.sh pull ╷ │ Error: Failed to load state: Failed to read remote state: stream error: stream ID 1; INTERNAL_ERROR; received from peer │ │ ╵ fatal error: Parameter validation failed: Invalid bucket name "": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$" ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/grove-cli/bin/ │ │ cli.py:253 in buildimage │ │ │ │ 250 │ │ typer.echo("Docker is not running / failed to reach it within │ │ 251 │ │ raise typer.Abort() │ │ 252 │ instance = Instance(instance_name) │ │ ❱ 253 │ instance.tutor_env_pull() │ │ 254 │ instance.clone_private_plugins() │ │ 255 │ │ │ 256 │ tutor_cache_from_args = [] │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ cache_from = [] │ │ │ │ cache_to = [] │ │ │ │ edx_platform_repository = 'https://github.com/openedx/edx-platform.git' │ │ │ │ edx_platform_version = 'remove-problem-studioview' │ │ │ │ image_name = 'openedx' │ │ │ │ instance = │ │ │ │ instance_name = 'pr-37795-139931' │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/grove-cli/grov │ │ e/instance.py:335 in tutor_env_pull │ │ │ │ 332 │ │ """ │ │ 333 │ │ use_terraform = terraform.is_infrastructure_ready(self.instanc │ │ 334 │ │ if use_terraform: │ │ ❱ 335 │ │ │ tutor_env_pull(self.tutor_env()) │ │ 336 │ │ else: │ │ 337 │ │ │ typer.echo(f"Unable to pull tutor env from s3 for {self.in │ │ 338 │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ self = │ │ │ │ use_terraform = { │ │ │ │ │ 'database': 'pr-37795-139931', │ │ │ │ │ 'forum_database': │ │ │ │ 'pr-37795-139931-cs_comments_service', │ │ │ │ │ 'password': 'K806b5TF9S2eaA47', │ │ │ │ │ 'username': 'pr-37795-139931' │ │ │ │ } │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/grove-cli/grov │ │ e/utils.py:139 in tutor_env_pull │ │ │ │ 136 │ │ 137 │ │ 138 def tutor_env_pull(env: dict): │ │ ❱ 139 │ execute(f"{SYNC_SCRIPT} pull", env) │ │ 140 │ │ 141 │ │ 142 def tutor_env_push(env: dict): │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ env = { │ │ │ │ │ 'TUTOR_ID': 'pr-37795-139931', │ │ │ │ │ 'TUTOR_ROOT': │ │ │ │ '/builds/opencraft/ops/axim-pr-sandboxes/instances/pr-37795-13993… │ │ │ │ │ 'TUTOR_K8S_NAMESPACE': 'pr-37795-139931', │ │ │ │ │ 'TUTOR_MFE_DOCKER_IMAGE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/pr-37795-139… │ │ │ │ │ 'TUTOR_MFE_DOCKER_IMAGE_CACHE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/pr-37795-139… │ │ │ │ │ 'TUTOR_DOCKER_IMAGE_OPENEDX': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/pr-37795-139… │ │ │ │ │ 'TUTOR_DOCKER_IMAGE_OPENEDX_CACHE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/pr-37795-139… │ │ │ │ │ 'TUTOR_MYSQL_HOST': │ │ │ │ 'axim-sandboxes-mysql-do-user-14983397-0.c.db.ondigitalocean.com', │ │ │ │ │ 'TUTOR_MYSQL_PORT': '25060', │ │ │ │ │ 'TUTOR_MYSQL_ROOT_USERNAME': 'doadmin', │ │ │ │ │ ... +19 │ │ │ │ } │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/grove-cli/grov │ │ e/utils.py:130 in execute │ │ │ │ 127 │ │ │ process.wait() │ │ 128 │ │ │ raise CommandFailedError(f"Command failed: {command}", out │ │ 129 │ │ if result > 0: │ │ ❱ 130 │ │ │ raise CommandFailedError(f"Command failed with status {res │ │ 131 │ │ │ 132 │ if capture_output: │ │ 133 │ │ return "\n".join(output) │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ capture_output = False │ │ │ │ command = 'bash │ │ │ │ /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-co… │ │ │ │ env = { │ │ │ │ │ 'TUTOR_ID': 'pr-37795-139931', │ │ │ │ │ 'TUTOR_ROOT': │ │ │ │ '/builds/opencraft/ops/axim-pr-sandboxes/instances/pr-… │ │ │ │ │ 'TUTOR_K8S_NAMESPACE': 'pr-37795-139931', │ │ │ │ │ 'TUTOR_MFE_DOCKER_IMAGE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/p… │ │ │ │ │ 'TUTOR_MFE_DOCKER_IMAGE_CACHE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/p… │ │ │ │ │ 'TUTOR_DOCKER_IMAGE_OPENEDX': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/p… │ │ │ │ │ 'TUTOR_DOCKER_IMAGE_OPENEDX_CACHE': │ │ │ │ 'registry.gitlab.com/opencraft/ops/axim-pr-sandboxes/p… │ │ │ │ │ 'TUTOR_MYSQL_HOST': │ │ │ │ 'axim-sandboxes-mysql-do-user-14983397-0.c.db.ondigita… │ │ │ │ │ 'TUTOR_MYSQL_PORT': '25060', │ │ │ │ │ 'TUTOR_MYSQL_ROOT_USERNAME': 'doadmin', │ │ │ │ │ ... +19 │ │ │ │ } │ │ │ │ environ = { │ │ │ │ │ 'PYTHONPATH': │ │ │ │ '/builds/opencraft/ops/axim-pr-sandboxes/grove/tools-c… │ │ │ │ │ 'CI_DEFAULT_BRANCH_SLUG': 'main', │ │ │ │ │ 'TF_VAR_velero_enabled': 'false', │ │ │ │ │ 'CI_PROJECT_NAMESPACE': 'opencraft/ops', │ │ │ │ │ 'DOCKER_ENV_CI_JOB_STATUS': 'running', │ │ │ │ │ 'GITLAB_USER_ID': '14467504', │ │ │ │ │ 'DOCKER_VERSION': '28.5.1', │ │ │ │ │ 'DOCKER_ENV_CI_PROJECT_DESCRIPTION': 'This cluster │ │ │ │ repository is for hosting PR sandboxes for │ │ │ │ Axim.\r\n\r\nDevelopment epi'+109, │ │ │ │ │ 'CI_RUNNER_VERSION': '17.11.0', │ │ │ │ │ 'DOCKER_ENV_CI_BUILDS_DIR': '/builds', │ │ │ │ │ ... +419 │ │ │ │ } │ │ │ │ hide_output = False │ │ │ │ key = 'TUTOR_S3_REGION' │ │ │ │ line = 'Invalid bucket name "": Bucket name must match the │ │ │ │ regex "^[a-zA-Z0-9.\\-_]{1,255'+249 │ │ │ │ output = [] │ │ │ │ process = │ │ │ │ result = 1 │ │ │ │ val = 'nyc3' │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────╯ CommandFailedError: Command failed with status 1: bash /builds/opencraft/ops/axim-pr-sandboxes/grove/tools-container/scripts/sync_tutor _env.sh pull section_end:1767787551:step_script section_start:1767787551:cleanup_file_variables Cleaning up project directory and file based variables section_end:1767787551:cleanup_file_variables ERROR: Job failed: exit code 1