Troubleshooting kongctl

This reference covers common issues and their solutions when using kongctl.

“No changes detected” when changes exist

Symptom: Modified configuration but plan shows no changes, or resources appear unchanged after apply.

Causes:

1. Resource already matches desired state
2. Invalid resource references
3. Namespace mismatch

Solutions:

Dump current state and compare with your configuration:

kongctl dump declarative --resources <your-resource-type> --output-file current-state.yaml
# or
kongctl dump tf-import --resources <your-resource-type>
diff current-state.yaml your-config.yaml

Check resource references are spelled correctly and match a declared ref:

grep "ref:" your-config.yaml

Verify the resource has the expected namespace label in Konnect:

kongctl get apis -o json

“Resource not found” errors

Symptom: Error during plan or apply referencing a non-existent resource.

Example:

Error: resource "my-portal" not found

Solutions:

Check whether the resource exists:

kongctl get portals | grep my-portal

Verify the ref spelling in your configuration files:

grep -n "my-portal" *.yaml

Ensure dependencies are applied first when resources reference each other:

kongctl apply -f portals.yaml
kongctl apply -f apis.yaml
kongctl apply -f publications.yaml

“Authentication required” or “unauthorized” errors

Symptom: Commands fail with authentication errors.

Solutions:

1. Verify you’re logged in:

   kongctl get me

   Copied!

2. Re-authenticate:

   kongctl logout
   kongctl login

   Copied!

3. If using a PAT, verify it’s set:

   echo $KONGCTL_DEFAULT_KONNECT_PAT | head -c 20

   Copied!

4. Check that the token hasn’t expired in Konnect.

5. Verify that region is configured as expected.

Browser login timeout

Symptom: Browser authorization doesn’t complete in time.

Solutions:

1. Complete authorization within the time limit (usually 15 minutes).

2. If timeout occurs, start over:

    kongctl login

   Copied!

3. Verify that your browser isn’t blocking the redirect.

Multiple authentication methods conflict

Symptom: Unexpected authentication behavior or wrong credentials being used.

Solution: Check and clear auth methods, if needed.

kongctl resolves authentication in this priority order (highest to lowest):

1. --pat flag
2. KONGCTL_<PROFILE>_KONNECT_PAT environment variable
3. Stored token from kongctl login

To start fresh and clear all auth methods:

unset KONGCTL_DEFAULT_KONNECT_PAT
kongctl logout
kongctl login

If you manage profiles outside the default path, you may also need to remove ~/.config/kongctl/.<profile>-konnect-token.json manually.

YAML parsing errors

Symptom:

Error: yaml: unmarshal errors:
  line 10: cannot unmarshal !!str `true` into bool

Solution: Ensure field values use the correct YAML type. For example, for a boolean value:

authentication_enabled: true

Duplicate resource references

Symptom:

Error: duplicate resource ref "my-api" found

Solution: Each ref must be unique across all loaded configuration files. Find duplicates and rename them:

grep -n "ref: my-api" *.yaml

Invalid field values

Symptom:

Error: invalid value for field "visibility": "internal"

Solution: Check the allowed values in the supported resources reference:

api_publications:
  - ref: my-pub
    visibility: private  # Allowed values: public, private

File not found errors

Symptom:

Error: failed to process file tag: file not found: ./specs/api.yaml

Solutions: Look for the following common errors:

1. Incorrect relative path: paths must be relative to the config file, not the working directory:

   spec: !file ./specs/api.yaml

   Copied!

2. Wrong base directory: If your spec file is outside the config file’s directory:

   project/
   ├── config/
   │   └── main.yaml
   └── specs/
       └── api.yaml

   In config/main.yaml, you would access the file like this:

   spec: !file ../specs/api.yaml

   Copied!

   If you see path resolves outside base dir, use the --base-dir flag or the KONGCTL_DEFAULT_KONNECT_DECLARATIVE_BASE_DIR environment variable to set an allowed boundary.

3. File permissions: Verify the file is readable:

   ls -la ./specs/api.yaml
   chmod 644 ./specs/api.yaml

   Copied!

Invalid YAML tag extraction path

Symptom:

Error: path not found: info.nonexistent.field

Solution:

1. Review the YAML path closely, looking for typos or incorrect paths.

2. Use dot notation for arrays instead of bracket array syntax:

   server: !file ./spec.yaml#servers.0.url

   Copied!

Malformed YAML tag syntax

Symptom:

Error: failed to parse file reference: invalid tag format

Solution: Check the correct map format for !file:

title: !file
  path: ./spec.yaml
  extract: info.title

Large file errors

Symptom:

Error: file size exceeds limit: ./large-spec.yaml (12MB > 10MB limit)

Solutions:

Split large files, or extract only the values you need rather than loading the entire file. For example, with a large file named huge-openapi-spec.yaml, access only the data you need:

name: !file ./huge-openapi-spec.yaml#info.title
version: !file ./huge-openapi-spec.yaml#info.version

Unknown resource references

Symptom:

Error: resource "my-api" references unknown portal: unknown-portal

Solutions: Look for the following common errors:

1. Typo in the ref value: The ref must exactly match the ref field of the target resource:

   portals:
     - ref: developer-portal  # Exact ref value
   
   api_publications:
     - ref: api-pub
       portal_id: !ref developer-portal # Matches the portal ref

   Copied!

2. Nested vs separate resource conflict: Don’t declare the same ref both nested and at the root:

   # Wrong - v1 is declared twice
   apis:
     - ref: my-api
       versions:
         - ref: v1  # Nested
   
   api_versions:
     - ref: v1      # Conflict
       api: my-api

   Copied!

External ID vs reference confusion

Symptom:

Error: resource references unknown control_plane_id: my-control-plane

Solution: Fields like control_plane_id expect a Konnect UUID, not a declarative ref. Use !ref to resolve a declarative resource’s ID:

api_implementations:
  - ref: impl
    service:
      control_plane_id: !ref my-control-plane#id

Plan generation hangs

Symptom: kongctl plan doesn’t complete.

Solutions:

Enable debug logging to see where it’s stuck:

kongctl plan -f config.yaml --log-level debug

Test network connectivity to Konnect:

kongctl get portals

Try planning a smaller configuration to isolate the issue:

kongctl plan -f single-resource.yaml

Circular dependencies

Symptom:

Error: circular dependency detected: api1 -> api2 -> api1

Solution: Restructure your configuration to break the cycle by introducing a shared base resource or removing the circular reference.

Invalid plan file

Symptom:

Error: failed to read plan: invalid plan format

Solutions:

Validate the plan file is valid JSON:

cat plan.json | jq . > /dev/null

Regenerate the plan if it’s corrupted:

kongctl plan -f config.yaml --output-file plan.json

Stale plan artifact

Symptom: You see the following error:

Error: plan is out of date - resource already exists

Solution: Regenerate the plan to reflect the current live state:

kongctl plan -f config.yaml --output-file new-plan.json

Inspecting plan contents

View the plan summary:

jq '.summary' plan.json

List all planned operations:

jq '.changes[] | {action: .action, type: .resource_type, ref: .resource_ref}' plan.json

Filter to specific operations:

# Show only CREATE operations
jq '.changes[] | select(.action == "CREATE")' plan.json

Partial apply failures

Symptom: Some resources are created successfully, others fail. Apply reports errors for specific resources.

Solution: Fix the failing resource configuration and re-run apply. The apply command is idempotent, so existing resources are skipped:

kongctl apply -f config.yaml

Protected resource blocking changes

Symptom: You see the following error:

Error: Cannot modify protected resource "production-api"

Solution: Temporarily set protected: false, apply your changes, then re-enable protection:

apis:
  - ref: production-api
    kongctl:
      protected: false  # Changed from true

Sync deleting unexpected resources

Symptom: Resources are deleted that shouldn’t be.

Prevention: Always dry-run before syncing, especially in production:

kongctl sync -f config.yaml --dry-run

Use namespaces to limit sync scope to only your team’s resources. Check which resources are managed by kongctl before syncing:

kongctl get apis -o json

Slow plan generation

Symptom: Plans take a long time to generate.

Solution: Check the logs to identify the problem, and split large configurations as needed.

Enable trace logging to identify slow API calls:

kongctl plan -f config.yaml --log-level trace

Split large configurations into smaller files and plan them separately:

kongctl plan -f apis-batch-1.yaml
kongctl plan -f apis-batch-2.yaml

Check for 429 (rate limiting) status codes in trace logs.

High memory usage with file tags

When loading the same large file multiple times, extract only the fields you need. Files are cached within a single execution, so referencing the same file multiple times is efficient:

# Instead of loading the full spec repeatedly
apis:
  - ref: api-1
    name: !file ./common.yaml#api.name        # Loaded and cached
    description: !file ./common.yaml#api.desc # Uses cache
  - ref: api-2
    team: !file ./common.yaml#team.name       # Uses cache

Enable debug logging

Show detailed operation logs:

kongctl apply -f config.yaml --log-level debug

Show API requests and responses:

kongctl apply -f config.yaml --log-level trace

You can also set these via environment variables:

export KONGCTL_LOG_LEVEL=debug
kongctl plan -f config/

Trace log analysis

When trace logging is enabled, look for:

• 4xx/5xx HTTP status codes
• Slow response times
• Unexpected response bodies

Example trace output:

time=2024-01-15T12:00:00.000Z level=TRACE msg="HTTP request" method=GET url=https://us.api.konghq.com/v2/portals
time=2024-01-15T12:00:01.000Z level=TRACE msg="HTTP response" status=200 duration=1s

Step-by-step debugging workflow

Use this sequence to isolate issues:

1. Validate YAML syntax:

   cat config.yaml | python -m yaml

   Copied!
2. Test authentication:

   kongctl get portals

   Copied!
3. Generate plan with debug logging:

   kongctl plan -f config.yaml --mode apply --log-level debug --output-file plan.json

   Copied!
4. Review the plan:

   cat plan.json | jq '.changes'

   Copied!
5. Dry-run the apply:

   kongctl apply --plan plan.json --dry-run

   Copied!
6. Apply with trace logging:

   kongctl apply --plan plan.json --log-level trace

   Copied!

Verify configuration

Verify that the configuration file is valid:

cat ~/.config/kongctl/config.yaml

Test API connectivity

Make a direct API call to test API connectivity:

kongctl api /v3/portals

Check version

Ensure you’re running the latest version:

kongctl version --full

Update kongctl if needed following the install instructions for your platform.

Error patterns

Useful environment variables

# Set log level globally
export KONGCTL_LOG_LEVEL=debug

# Use a specific profile
export KONGCTL_PROFILE=production

# Set PAT for the default profile
export KONGCTL_DEFAULT_KONNECT_PAT="your-token-here"

1. Always dry-run (--dry-run) before applying in production.
2. Use version control for all configuration files.
3. Test changes in a lower environment before production.
4. Use namespaces to isolate changes between teams and environments.
5. Enable trace logging when debugging unexpected behavior.
6. Review plans before applying by using the two-phase plan and apply workflow.
7. Validate YAML syntax before deploying.
8. Check file paths are relative to the config file, not the working directory.
9. Monitor file sizes to stay under the 10MB limit.
10. Use protected: true on production resources to prevent accidental deletion.

If you’re still experiencing issues, do the following:

1. Check GitHub issues for similar problems.

2. Review GitHub discussions.

3. Open a new issue containing the following information:
   • kongctl version (kongctl version --full)
   • Operating system
   • Command that failed
   • Full error message
   • Debug logs (with sensitive data redacted)
4. Contact Kong Support if you’re an enterprise customer.