Skip to main content

EKS Scanning Q&A

Chronom is designed to be used for scanning not only AWS resources, but also Kubernetes resources, especially resources inside an AWS EKS cluster.
this is why we have created a special feature that allows you to scan your EKS clusters, even if they are private and or not accessible from the internet.

Q: What can Chronom Do inside my Cluster​

A: By default, Chronom cannot scan your cluster, but you can easily configure it to do so​

Chronom uses the built in authentication mechanism of AWS EKS in combination with the Authorization mechanism of Kubernetes RBAC, to allow fine-grained, precise permissions that only grant Read Access.

Authentication​

Chronom uses the AWS IAM Role it already has to authenticate to the AWS EKS API, which in turn converts the identity of IAM to Kubernetes Identity.

For more information.

Authorization​

Once Chronom is authenticated, it then goes trough the Kubernetes Built-In Role-Based Access Control for it’s Authorization.
This means that Chronom will be able to access the kubernetes API trough the AWS EKS API, but will not be able to modify any resources.

The transition of “Authorization” is seamless and requires no additional intervention during the authentication and or the authorization process.

for more information.

Q: How can i configure Chronom to scan my cluster?​

A: Simply follow the Instructions inside the Chronom Alert to execute the Chronom-Cli command that will grant your Chronom access to the cluster​

Once Chronom detects an EKS Cluster that is not being scanned, it will automatically create an Alert that will guide you trough the process of configuring Chronom to scan the cluster.

❗ It is important to note that while Chronom itself does not require direct access to the Kubernetes API, The Configuration steps do require network and RBAC access to the Kubernetes API

If you do not see a notification, you can follow the steps below to manually construct the command:

For Publicly accessible clusters when using AWS CloudShell

  1. Identify the EKS Cluster Object inside your Chronom (You can do it by searching for it’s name or by filtering out only AWS - Eks Cluster objects).
  2. Left-Click on the object and choose “Configuration”.
  3. Locate the Configuration Field called “clusterCommand” and copy it.
  4. Open AWS Console CloudShell in the account where the EKS Cluster is deployed (It is also recommended to use the same region - for better performance).
  5. Execute the following commands in the order presented:
curl -O https://raw.githubusercontent.com/chronom-ai/chronom-cli/main/chronom-cli
chmod +x chronom-cli
./chronom-cli setup cloudshell
  1. Inside the same terminal in the same session execute the command you copied in step 3.
  2. That is it, your cluster will be scanned during the next scan.

For Clusters that are not accessible from AWS CloudShell

  1. Repeat steps 1 to 3 to find the the command.
  2. Open a Bash Shell in a location that has network access to the cluster API (most likely a VPN connection and or a JumpBox is required).
  3. Install the Pre-Requisites to run Chronom-Cli (a warning will be displayed if any dependency is missing).
  4. Execute the following commands in the order presented:
curl -O https://raw.githubusercontent.com/chronom-ai/chronom-cli/main/chronom-cli
chmod +x chronom-cli
  1. Inside the same terminal in the same session execute the command from step 1.
  2. That is it, your cluster will be scanned during the next scan - No network required.

Q: Why do i need to manually execute the commands?​

A: Due to the way Chronom is designed, we cannot automatically configure your cluster nor can we execute the commands for you​

By design, Chronom has Read-Only IAM Policy that is assumed by the Chronom Read Only Role during the scan operation.

Due to that fact, Chronom does not make any changes to your environment, regardless of the resource type.

The commands we provide are simple “wrapping” of existing AWS, Kubernetes and EKSCTL commands in a specific order for a specific pre-determined end result.

The commands are designed to be executed by the user who has access to the cluster, and therefor we cannot execute them on your behalf.

Q: What exactly does the chronom-cli configure chronom additional-cluster-eks do?​

A: The command performs 3 simple steps that will not affect the cluster configuration in any way​

Step 1 - Generate temporary credentials for the cluster​

Using the built in aws eks get-token along with aws eks describe-cluster Chronom-CLI is able to generate a time-limited Kubernetes Connection credentials, that are used to authenticate with the Native Kubernetes API Endpoint, allowing you to execute commands directly against the cluster, without affecting your current work or even changing the kubectl context on your shell (just in case you are multi tasking).

The credentials are being issued using the identity of the user who is authenticated currently, to find out which user it is execute aws sts get-caller-identity

If you are using AWS Roles to access the cluster (which is the recommended way), please execute the following command prior to executing the chronom-cli command:

export AWS_PROFILE="<your profile name>"

Step 2 - Create a new Read Only Cluster Role and Cluster Role Binding in the Cluster​

Using the authentication information from Step 1, the CLI then executes kubectl apply to the following YAML:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: chronomReadOnlyRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: chronomReadOnlyRoleBinding
subjects:
- kind: Group
  name: chronomReadOnlyGroup
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: chronomReadOnlyRole
  apiGroup: rbac.authorization.k8s.io

The above YAML creates two Kubernetes Objects:

  • ClusterRole - A new Role named chronomReadOnlyRole that has the following permissions:
    • kubectl get *
    • kubectl list *
    • kubectl watch *
  • ClusterRoleBinding - A new Role Binding named chronomReadOnlyRoleBinding that binds the Role to a new Group named chronomReadOnlyGroup.

Step 3 - Create a new Entry inside the aws-auth ConfigMap​

The final step is simply connecting all the dots.

by using the eksctl create identitymapping command, Chronom-Cli creates a new entry in the aws-auth configMap that binds the ARN of the Read Only Role to the Cluster Role that was created in step 2.

The binding does not affect any other bindings that might exist in the configMap, and is only used by Chronom to access the cluster.

If at any point you wish to remove the binding, you can simply execute the following command:

eksctl delete iamidentitymapping --cluster <cluster name> --arn <role arn>

The entire Code​

configure_additional_cluster_eks() {
    clusterName="$1"
    region="$2"
    chronomReadonlyRoleArn="$3"
    
    
    chronomReadOnlyRoleName=$(echo "$chronomReadonlyRoleArn" | cut -d'/' -f2)
    
    echo "# Generating Temporary Credentials for the EKS Cluster Kubernetes API Server"
    token=$(aws eks get-token --cluster-name "$clusterName" --region "$region" --query "status.token" --output text)
    certificate=$(aws eks describe-cluster --name "$clusterName" --region "$region" --query "cluster.certificateAuthority.data" --output text)
    endpoint=$(aws eks describe-cluster --name "$clusterName" --region "$region" --query "cluster.endpoint" --output text)
    
    pemFile=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 8 | head -n 1).pem
    echo $certificate | base64 -d > $pemFile
    echo "# Temporary Credentials generated successfully"
    
    
    echo "# Checking wether user has access to the cluster"
    canI=$(kubectl --server $endpoint --token $token --certificate-authority $pemFile auth can-i '*' '*' --all-namespaces || echo "no")
    if [ "$canI" == "no" ]; then
        echo "# Unfortunately chronom-cli failed to reach the cluster"
        echo "# The most common reason for this is either the user does not have access to the cluster or the cluster is private (or both)"
        endpointIp=$(dig +short "$(echo $endpoint | sed 's/https:\/\///g')" | head -n1)
        if [[ $endpointIp =~ ^10\. || $endpointIp =~ ^172\.1[6-9]\. || $endpointIp =~ ^172\.2[0-9]\. || $endpointIp =~ ^172\.3[0-1]\. || $endpointIp =~ ^192\.168\. ]]; then
            echo "# The cluster is private, please make sure you are connected to the cluster's VPC - This might not be the issue, but it's worth checking :)"
        fi
        echo "# Please make sure you execute the command with a user that has access to the cluster"
        cloudTrail=$(aws cloudtrail lookup-events --region "$region" --lookup-attributes "AttributeKey=ResourceName,AttributeValue=$clusterName" --output json)
        possibleUsers=$(echo "$cloudTrail" | jq -r '.Events[].Username' | sort | uniq)
        if [[ -n $possibleUsers ]]; then
            echo "# According to CloudTrail, the following users might have access to the cluster:"
            echo "$possibleUsers"
        fi
        echo "# If you are sure you have access to the cluster, please contact Chronom Support and we will be happy to assist you"
        echo "# You can contact us via email at support@chronom.ai"
        exit 1
    fi
    echo "# User has access to the cluster, proceeding with the configuration"
    
    
  kubectl --server $endpoint --token $token --certificate-authority $pemFile apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: chronomReadOnlyRole
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: chronomReadOnlyRoleBinding
subjects:
- kind: Group
  name: chronomReadOnlyGroup
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: chronomReadOnlyRole
  apiGroup: rbac.authorization.k8s.io
EOF
    eksctl create iamidentitymapping --cluster "$clusterName" --region "$region" --arn "$chronomReadonlyRoleArn" --group chronomReadOnlyGroup --username "$chronomReadOnlyRoleName"
    
    rm $pemFile
}

Q: I got an error saying “Unfortunately chronom-cli failed to reach the cluster”, what does it mean?​

A: The error you received means that Chronom-Cli could not access the cluster​

This error is quite common because of how AWS EKS works with AWS IAM.

While there are several reasons why this error might occur, the most common ones are:

1. The cluster is a Private EKS Cluster​

This means that the location you are executing the command from does not have network access to the cluster API (Most likely AWS CloudShell).

The solution is to execute the command from a shell that has network access to the cluster API, usually this means a VPN Connection or a JumpBox VM, but your milage may very depending on your organization’s architecture.

2. Your AWS Identity is not Authorized to access the Kubernetes API​

This means that the AWS Identity you are currently using to execute the command does not have access to the cluster.
This is not a problem, as it is recommended both by Kubernetes and AWS to limit access to the cluster to only the users who need it.

While this is not a problem, it does mean that you will need to execute the command with a different AWS Identity who has access to the cluster, or ask your cluster administrator to execute the command for you.

In attempt to make it easier Chronom-Cli will attempt to query CloudTrail and identify AWS IAM Identities that had successfully accessed the EKS Cluster, but this solution has it’s limits, mainly the fact that CloudTrail by default stores data for only 90 days. For more information.

3. A combination of 1 & 2​

It is possible that your EKS Cluster is set up with Best Practices in mind, therefor both reasons 1 & 2 are true, in which case you will need to execute the command from a shell that has network access to the cluster API, and also has an AWS Identity that has access to the cluster.

Q: Can i get a deeper explanation on what’s going on under the hood?​

A: Absolutely, Here is everything you need to know about how Chronom interacts with your cluster​

In order for Chronom to scan EKS clusters, we must first satisfy both the Authorization and Authentication conditions.

Satisfying Authorization Conditions​

What is Authorization?​

Authorization is the process of determining what actions a certain identity can / cannot perform.
Authorization has several strategies, but the most common one is Role-Based Access Control (RBAC).

RBAC is a strategy that allows you to create Roles that have a set of permissions, and then bind those roles to Identities (Users, Groups, Service Accounts, etc…).

The RBAC Strategy allows you to create fine-grained permissions that are easy to manage and audit.
This means that you can control both who can access the cluster, and what actions they can perform, but also on which resources.

In Kubernetes, RBAC is a built-in mechanism that is enabled by default, and is used to control access to the Kubernetes API.

How does Chronom use Authorization?​

Chronom relies on the built-in RBAC mechanism of Kubernetes in combination with the reliable AWS IAM mechanism to create a fine-grained, precise permissions that only grant Read Access.

For more information about RBAC in Kubernetes. For more information about IAM in AWS. For more information about AWS EKS Authentication.

Satisfying Authentication Conditions​

What is Authentication?​

Authentication is the process of verifying the identity of an entity, such as a user, a service account, or a system component.

Authentication is usually done by using a combination of a secret (such as a password or a private key) and an identity (such as a username or a public key).

Authentication is the process in which the entity proves it’s identity to the system, and can be done in several ways, the most common ones are:

  • Username & Password
  • API ID & API Secret
  • Public Key & Private Key
  • Client Certificate
  • Time-Based One-Time Password (TOTP)
  • Token
How does Chronom use Authentication?​

Chronom uses the AWS IAM Read-Only Role it already has to authenticate to the AWS EKS API, which in turn converts the identity of IAM to Kubernetes Identity.
The process itself is seamless and requires no additional intervention during the authentication and or the authorization process.

Chronom relies on the built-in authentication mechanism of AWS EKS to authenticate to the Kubernetes API, which is constructed of two parts:

  • AWS IAM (Already satisfied by the IAM Policy).
  • EKS aws-auth configMap (More information).

To configure the aws-auth configMap (a built-in object in any AWS EKS Cluster), Chronom-CLI uses the eksctl commands to add the Read-Only IAM Role to the configMap. This means that the Role will now be treated as a Kubernetes User inside the cluster, and any RBAC or Audit Logs will reflect that.

Of course, there is nothing preventing you from manually updating the ConfigMap yourself, but we have found that using the AWS recommended way with eksctl cli is simpler and therefore less prone to errors.

The ConfigMap holds no sensitive authentication information and simply stores the following fields:

  • RoleArn - The ARN of the Read-Only Role Chronom already uses to scan AWS.
  • Username - An arbitrary Name that will be logged whenever Chronom scans the cluster.
    In the cli we configure it to be the name of the role, to allow easier traceability.
  • Groups - The permission group that is bound to the Identity, determining what actions it will have.
    In the cli we configure it to be “chronomReadOnlyGroup” which is created during the Authorization stage.

Q: What about Kubernetes Secrets?​

A: Chronom does not store your Kubernetes Secrets as clear text, period.​

During the scan process, Chronom will scan all the Kubernetes Secrets inside the cluster, and will store them in the Chronom Database.
Although Chronom has a Read-Only IAM Policy, we understand that some organizations might be uncomfortable with storing their Secrets in clear text, even if it is only for a short period of time.
This is why we have created a special mechanism that allows Chronom to store the Secrets in a Hashed format, while still allowing Chronom Algorithm to detect changes in the Secrets as well as create Relationships between Secrets and other resources.

Chronom uses two separate mechanisms to store the Secrets:

1. Hashed Secrets​

Chronom uses a Hashing Algorithm (SHA256) to hash the Secrets before storing them in the database.
This means that the Secrets are not stored in clear text, and cannot be decrypted.
We have chosen SHA256 because it is a one-way hashing algorithm, meaning that it is impossible to decrypt the Secrets, even if you have the Hash.
This allows Chronom to detect changes in the Secrets, without storing them in clear text.

2. Encrypted One time Secret​

Chronom uses the AES256 Encryption Algorithm to encrypt the Secrets before storing them in the database.
But unlike the Hashed Secrets, the Encrypted Secrets are encrypted using a One-Time Secret that is generated for each scan, meaning that once the scan is complete, the Secret is no longer accessible no matter what.
In addition to the One-Time Secret, Chronom also uses a second Key that is being rotated at a random interval, meaning that even if you have the One-Time Secret, you will not be able to decrypt the Secrets after the Key has been rotated.
By using two random keys, Chronom ensures that even if the One-Time Secret is compromised, the Secrets will still be safe.

Chronom uses both mechanisms to ensure that the Secrets are safe, and that even if the database is compromised, the Secrets will not be accessible, and as a matter of fact, the Secrets stored in Chronom are not even accessible to YOU, the user, as they are only used by the Chronom Algorithm to detect changes and create Relationships.

You may notice that some secrets have "********" as their value, instead of the one time encrypted secret.
This is because, as mentioned before, Chronom only uses the Secrets to detect changes and create Relationships, and therefor does not need to store certain Secrets in the database, and instead stores a placeholder value.
(For example: Certificates, Private Keys, etc…).