A security issue can make a DoS attack in the ASM.
We find a docker security issue to exhaust all struct files in Linux Kernel and causing a DoS attack in the Alibaba ASM environment.
Reproduction steps:
- Follow the ASM tutorial to set up ASM clusters. We use two Virtual Machine with 16G memory, 120G ESSD Disk, CentOS 7.7 OS, Kubernetes Version V1.18.8-aliyun.1, and Docker Version 19.3.5, to set the Alibaba Kubernetes Cluster. All those settings are done through Alibaba ASM UI.
- Deploy the docker unprivileged malicious container with UID 1000, dropping all capabilities, using limited memory 4G, running on special core, and disable privilege escalation. We run the malicious container in a separate Kubernetes Namespace.
- Inside the malicious container, we repeatedly make the timerfd_create/open syscalls. In the malicious container, we start 2100 processes, each process with a 1024 max number of open files, which is the default ulimit value. In total, around 2097152 number struct-file are consumed and there is no available struct-file in the kernel. As a result, the victim container can't create any new files. The total consumed memory of the malicious container is less than 4G, which is small, so the memory control group cannot help.
Is there any way to defend against this attack inside the Alibaba ASM environment? Looking forward to your reply!
版权声明：本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
The security issue you've described involves a Denial of Service (DoS) attack through exhaustion of struct file resources in the Linux kernel, leveraging Docker containers within the Alibaba Cloud Container Service for Kubernetes (ACK) managed by Alibaba Cloud Application Service Mesh (ASM). To mitigate such an attack within the ASM environment, consider implementing the following strategies:
-
Resource Quotas and Limit Ranges: In Kubernetes, you can define Resource Quotas at the namespace level to limit the aggregate resource consumption, including the number of pods and other resources. Although this doesn't directly limit the number of open files per pod, it can help prevent the scale of the attack by limiting the total number of pods that can be created in a namespace.
-
Pod Security Policies (DEPRECATED): While Pod Security Policies (PSP) have been deprecated in favor of Pod Security Admission, if your cluster still supports PSP, you could create policies to restrict capabilities like the ability to run processes with high resource usage or specific syscalls. However, note that you should start transitioning to Pod Security Admission Controller as PSP is no longer recommended.
-
Pod Security Admission: This is the recommended approach replacing Pod Security Policies. Configure Pod Security Admission settings to enforce restrictions on what types of pods are allowed to run. You can set rules to disallow or limit certain behaviors, such as running privileged containers, using host namespaces, or making specific system calls.
-
Seccomp (Secure Computing Mode): Seccomp allows you to filter out potentially dangerous syscalls made by containers. By default, Docker uses a default profile that blocks some dangerous syscalls. You can customize a seccomp profile to specifically block
timerfd_createandopensyscalls or any others deemed risky for this type of attack. -
Namespace Resource Limits: Investigate if there's a way to set ulimits at the namespace level or container level more granularly than the default, to restrict the maximum number of open files per process. This would directly address the exploit vector you've outlined.
-
Kubernetes Event Monitoring and Alerting: Enhance monitoring and alerting around Kubernetes events, especially those related to resource utilization and pod creation, to quickly detect abnormal behavior that might indicate an ongoing attack.
-
Intrusion Detection/Prevention Systems (IDS/IPS): Consider deploying an IDS/IPS solution that integrates with your Kubernetes cluster to monitor and block malicious activity, including unusual syscall patterns.
-
Regular Updates and Patches: Ensure your Kubernetes cluster, Docker, and all components are regularly updated to the latest stable versions, which often include security patches addressing known vulnerabilities.
Given the complexity and potential impact of the described attack, it's advisable to engage with Alibaba Cloud support directly to discuss these mitigation strategies and possibly receive guidance on additional safeguards specific to the ASM environment. They may also be able to provide insights into upcoming features or configurations that could further harden your setup against such attacks.
