Acto | Yuxuan's Homepage

Cloud management platforms like Kubernetes rely on operators — programs that automate system operations using a state-reconciliation model. When an operator has a bug, it can silently leave managed systems in incorrect states, causing subtle and hard-to-diagnose failures in production.

Acto is the first automatic end-to-end testing framework for cloud system operators. It uses a state-centric approach: Acto continuously drives an operator to reconcile the managed system across a large state space, checking at each step whether the system correctly reaches the declared desired state.

Key contributions:

State-transition modeling of operator behavior for systematic state-space exploration
Automatic oracles that check correctness without requiring user-defined assertions
Found 56 serious new bugs (42 confirmed, 30 fixed) across 11 Kubernetes operators

(Gu et al., 2023)

Published at SOSP ‘23. Code available on GitHub.

References

2023

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

Jiawei Tyler Gu, Xudong Sun, Wentao Zhang, and 5 more authors

In Proceedings of the 29th Symposium on Operating Systems Principles, Koblenz, Germany, 2023

Abs DOI Bib PDF Code

Cloud systems are increasingly being managed by operation programs termed operators, which automate tedious, human-based operations. Operators of modern management platforms like Kubernetes, Twine, and ECS implement declarative interfaces based on the state-reconciliation principle. An operation declares a desired system state and the operator automatically reconciles the system to that declared state.Operator correctness is critical, given the impacts on system operations—bugs in operator code put systems in un-desired or error states, with severe consequences. However, validating operator correctness is challenging due to the enormous system-state space and complex operation interface. A correct operator must not only satisfy correctness properties of its own code, but it must also maintain managed systems in desired states. Unfortunately, end-to-end testing of operators significantly falls short.We present Acto, the first automatic end-to-end testing technique for cloud system operators. Acto uses a state-centric approach to test an operator together with a managed system. Acto continuously instructs an operator to reconcile a system to different states and checks if the system successfully reaches those desired states. Acto models operations as state transitions and systematically realizes state-transition sequences to exercise supported operations in different scenarios. Acto’s oracles automatically check whether a system’s state is as desired. To date, Acto has helped find 56 serious new bugs (42 were confirmed and 30 have been fixed) in eleven Kubernetes operators with few false alarms.
@inproceedings{10.1145/3600006.3613161, author = {Gu, Jiawei Tyler and Sun, Xudong and Zhang, Wentao and Jiang, Yuxuan and Wang, Chen and Vaziri, Mandana and Legunsen, Owolabi and Xu, Tianyin}, title = {Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management}, year = {2023}, isbn = {9798400702297}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3600006.3613161}, doi = {10.1145/3600006.3613161}, booktitle = {Proceedings of the 29th Symposium on Operating Systems Principles}, pages = {96-112}, numpages = {17}, keywords = {kubernetes, operation, system management, cloud, reliability, operation correctness, operator}, location = {Koblenz, Germany}, series = {SOSP '23}, bibtex_show = true, }