Test Automation for Application Security

This blog is co-authored by Matthew McCullough and is part three of a four-part series about DevSecOps.

Previously, the series explored a framework for continuous security and looked at one aspect of maintaining application security, a software Bill of Materials (BOM,) and associated vulnerabilities. This blog focuses on application security and how Cisco validates its software based on industry and internal security standards.

After an application is developed, multiple tests are run (e.g., unit, functional, regression, smoke, fuzzing) to ensure the application is ready to be deployed to Production. But beware. If integrated security testing isn’t included and scanning for web vulnerabilities isn’t completed, the app in production is vulnerable to an increasing number of attacks using various means such as injection, Cross-Site Scripting (XXS), or simple misconfigurations.

At Cisco, we use a variety of services that test and validate the security of our applications prior to release.  One of these internal services made available to our developers is the Cloud Automated Validation Engine (CAVE). CAVE provides Dynamic Application Security Testing (DAST), Secure Sockets Layer/Transport Layer Security (SSL/TLS) validation, and host configuration auditing. It also checks the application design against Cisco’s Security Controls Framework (SCF) as part of the Cisco Secure Development Lifecycle (SDL).

Components of CAVE

How developers run tests prior to deployment varies between our internal and public applications. To accommodate these different testing use cases, CAVE was built as a completely containerized solution that makes it highly versatile and scalable. Teams can spin-up one of the various CAVE containers, send a few attributes, and get meaningful results in minutes.  Developers who have access to an automated continuous integration/continuous delivery (CI/CD) pipeline or an automated testing framework can use the shared Jinkins shared library.

Dynamic Application Security Testing

One of the most prominent and beneficial features of CAVE is its ability to scan web applications for vulnerabilities using its Dynamic Application Security Testing (DAST) scanner. DAST provides black box testing for our web applications and APIs. The Open Web Application Security Project (OWASP) publishes a list of the top 10 Web application security risks and provides a Web Security Testing Guide. We have integrated the guidance provided by the OWASP in addition to our own validation tests to provide full coverage for emerging attack types.

Crypto Validation

The well-known Heartbleed Bug, which found vulnerabilities in SSL/TLS communications in OpenSSL, showcases why we should be validating the crypto being utilized. We utilized CAVE’s crypto validation container to verify that not only is the most secure version of TLS being used but also that a trusted, secure, and valid certificate is being used.

Host Checks

When web applications run on traditional hosts or virtual containers, host security validation cannot be ignored. No matter how secure application code might be, if the host is left misconfigured, secure development efforts are not effective.

A good starting point for identifying insecure host configurations is to use the benchmarks provided by the Center for Internet Security (CIS). CIS publishes hardening guidelines for all major operating systems and releases benchmarks that validate those guidelines. These benchmark baselines have been automated in CAVE, with checks included to simplify deployment as part of Cisco containerized services. Developers get a better understanding of the security of the application and what the application is running on.

Compliance Reporting

Whether developing an internal application or a public software product, developers will most likely need to demonstrate their product’s compliance with one or more industry standards.  Once a scan using the CAVE container is completed, the tool automatically matches the results against Cisco’s SCF. The SCF match can then be applied to individual certifications for use as artifacts.

Often teams must demonstrate compliance with certifications such as SOC2 or FedRAMP to external auditors. These artifacts provided through CAVE make demonstrating compliance easier for developers. Results are displayed to the user (Figure 1) and can also be exported in JSON to our central reporting hub where we have visibility into all our security compliance checks throughout all of Cisco’s offerings.

Lessons Learned

Cisco’s security tooling has evolved over time to adapt to the needs of our developers and customers. Here are a few lessons we’ve learned while developing CAVE that may prove valuable to others as they create application security services.

• Make your own rules – If you’re relying on out-of-the-box compliance tools or basing your security validations solely one industry-standard validations, you’re not doing enough to ensure the security of your application. Standard validations are written with a one-size-fits all approach. At Cisco, we have developed our own internal validations based on knowledge of our own solutions and company best practices from decades in the networking industry.
• Start with a good seed file – The tools we use are only as good as the data we provide to them. Simply pointing validation tools to an application URL and letting them crawl the hierarchy can miss potentially vulnerable sections. To increase a scan’s coverage, it’s important to provide a seed file that shows the tool how users navigate that application and let it crawl from there. In CAVE, that seed file usually takes the form of an HTTP Archive (HAR) file. Different users interact with your application in unique ways, create multiple seed files that capture each user role. For APIs, an OpenAPI definitions file is used, which outlines what should be scanned to ensure much wider coverage.
• There’s no replacement for manual penetration testing – While we try to automate most of our tests, sometimes there’s no substitute for manual testing. Automated security testing does not replace the need for manual penetration testing but is a great complement to it. Results from manual penetration testing can be fed back into an automation tool and used to create custom rules. Additionally, successful attack payloads from one product can be shared with other products to further enhance security.
• Warn before non-compliance – Compliance is usually pass or fail; you either meet the validation criteria or you don’t. However, we have found a lot of value in providing warning for future non-compliance due to certificate expirations. One example of this is in our crypto validation. CAVE does a relatively simple check for a certificate’s lifetime and then adds a warning if a certificate is close to expiring. This adds an extra safety net to better prevent unintentional expirations and notifies the developers before they are non-compliant.
• Make it simple – When we try to create tools that accommodate all of Cisco’s unique development environments and teams, we often find ourselves over-engineering and making complex internal solutions. With CAVE, we found simpler is better. CAVE’s containerized model allows for development teams to choose where and how they deploy it and customize it where needed.

Summary

With the complexity and frequency of web application attacks increasing, developers must invest in embedding security automation within their pipelines. Relying on manual security tests or usability testing to discover vulnerabilities alone doesn’t scale with the number of potential attack vectors that can slow down release velocity. There are some valuable tools and resources available to help validate application security.

We’d like to hear from fellow practitioners about what has worked for you in continuously testing the security of your applications. Please post your comments below and make sure to read the next blog in the series!

Visit our Trust Center

to learn about Cisco’s long-term commitment

to security and trust journey

Share: