Skip to content

Understanding the Pipeline

Every container in Iron Bank utilizes the Container Hardening pipeline. At a high level, the pipeline is responsible for the following:

  • ensure compliance with CHT project structure guidelines and check for the presence of required files.
  • retrieve the resources used in the container build process and build the hardened container.
  • perform scans of the container in order to evaluate for security vulnerabilities.
  • publish scan findings to the VAT (Vulerability Assessment Tracking tool) and the hardened container to Registry1.

Pipeline Documentation

This page provides an overview of the pipeline and includes information which may be helpful in getting started with the pipeline when onboarding. For in-depth information on the pipeline, including specific stage writeups and to view the code, please access the IronBank Pipeline Documentation.

Getting Started

The first thing you need to know when beginning to onboard an Image to Iron Bank is which Base Image you would like to use. The Container Hardening pipeline can be slightly different depending on this information, however we have built in logic to check this and build the image regardless of which Iron Bank base image you use. Check out our guide for Choosing a Base Image.

The pipeline contains a linting stage which occurs early on in the running of the pipeline which enforces the correct repository structure and presence of required files. This stage is run in order to ensure compliance with Container Hardening team requirements. Please ensure that your repository structure meets the guidelines provided by the Container Hardening team and that the necessary files exist, otherwise, the Container Hardening pipeline will fail as a result.

Artifacts

The Container Hardening pipeline utilizes GitLab CI/CD artifacts in order to run stages which are dependent on previous stage resources and to provide Repo1 contributors with access to the artifacts produced by the pipeline which are relevant to them. This can include a tar file of the built image or a scan results file which contributors can use to enter justifications for any security findings, for example.

For more information regarding specific artifacts for each stage, please refer to the following documentation from the Container Hardening pipeline project.

Project Branches

The Container Hardening pipeline runs slightly differently depending on the hardened container project branch it is run on. The reason for this is to ensure a proper review of the repository content before the code is merged into a development branch or master branch, which publish items to production systems.

Branch Pipeline actions
Feature branches Does not publish images to public Registry1 or findings results to the Iron Bank
development Does not publish images to public Registry1 or findings results to the Iron Bank. Merging is required to be performed from the Container Hardening Team, but you can open the merge request
master Publishes images to public Registry1 and findings results to the Iron Bank. Merging is required to be performed from the Container Hardening Team, but you can open the merge request

Pipeline Stages

pipeline-overview

.pre / Lint

These preliminary stages set up the workspace and work together to enforce the presence of required files and directories in the repository. The formatting of various project files (Dockerfile, hardening_manifest.yaml, etc.) is also validated.

Import Artifacts

This stage will import any external resources (resources from the internet) provided in the hardening_manifest.yaml file for use during the container build. It will download the external resources and validate that the checksums calculated upon download match the checksums provided in the hardening_manifest.yaml file.

Build

The container build process takes place in an offline environment. The only resources that the build environment has access to are the resources included in the hardening_manifest.yaml, images from Registry1, and the following sources:

  • Pypi, for Python based containers using the pip package manager,
  • Golang proxy for go based containers.
  • npm for Node.js images.
  • Rubygems for Ruby projects that use gem and bundler.

Any attempts to reach out to the internet will be rejected by the cluster's egress policy and the build stage will fail.

For more insight into the build stage, please refer to the following documentation from the Container Hardening pipeline project.

Post Build

The post-build stage of the pipeline runs two jobs.

create-tar outputs a Docker archive of the image, as a tarball. This tarball can be downloaded, and run locally to test the built image.

docker load -i <path-to-archive>

The create-sbom job makes use of Anchore's syft product, to generate spdx, CycloneDX, and syft JSON reports.

SBOM Access Log

In this stage, we take the access log, created by the build job, and the syft JSON report, and parse them. The purpose is to allow for the pipeline to be able to diff the parsed outputs, to determine if a build resulted in any changes to the software in the image.

Scanning

Security Vulnerability and Compliance Scanning

These scans are run to ensure any vulnerabilities are accounted for (such as dependency security flags). The scan results are then generated so that an auditor can manually review potential security flags that were raised in the scan. Should the auditor determine the scan results satisfactory, publication of the container(s) to the Iron Bank will occur.

The pipeline begins by pulling the container image from Repo One, and proceeds to run a series of scans on the container:

  • OpenSCAP Compliance

The OpenSCAP Compliance scan enables us to provide configuration scans on container images. The particular security profiles that we use to scan container images conforms to the DISA STIG for Red Hat Enterprise Linux 8 and 9.

  • Twistlock

The Twistlock scan checks container images for vulnerabilities. Please visit paloalto networks for more information.

  • Anchore

The anchore-scan job run vulnerability, compliance, and malware scans.

The anti-virus/anti-malware scanning is performed by the open source tool ClamAV. ClamAV has a database of viruses and malware which is updated daily.

Please contact Anchore directly for an enterprise license or more information. You can visit the Anchore documentation to get started and learn more.

After these scans have completed, the pipeline takes an output of the scan findings, and stores them to be accessed through the Iron Bank. This concludes the Scanning portion of the pipeline.

VAT

The VAT stage uses pipeline artifacts from the scanning stages to populate the Vulnerability Assessment Tracker (VAT) at vat.dso.mil. Users who maintain the hardening of an image use the VAT to track the known findings for an image, and provide justifications for findings that are not able to be mitigated. The VAT also calculates the Acceptance Baseline Criteria (ABC) status and Overall Risk Assessment (ORA) score.

CSV Output

The csv-output stage will generate CSV files for the various scans, and the Justifications spreadsheet. These documents can be found in the artifacts for this stage. The Justifications spreadsheet is created so that Container Hardening team members, and vendors/contributors can access the list of findings for the container they are working on. It is a compilation of the findings generated from the scanning stages.

Check CVEs

This stage utilizes the API response artifact from the VAT stage, to log all findings which have not been justified or reviewed. With the release of the ABC/ORA, this job is now for informational purposes, to alert project maintainers to new or unjustified vulnerabilities.

Update Staging Attestions

Images built from the development and feature branches do not have their VAT predicate attestations uploaded by default. But there are cases where you do want to use an image from staging (which are built from development and feature-branches) as your base parent image. The pipeline will fail because vat attestation predicates from the parent image are needed for an accurate attestation chain of the child image.

To enable using a staging parent image:

Run the parent image pipeline on the protected non-master branch with the PUBLISH_VAT_STAGING_PREDICATES CI variable set to "True".

Note that only maintainers of a project can make a branch protected.

When that pipeline completes, run the child image pipeline with STAGING_BASE_IMAGE CI variable set to "True". You child pipeline will now be able to pass the VAT stage because it can download the vat attestations of the parent image.

Master Branch runs

Info

As mentioned above, the following stages of the pipeline will only run on master branches.

Documentation

This stage provides validation of the container to end users by generating documents that contain information about the pipeline run and image that was built. An example would be the versions of the scan tools used, and the image digest created after it is built.

Harbor

Pushes built images, SBOM files, and VAT response file as an attestation, to registry1.dso.mil/ironbank. The image and SBOM attachment are signed using cosign. See the ironbank-pipeline cosign documentation for instructions on how to perform image validation, and download .sbom, .att, and .sig artifacts.

S3 Publish

Uploads artifacts to S3, which are displayed/utilized by the Iron Bank website (e.g. scan reports, project README, project LICENSE).