GitLab's On-Premise Device Cloud CI/CD Component Brings Efficiency

September 11, 2024

Blog

In the world of embedded systems development, efficient testing and deployment processes are crucial for success. The GitLab Device Cloud CI/CD component offers a solution to streamline these processes by enabling hardware-in-the-loop (HIL) testing and global device sharing.

This approach addresses common bottlenecks in embedded development workflows, allowing for faster iteration cycles and reduction in labor and hardare under-utilization costs due to inefficient manual shared hardware scheduling.

This lightweight, on-premise device cloud is built entirely with GitLab functionality within a GitLab CI/CD component and offers a standardized, secure, and efficient way to manage and share hardware resources, accelerating embedded development workflows and reducing costs.

This is an orchestration system that allows for the allocation and deallocation of runner gateways attached to embedded, IoT, or mobile devices. At its core, the On-Premise Device Cloud Component creates pools of devices that can be allocated and deallocated (orchestrated) within GitLab CI pipelines. It gives exclusive access to the allocated pipeline and evicts any jobs that attempt to run from sources outside the current pipeline.

The component was designed and built by me and is maintained as a community component.

Key Benefits

The key benefits roughly sort into strategic gains in velocity and reduction of pain in terms of Cost.

Accelerated Development Cycles: By integrating hardware-in-the-loop testing directly into CI/CD pipelines, developers can receive faster feedback on their code changes. This significantly reduces the time between writing code and testing it on actual hardware.

Global Device Sharing: The On-Premise Device Cloud enables teams to share expensive or limited hardware resources across different locations. This maximizes device utilization, wasted developer wait time and can reduce overall prototype hardware costs.

Embedded Hardware Agnostic: The On-Premise Device Cloud has a fundamentally simple technical structure that should allow it to operate against virtually any Hardware Under Test (HUT). A host device has GitLab Runner configured for it. The Runner supports many host devices from Raspberry PIs to full Windows Desktop setups - whatever is required for the normal hardware interfacing at the developer desk. Frequently this will involve wiring up debug interfaces. This Hardware Gateway approach means that GitLab Runner should be able to manage any hardware that you can currently manage from a desktop.

Incremental Testing: The solution supports a gradual approach to testing, from emulation to representative hardware and finally to production devices. This allows teams to catch issues earlier in the development process, reducing the cost and time associated with late-stage bug fixes.

Automated Device Orchestration: By automating the process of allocating and deallocating devices, the On-Premise Device Cloud removes manual bottlenecks in embedded workflows. This automation reduces human error and frees up developer time for more critical tasks. It ensures exclusive pipeline to device locking to ensure that only jobs from the allocated pipeline are allowed to successfully launch on an allocated runner.

Frictionless Developer Experience: Since it is implemented as a GitLab CI/CD Component and is designed with a least configuration principle, the developer experience is simplified to including the component and giving a few input parameters. The On-Premise Device Cloud Component is also discoverable in the GitLab CI/CD Component Catalog on GitLab.com and can be copied to self-managed instances.

Flexible Configuration: The component supports both simple and complex setups, allowing teams to scale their testing infrastructure as needed. It works with both shell runners and container runners, providing flexibility in how tests are executed. Multiple different hardware pools can be leveraged in the same pipeline for more complex scenarios.

Enhanced Security: The solution implements least privilege ensuring that access to shared hardware resources requires the least possible permissions to the least scope of orchestration data.

Hardware Efficient Pipelines: Pipelines only need to allocate hardware for the jobs that need it and, when necessary, non-hardware-in-the-loop jobs can still run on regular cloud runners within hardware in the loop segments. This helps ensure that hardware is only allocated for necessary tasks and nothing else.

Maintenance Flexibility: Runners can be easily drained or taken offline for maintenance by simply changing allocation variables, minimizing disruption to ongoing development work. Automation teams can use CI jobs in the hardware pool data project to create performance cluster maintenance such as detecting orphaned allocations, permanently offlined hardware, hanging hardware and other maintenance conditions.

Implementing the GitLab Device Cloud Component for Embedded Development

The GitLab Device Cloud Component provides a solution for managing hardware-in-the-loop (HIL) testing and global device sharing for embedded development workflows. We will now walk through the steps to implement this component in your GitLab CI/CD pipeline.

Prerequisites

• A GitLab instance with CI/CD capabilities

• Access to create projects and groups in an existing GitLab group

• Hardware devices to be used as runner gateways (e.g. Raspberry Pis)

• Embedded devices for testing (e.g. Raspberry Pi Picos) - optional if you are doing a proof of concept that does not access the hardware under test.

You can still see the Device Cloud in action with a single device POC, as it still has to allocate and deallocate the device to use it.

Group Organization of Example

The example names used below will assume this group structure in GitLab. It is important to understand that the group structure is not dictated by the example and not to assume that only one runner pool type can be active in one group hierarchy at a time. It will be fundamentally more difficult to manage if the runners are not placed in a parent group of all the projects they might need to be used in. This is simply a normal constraint of how runners work and not specific to this component.

Embedded Projects Runner Group:
../some/group/embedded

Embedded Projects that Use the On-Premise Device Cloud:
children of ../some/group/path/embedded like ../some/group/path/embedded/infotainment/project1, ../some/group/path/embedded/enginecontroller/project3, etc.

Projects For Device Pool Data Management : ../some/group/path/embedded/deviceclouds/rpipicow (These are projects not groups)

The location of the Device Pool Projects is designed for least privilege so that a given token for accomplishing orchestration does not inherit down a group hierarchy (the token needs higher permissions to manage CI/CD Variables)

Step 1: Oneliner Template in Code Editor

Open your favorite code editor and paste in this oneliner command line template:

wget https://gitlab.com/guided-explorations/embedded/ci-components/device-cloud/-/raw/main/Device-Cloud-Pool-Mgmt/PROVISIONDEVICEGWRUNNER.sh?ref_type=heads -O /tmp/PROVISIONDEVICEGWRUNNER.sh ; sudo bash /tmp/PROVISIONDEVICEGWRUNNER.sh --runnergroupurl=https://gitlab.com/mygroup/embedded-runners --runnergroupaccesstoken=glpat-XXXXXXXXXXXXXXXXXXXX --devicepooldataprojectpath=some/group/path/embedded/deviceclouds/rpipicow --devicepooldataprojectaccesstoken=glpat-YYYYYYYYYYYYYYYYYYYY –pipelineidlocking

As you proceed through the following steps, update this template command line as noted.

Step 2: Create or Locate a Suitable Runner Group

Generally, this would be a parent of the code projects that will need to use the runners.

This value will be used in the provisioning script for the argument; --runnergroupurl (and you can copy it directly into the command line you are building up for Step 3).

Create a new group for runners or use an existing one:

• In this example we are representing this group as some/group/path/embedded
• Copy the full group url including the URI, for example: https://mygitlabinstance.com/some/group/path/embedded

Step 3: Create the Device Pool Management Project

Create a new GitLab project to house the HIL pool data:

• Navigate to (or created) your desired GitLab group (../some/group/path/embedded/deviceclouds)
• Click "New project"
• Name it "rpipicow"
• Set visibility as needed (internal or private recommended)

Set the project's friendly name:

• Go to Settings > General
• Set the "Project name" to something descriptive like "Pico W Device Cloud"

Create a project access token.

The token value will be used in the provisioning script for the argument: --devicepooldataprojectaccesstoken (and you can copy it directly into the command line you are building up for Step 3) and for the variable DEVICE_POOL_TOKEN.

• Within the Project, Go to Settings > Access Tokens
• Create a new token with:
  • Name: "PROVISIONING_FOR_RPIPICOW"
  • Role: Maintainer
  • Scope: api

Copy the project path only (from the root of the instance url):

This value will be used in the provisioning script for the argument; --devicepooldataprojectpath (and you can copy it directly into the command line you are building up for Step 3)

From the project homepage, copy the path after the hostname (e.g. "some/group/path/embedded/deviceclouds/rpipicow")

Step 4: Configure the Runner Group

In this example we are allowing the projects under some/group/path/embedded to all inherit the variables to locate and access the CI/CD Variables.

Another more granular approach is to have groups request these values in order to configure a given project to access a given Device Cloud. In that case these variables would then be coded at the individual project level rather than the top group level like below. It might add a little more security to not give the Device Cloud users more than guest permissions to the some/group/path/embedded/deviceclouds group so that they cannot find the group ids without requesting them through appropriate channels.

These variables should never be stored right in pipeline files as one is a secret and the other discloses the project location.

In this example we are using some/group/path/embedded so that all projects inherit the location and access token for the given Device Cloud.

Create a CI/CD variable for the device pool token:

• In the runner group, go to Settings > CI/CD > Variables
• Add a new variable:
• Key: DEVICE_POOL_TOKEN
• Value: [paste the project access token created earlier]
• Type: Variable
• Environment scope: All (default)
• IMPORTANT Protect variable: Unchecked
• Mask variable: Checked

Create a group access token:

This token name will appear as the user who created the runner.

This value will be used in the provisioning script for the argument; --runnergroupaccesstoken (and you can copy it directly into the command line you are building up for Step 3)

• In the runner group, go to Settings > Access Tokens
• Create a new token with:
  • Name: "RUNNER_PROVISIONING_FOR_RPIPICOW"
  • Role: Owner
  • Scopes:
    • Create runner
    • Read API
    • Read repository
• Copy the generated token value

Step 5: Provision Runner Gateways

For each Raspberry Pi or other device you want to use as a runner gateway you will need to run a constructed command line.

You can still see the Device Cloud in action with a single device as it still has to allocate and deallocate the device to use it.

1. SSH into the device
2. Run the provisioning script with the following command (replace placeholders with your values). This is the command line you have been building in the text editor:

wget https://gitlab.com/guided-explorations/embedded/ci-components/device-cloud/-/raw/main/Device-Cloud-Pool-Mgmt/PROVISIONDEVICEGWRUNNER.sh?ref_type=heads -O /tmp/PROVISIONDEVICEGWRUNNER.sh ; sudo bash /tmp/PROVISIONDEVICEGWRUNNER.sh --runnergroupurl=https://gitlab.com/mygroup/embedded-runners --runnergroupaccesstoken=glpat-XXXXXXXXXXXXXXXXXXXX --devicepooldataprojectpath=some/group/path/embedded/deviceclouds/rpipicow --devicepooldataprojectaccesstoken=glpat-YYYYYYYYYYYYYYYYYYYY --pipelineidlocking

3. The script will install GitLab Runner, register it with your GitLab instance, and provision the runner into the Device Cloud by creating a CI/CD variable.
4. For the first one you run, validate that there is a new runner under (update for your actual group) https://mygitlabinstance.com/some/group/path/embedded/-/runners
5. Make note of the shortest runner tag.
6. In the Device Cloud Data Project, validate that there is a new CI/CD Variables whose name matches the tag and whose value is "ONLINE_AND_AVAILABLE". In this example that would be found by expanding "CI/CD Variables" at: https://gitlab.com/dsanoy-demo/experiments/rpi-pico-w/-/settings/ci_cd/rpipicow/-/settings/ci_cd
7. Repeat running the oneliner for each runner gateway device (you can choose whether to manually verify each addition or not)

Once this step is complete, you have a provisioned on-premise device cloud and the projects under https://mygitlabinstance.com/some/group/path/embedded automatically inherit the runners and the two variables required to allocate and deallocate individual devices in the device cloud.

Step 6: Implement the Device Cloud Component in Your Project

1. In your embedded development project, create or edit the .gitlab-ci.yml file.

2. Add the following configuration to include the Device Cloud Component:

include:

component: 'gitlab.com/guided-explorations/embedded/ci-components/device-cloud/[email protected]'

example-non-hil-job:

stage: build

image: hello-world

script: echo "Hello from a pipeline fake build job."

device-ci-allocate:

stage: hil_test

hil-childpipeline:

stage: hil_test

• needs:
  • job: device-ci-allocate
  • artifacts: true

 variables: RUNNER_LOCKING_PIPELINE_ID: $CI_PIPELINE_ID

device-ci-deallocate:

• needs:
  • job: device-ci-allocate
  • artifacts: true
  • job: hil-childpipeline
  • artifacts: true

stage: hil_test

For efficiency you would not want to allocate a Device Cloud device before you needed it (for instance before your build job)

Create a file named device-pipeline.gitlab-ci.yml in your project root with your HIL testing jobs:

While non-hil jobs are allowed in here, the Device Cloud device also stays allocated while they run, so place them after the deallocate job in the main pipeline if that is possible.

stages:

test

hil_test_job:

stage: test

tags: $DEVICE_SELECTED_ID

                script: echo "Running hardware-in-the-loop test. Look at the above job startup information to learn which runner gateway was selected from the device cloud."

non_hil_job:

stage: test

script: echo "Running a job that doesn't need hardware access"

Step 7: Test and Verify

1. Commit and push your changes to trigger a pipeline run.

2. Monitor the pipeline execution in the GitLab CI/CD interface.

3. Verify that jobs are being allocated to the correct runners and that hardware-in-the-loop tests are executing as expected by digging into the job logs.

Conclusion

By following these steps, you've implemented the GitLab Device Cloud Component for managing hardware-in-the-loop testing in your embedded development workflow. This setup allows for better utilization of hardware resources, global device sharing, and improved CI/CD processes for embedded projects.

Additional Resources

Device Cloud Component Documentation

Embedded DevOps Workshop: Refactoring to GitLab CI and Modern Security and Compliance

Remember to regularly update your runners and the Device Cloud Component to benefit from the latest features and security improvements.