This page contains the raw transcript as generated by WebEx. It has been formatted for clarity and to fit the page, but otherwise is as WebEx generated it.

Virtual Workshop on Multiproject CI/CD (2024-12-6 13:03 GMT-5) - Transcript

Attendees

• Francisco Lozano
• anujgtm
• David Swenson
• Ryan Mulhall
• Ryan Richard

Transcript

Ryan Richard: So I just started the recording. So while while a lot of people did sign up, nobody wanted to give a talk, so I the meeting will probably be a lot shorter than what was originally planned, so I do still wanna go over the best.

Ryan Richard: Best practices for multi project CICD and that’ll have a little bit of background in it as far as what I’m talking about when I say multi project CICD and why I’m interested in that. And then afterwards, I, I’d like to have an open discussion, answer any questions you have, but also try and.

Ryan Richard: Better understand what uses people have for multi project CICD. So with that, does anyone have any questions while I’m setting up the share and so yeah, feel free to come off mute and.

Ryan Richard: Ask questions throughout the talk or put them in the chat. I may not always see the chat, the Webex windows on a different screen. So my name’s Ryan Richard. I’m organizing these virtual workshops on multi project CICD. I’m a scientist.

Ryan Richard: The Names national lab. And I’m doing this as part of the BSS BSSSW fellowship, which is the better Scientific Software fellowship. If you’re not familiar with it, that’s run by an organization whose website is bSSW.io.

Ryan Richard: There’s a lot of useful resources there and these workshops are ultimately hope putting together resources so that I can put additional resources on the bSSW.IO website related to multi project CICD.

Ryan Richard: So, that is my slide. There we go. So a lot of this work that I’m going to talk about was motivated by my role as the lead developer of. I’ll talk a little bit about what NW Camex is, but for the purposes of this talk, it’s the architecture that’s more.

Ryan Richard: Important than what it actually does. So we were, we’re a very small team at the moment, about five or six of us, and so we were looking for ways to make our CICD more efficient in terms of reducing the amount of maintenance we have to do on it.

Ryan Richard: There was plenty of work out there for what I’ll call single project CICD, but that was very limited or with the very limited resources for multi project. So I’m not really gonna go into best practices for single project CICD. This is really focused on how do you scale it across projects.

Ryan Richard: And as I mentioned, I, I was awarded the 2024 BSSW fellowship for this work and that’s what I’m going to be talking about. So very high level what is in WCAM X and WCAM X is a computational chemistry software designed to simulate chemistry. What matters for the purposes of this talk is we decided to write it as a series of plugins. So we have essentially a.

Ryan Richard: Software developer kit, which in this graph is represented by the sim, the box labeled sim D That’s the common library that everybody uses when they go to write their plugin, and then everybody writes a plugin based off that and that their plugin lives in its own library. Typically it’s in its own GitHub repo.

Ryan Richard: There’s nothing restricting it to being on GitHub, it could be on GitLab or hosted locally. But the important point here is every plugin ends up being its own get repository. And the reason behind this is we wanted to have the plugins be largely standalone reusable library.

Ryan Richard: Just help integrate them into other software in the ecosystem besides just a WKMX. And I think I’ve heard from a number of people, this is a sort of a paradigm that’s appearing in other areas of science. Oh, there’s a lot more focus on modular software. So I suspect there are other people having the.

Ryan Richard: These sorts of problems and that’s what these workshops were, really designed at getting at is how do we manage the CICD for huge ecosystems of related libraries?

Ryan Richard: So, why did we want a modular ecosystem? So same reasons you, you have probably heard about why modularity’s good. We wanted to encapsulate, we have encapsulation and separation of concerns, makes the development of a complex software system easier. I mentioned the reuse.

Ryan Richard: And we also like it because it helps us with our rapid prototyping. You can grab this is one of the reasons why a lot of people like Python. There’s a lot of Python modules out there. You can sort of pick and choose the ones you want and that gives you an excellent starting spot for writing your scientific software. The other thing we like about it in the scientific software.

Ryan Richard: Where is it helps the contributions be better attributed. So when you, in particular in competitional chemistry, we had this long history of these monolithic packages, you’ll have 4000 authors, four thousands of exaggeration.

Ryan Richard: Easily a couple hundred though, and so your your work gets lost in the offer list, but with small individual reusable modules, it’s better to get it’s easier to get. So what the problem we started to run into real quickly is we had, there’s a lot of infrastructure.

Ryan Richard: Structure and utilities to support these repos, and that ends up looking very similar in every one of these repos. So each one of the repos has its own build system, the Nwks is a C plus is largely C++ it has some Python, and so you’re gonna end up having a lot of cmake files.

Ryan Richard: That get duplicated across repos, a lot of formatting for the C++ etc. Furthermore, these tend to be a lot of these plugins are developed by similar groups of people and so we have this same sort of formatting standards because they are the same people, they, you know, you have a tendency to format your code the same way regardless of which.

Ryan Richard: Repo it’s in, so we have similar formatting, we have similar configurations for the docs, the documentation, and so being software engineers or sort of research software engineers, I should say, we wanted to.

Ryan Richard: To sort of streamline this and obey the dry principle if you’re not familiar with dry, it stands for don’t repeat yourself. It’s this idea that everything should have a single source of truth. This can be challenging when we’re talking about utilities and by utilities I mean basically anything that’s not the main software product, so all the software designed to support the software. That’s because a lot of times this is developed more ad hoc, you’ll you’ll have sort of one off python.

Ryan Richard: Scripts or something that’ll maybe parse your file and build your documentation or something. The scripts tend to be boilerplate heavy. It’s sometimes can be hard to remove or factor out that boilerplate, and at least within computational chemistry, we also have to overcome this long standing history of this is just how it’s been done. We copy paste from one repo to another and tweak it. And so there’s a lot of people that haven’t even been thinking about how can we apply dry to the utilities and infrastructure supporting our software.

Ryan Richard: At least again within computational chemistry, and I’m hoping to hear after the talk from other people about their experiences and their domains. So just to make sure we’re all on the same page, probably should have had this sooner. What is CICD? So CICD stands for continuous integration continuous deployment.

Ryan Richard: Roughly speaking, it sort of follows the steps shown in the diagram, if you’re familiar with GitHub actions, GitHub Actions is a common work for implementing CICD. There’s also Travis CI Jenkins. Gitlab has its own thing. I forget what it’s called. I’m not much of a.

Ryan Richard: Lab person. The the main idea is we want to automate the building, testing, et cetera, of the software, and we can do that using the online resources provided by Travis CI Jenkins GitHub, et cetera.

Ryan Richard: As I mentioned, we, the purpose of this workshop is really to focus on reusing CICD across projects. A lot of people I’ve talked to, they will say that, you know, CICD is project specific. And while that I think that’s true when you’re developing a single project, when you start developing plugins.

Ryan Richard: Or modules and they’re all targeting the same framework. There ends, this sort of ends up being, the reality is you undermine that assumption. You end up reusing a lot of infrastructure from plugin to plugin module to module. And again, we want to make sure we have a single source of source of truth because if that framework changes, e.g., how we have to build, you don’t want to go through 400 plus.

Ryan Richard: Ends and change the build system or something like that. And so I founded the multi project DevOps GitHub organization. You, if you Google multi project DevOps GitHub, you should, it should be the 1st hit. And the purpose of this organization is really to explore how to maintain multi pro.

Ryan Richard: Project ecosystems. The named DevOps is a little bit aspirational right now it’s really just focused on multi multi project CICD.

Ryan Richard: So, we went through several attempts in trying to figure out what’s the best practices for multi project CICD. So we started with what everybody told us to do, which was copy paste and tweak. It was very tedious. We very.

Ryan Richard: Often would run into problems where we forgot to include, say, a C a flag for the C compiler and you now have to update 30 repos because you forgot that flag. So we very quickly wanted to move on from that.

Ryan Richard: We tried get sub modules and sub trees, if you’ve ever tried this before in your own ecosystem, maybe it’s worked for you, but the vast majority of people I talk to say it’s very confusing, it’s hard to do and.

Ryan Richard: Error prone. Yeah, it requires a lot of developer discipline if you’re not familiar with it. You have to remember to always pull the, the sub modules et cetera. Ultimately we moved on from that because our developers were not happy with that.

Ryan Richard: So GitHub had some actions that existed. They were for auto synchronizing, so you would essentially put the files in a centralized repository, so we would have like one CMake list.text that’s sort of templated, you would have to go through and change the names.

Ryan Richard: Of the libraries et cetera, but it sort of had the rough structure and then you would use this action to automatically synchronize that file across all the repos. So this was kind so you can think of this as automating the copy paste, it was better because it was one source of truth.

Ryan Richard: What we found was the, the triggering was a little touchy. So sometimes the triggering wouldn’t occur and so the changes wouldn’t actually propagate, other times it would be constantly rewriting the files and could even get into loops where it saw that a file was.

Ryan Richard: Changed and so then it would change another file, but because that file changed, it went back. Thankfully GitHub stops those kinds of loops after a couple of iterations but yeah that was that was troublesome. We moved on to docker files after that and started version, we would version control the docker files. That was, that was ok. It ultimately led a lot of the repetition to just be in the docker file, so it didn’t completely solve the problem.

Ryan Richard: We looked into making actions for GitHub actions, so unfortunately GitHub actions is both the name of the CICD framework and also a name of the component in the framework, so that’s a little confusing, and so right here what I’m talking about is writing a component for that framework.

Ryan Richard: So essentially you would write a little, application that knows how to say build a code using our build system, given a couple of parameters. So we could, that worked, that was a very.

Ryan Richard: Nice solution other than writing GitHub actions is very heavy handed. That’s, there’s a lot of development that goes into writing a GitHub actions, and this just felt like overkill for trying to just modularize the CICD.

Ryan Richard: Thankfully, about the time we got to this point, GitHub apparently had also realized it was a problem and they introduced reusable workflows and composite actions. When we started down this endeavor, GitHub Actions actually did not have reusable workflows and composite actions, which is.

Ryan Richard: Why we didn’t just jump to this solution. I think that only came out in maybe the last two to three years something like that. There is a similar feature in GitLab and ultimately reusable workflows composite actions is what we decided to go with, and that’s really what a lot.

Ryan Richard: Of the best practices I’m gonna talk about are based off of and that’s an example of a composite action there yeah so there’s a little bit of background, so what makes multi project CICD different than CICD? So again, it’s sort of this idea of reuse. So on the left side of the figure, you see that there’s sort of two pipelines. There’s one for package one, one.

Ryan Richard: For package two, they have sort of some similar components in there that they, e.g. both do a configure build and install s stage, but ultimately they’re maintained as two separate scripts. In the multi project view, we view it as a reusable pipeline that we feed different packages through. So it’s a fundamentally different way of thinking about it, trying to treat the pipeline as a single source of truth.

Ryan Richard: One of the 1st one of the big differences in our experience between multi projects and single projects CICD is the fact that since you now have a bunch of projects relying on a single pipeline, if that pipeline breaks, you don’t just break the one repo that the pipeline’s associated with you.

Ryan Richard: Break the ecosystem that that pipeline’s associated with. There’s also security concerns. We’re an open source package. We’re not as concerned with security as a lot of other packages are. That’s a problem. I don’t know the answer to, and I’m just going to mention it does exist any time you’re sharing ecos anytime you’re sharing software across an ecosystem, there are security concerns, but that’s not something we attempted to solve here.

Ryan Richard: So, after a bunch of trial and error and looking on the internet and seeing what other people have done, we sort of came up with these four best practices for multi project cicds. The 1st one is dry. Don’t repeat yourself. Just like all other software, you really.

Ryan Richard: Really should be striving to follow good software engineering principles, and not repeating yourself, i. E. Having a single source of truth is one of the key principles. You also should be treating your infrastructure as codes. If you’re not familiar with this concept, it’s the idea that you should be using configuration files.

Ryan Richard: Or scripts to set up the environment rather than GUIs. This allows you to version control your steps and to ensure that you’re getting a consistent development environment in the CICD pipelines. There’s this other idea of treat code as codes, so I made up this term, I don’t think anyone else has used this, but when I talked to software engineers, they didn’t have a catchy little terminology for this best practice. They said this is really sort of a.

Ryan Richard: It’s obvious to them that you should be doing this, but in my experience, this is not obvious in the scientific computing circles. And so what does code does code mean? It means that you need to treat all of the code using best practices, not just the main software that is your.

Ryan Richard: Your package. So in our case of nwcax, e.g., we were of course using best practices for the main source code of nwcax, but what code is code refers to is you need to also be using the best practices for all your scripts that support it, your utilities.

Ryan Richard: Those sorts of things.

Ryan Richard: And I alluded to it on the last slide. The other main multi best practice here is testing. In my experience a lot of people don’t have formal tests in place for CICD, often it’s you write a CICD workflow and you trigger it, see if it works, if it did, great, if it didn’t, then.

Ryan Richard: You go back and you iterate until it works. Testing becomes a lot more important. Again, you’re gonna have an e ecosystem depending on these things, you can’t just be pushing changes to the workflows and hope that they work because you may end up bricking an entire ecosystem. What’s good about this is since you we’ve factored out the CICD into one place, and I’ll go into that on the next slide. It’s easier to test it.

Ryan Richard: Because it’s only in one repo. We don’t have to worry about testing multiple repos simultaneously. So as far as GitHub specific best practices, which is what I’ve really been focusing on, you sort of have these two things, you have composite actions, which you can think of a composite action.

Ryan Richard: And as reusing a task, what a task constitutes is really up to the developer, you could have a task B as something as fine grained as say just running a configuration script for like a C++ build or it could be the entire configure build install step.

Ryan Richard: Really up to you. And the other thing you have are reusable workflows, which reusable workflows are sort of drivers wrapped around the actions. So it’s a series of actions run together, and you’re reusing that series of commands.

Ryan Richard: It’s suggested that you use one workflow per trigger condition, so you have a workflow for on pull requests, so when someone makes a pull request, you have one workflow for when someone pushes to your main or master branch, et cetera. Don’t try to reuse the workflow.

Ryan Richard: Goes across push across trigger conditions. More often than not, there’s little differences. It’s better to modularize the internals of the workflow than to try and put a bunch of if else logic through the workflow to make it so that it works under both trigger conditions.

Ryan Richard: It’s recommended you consider dedicated repositories for your composite actions. This is because GitHub tracks versioning on a per repo basis. In particular.

Ryan Richard: You can or an alternative is to store them in the.github repo and just be very meticulous with your commits. And what I mean by this is if you only modify one composite action per commit, then your get versions will still essentially map to a single action. The problem is if.

Ryan Richard: You change multiple actions in one commit, then there’s not really a good way to distinct distinguish whether you want to use say version one of action one and version two of action two since they both are tied to the same commit.

Ryan Richard: Reusable workflows are meant to be used across an organization usually, so they are recommended that you put them in a single.github repo. Composite actions are usually thought of as being more general, more.

Ryan Richard: Useful outside the organization, that’s why they were suggested as their own repos. If you have a lot of them though, yeah, it can be, there can be a trade off to that. And you should always be referring to the actions reusable workflows by version. So multi, so those best practices are a summary of the resources I’ve put together so far and the results of previous workshops et cetera.

Ryan Richard: You can find the work on my endeavor into trying to figure out best practices for this on the multi project dev OPs website. There’s some tutorials and resources there. There’s the best practices and more importantly, if you really wanted, well, I don’t know if it’s more important, but there’s also the extra.

Ryan Richard: Donations for how we came to those best practices. So they, most of them didn’t just come out of thin air, there’s actually logic behind why that’s there. And I’ll also put do a shout out to one of my colleagues. He put together a nice YouTube video explaining.

Ryan Richard: How we came to our sort of multi project devops, or multi project CICD workflow, and that that video the link is available through the multi project website. So with that, sort of gonna summarize here some modularity.

Ryan Richard: In my opinion really complicates the CICD workflows, if you want to apply dry, don’t repeat yourself to CICD, then there’s additional considerations. Github has made it a lot easier than when I started this endeavor. So when I started this endeavor.

Ryan Richard: That wasn’t as easy as it is now, thanks to github’s reusable workflows, and composite actions. Reusable workflows are again sort of a driver and composite actions are really geared at reusable tasks. And I’m.

Ryan Richard: Looking for collaborators to really flesh out more resources, maybe come up with some recipes for examples of composite actions or those kinds of things for the website or if somebody was interested in, e.g., repeating these processes for getlab so we could reach a better group of the science.

Ryan Richard: Scientific or a larger group of the scientific software community. And with that, I acknowledge the people who funded this and helped work and I’m happy to answer any questions you might have.

Ryan Richard: So I’m going to take the silence to be NO questions. So then let me ask, I guess.

Ryan Richard: The people that are here, is there anyone that is a CICD maintainer and is facing similar problems?

Ryan Richard: Oh, you’re muted.

David Swenson: There we go. I I sort of am. So I work for an organization where I’m sort of the infrastructure lead for, to help coordinate the infrastructure leads on other projects.

David Swenson: And we have several projects each, each of which is kind of an ecosystem, much like what you’re talking about, where they may even have different kinds of plugins, very different plugins involved, and so in that sense, I’m I’m very interested in this. I don’t think we’ve had as much of a pain point on it yet.

David Swenson: And part of that is we’re very Python focused. And so a lot of the the you know compilation issues, e.g., just aren’t a thing for us and and making sure that’s the same install or condo install is just very easy, but I’ve always been a little bit bothered in principle that we have different.

David Swenson: CI workflows and when something happens like we follow the best practices around supporting versions of old versions of python. And so we move those on and and stop supporting old versions when that needs to happen. But that means that that has to happen through every project in the ecosystem and it takes a while for that to propagate cause people forget.

David Swenson: About some of the projects that we have with that we’re still supporting. So yeah, we do that. We have we had some trouble with that, but it’s not, it hasn’t been enough of a pain point for us to need to really look at the, at the sort of centralized solution yet.

Ryan Richard: Alright, thanks for sharing. Anyone else? So I I guess I can add that, yeah, what you just said is very consistent.

Ryan Richard: With some of the other things I’ve heard from people, that are working in Python ecosystems. They, they, they feel that, yeah. So I don’t know if this is really just a comp yeah a compiled ecosystem thing because the other people that seem to be having this problem are four train developers, which again I think it’s gonna be this for the same reasons the the build system just.

Ryan Richard: Makes things a lot more complicated. I’ve also heard from a number of people that say when they switched over to Docker, these problems went away that the containerization just made the made it easier. So yeah, maybe these are just maybe this is a problem for the DOE software stack, so I’m department.

Ryan Richard: Bit of energy, I should find my abbreviation there because we do have a lot of compiled software because we’re targeting the exoscale machines and HPC in general. So yeah, what you said is very consistent with that. So my other questions then I guess are Is there other considerations that we I should be thinking about in terms of putting best practices together? Am I overlooking something?

David Swenson: So I don’t know if I see overlooking, but there’s a another perspective on this that is something I’m aware of it’s not it’s adjacent to my community, which is tooling that facilitates ensuring that plugins still work. Externally contributed plugins still work. And so especially since you’re also in the chemistry space, you might be familiar with MDN out.

Ryan Richard: Process.

David Swenson: And so they’ve recently basically they hit a point where they had too many, too many contributions coming into the core package. They didn’t want to maintain it all. They wanted to keep some of those contributions external, and to facilitate that, they created what they called MDA kits. And that includes some CICD solutions where they actually are running the, the external contributions like against their canary builds so that they are warned if something changes and so they can also warn those downstream developers that they need to.

David Swenson: Something. It’s a, it has a lot in common with what you’re talking about taking it from a very different perspective, but it might be something worth worth looking at and and considering what they’re doing with that.

Ryan Richard: Yeah, thanks for bringing that to my attention. Yeah, NO, this is definitely something our users have also expressed interest in. I hadn’t really thought about it from this perspective of how they could be connected. But yeah, NO, this is definitely a problem we face also, you know, what happens when.

Ryan Richard: One of the plugins changes and yeah, it breaks the ecosystem, it would be good to know ahead of time. So what we, for whatever, for what it’s worth, we had been thinking of trying to roll out like common CICD for nightly builds, and so it wouldn’t necessarily solve your problem, but you would know ahead of time, hey, you broke it.

David Swenson: Yeah, the, the approach that the MD analysis people did, I think it’s it’s not the one I would have taken, but it seems to they’re very happy with it so far, which is that they are the ones who are running the CICD so it’s on the core product that they get the notification that the error is occurring. And so that’s yeah to me that’s a, that’s a little bit different, but that is also, that’s useful for knowing that you might have broken something downstream. Like you don’t get that notification until the developer gets mad at you otherwise, right?

Ryan Richard: Yep, yeah. So it looks like it’s just just yeah.

David Swenson: Yeah if you start going very far down that sort of a road on that, please do get in touch with me because that’s part of my current funding is to do some, some sort of things around this sort of plugin infrastructure thing. Okay. Our focus is on the idea that.

David Swenson: We’re trying to find ways to connect the developers of plugins to the user community. And so that’s more about making a website that’s like a centralized repository for any ecosystem, to, so that, you know, users can find plugins and plugin developers can get their users but the CI CD aspect of it is something I’m really interested in as well.

Ryan Richard: Cool. What’s Do you have a website or something that I could.

David Swenson: We we don’t have anything on that and not unfortunately our website is terrible, but it’s OMSF.io, open Molecular software Foundation.

David Swenson: And yeah, our website is currently, we’re waiting for the new version of the website to finally get delivered. It’s only a year late, you know? But once that’s there because it’s the other problem is I’m not even actually listed on that website, but my email is my 1st name, last name david.smart.

Ryan Richard: So feel free to get in touch if.

Ryan Richard: Yeah, Sounds good. Yeah, I am hoping that I’m going to be moving back towards doing scientific software after this and away from the maintenance. Part of the reason I did this was because I was trying to bring awareness to the need for the software maintenance and I.

Ryan Richard: Think that’s resonated with some of the people around the lab, and so I’m hoping that I will actually get because I’m supposed to be doing quantum mechanics actually CICD. Yeah. So I’m optimistic that maybe I I will get to pivot, but I will definitely for if that does have regardless I will let people know about your interest and yeah, if we start moving in that direction, we’ll be in touch.

David Swenson: Sounds good. Yeah, especially like if cause that’s something where we’re supposed to and I mean we are the open look at software foundation. We are supposed to help with open source software and and look at the sciences across the board. It’s not just the things that we are hosted prod hosting our projects, so if if there’s something that we can do that helps you guys as well, that’s that’s very much in our mission as a non profit.

Ryan Richard: But it’s it’s to do that. So yeah. Yeah, maybe there’s actually, yeah, like a collaboration opportunity here. I don’t know, I don’t know what you usually do for funding, but DOE has NNSF have both been pretty supportive of partnering with the outside resources.

David Swenson: Yeah, right now, right now I’m on an NSF post grant. Most of our organization, we’re more on the classical micro dynamic side of things and it’s all the near bio side of things. So force field development, free energy calculations and protein structure prediction, all of which are easily funded by pharma. We get a lot that’s where most of our money comes from. We are the open source alternative to the expensive thing that they pay for.

Ryan Richard: Gotcha, yeah. Yeah, unfortunately I I’m the very low level electronic structure by training and yeah, it’s a hard time convincing them that they need a supercomputer to compute the energy of two atom systems. Ignore me.

David Swenson: Yeah, NO, that’s where I started out before I went to even less useful stuff with semi classical dynamics, which is really ripthless, but yeah so.

Ryan Richard: Alright, well it was good to meet you unless you have anything else.

David Swenson: I don’t I’ve been interested in seeing I’ve been interested seeing where it goes and and i’ll continue to keep an eye out on this so thanks for doing the work on that.

Ryan Richard: Alright, thanks for coming. Bye. Bye.