It's been about a month since I began working as an OPW (Outreach Project for Women) intern on the Yocto project and the first thing I'd like to do is thank everybody for being so helpful ever since I applied. I'd like to thank Beth for spending hours on the phone teaching me git and bitbake lore. I'd like to thank Saul for meeting up with me in Boston and taking me on a poky and bitbake tour. I'd like to thank everybody on the IRC channel for their helpful hints when I was mostly clueless. And I'd like to thank Marina (from OPW) for her help during the application process and efforts to get near-Boston residents in open-source to get to know each other better.
It seems like just yesterday that I had my first official conference with Beth and yet it seems like a whole lot has happened since then. I've been collaborating with Andrei on a security layer and in order to limber up with a fairly serious contribution I've been working on a package for Snort. Snort is an IDS (Intrusion Detection System) and IPS (Intrusion Prevention System). It has a front-end, a middle-end, and a back-end, just like a compiler. In Snort's case, the front-end tackles many tasks, from reassembling packets to detecting port scans. The middle-end is the detection engine. It recognizes known exploits via their signatures. The back-end outputs a report of the events Snort has detected in various formats. All of these steps are highly dynamically configurable.
My first week started out a little slowly because the root filesystem failed to build properly, something it hadn't done during the application process. This turned to be due to some recipes that were failing to build properly, probably because some process had been abruptly stopped leaving their respective work directories in a bad state. Unfortunately, I misinterpreted the situation, believed that the root filesystem was being built and was very frustrated when I could not run the security executables on booting the image. The big lesson here, which has been reinforced in subsequent weeks is that it's a good idea to clean and re-bake suspicious recipes, and maybe even the recipes they depend on, before trying to track down apparent bugs.
Putting together a package for Snort started out easy but got progressively more difficult. Beth had already guided me through the recipe basics during the application process, so I put together basic recipes that successfully downloaded and checksummed the upstream sources without any real trouble. Next came the autotools stage and the patching of the configure scripts and Makefiles. I spent a lot of time in the autotools manuals for this one; while I've used autotools in the past and I understand the general principles patching configure scripts is new.
Once that was completed my next goal was to make sure that Snort was actually doing something on the guest machine. Pretty soon I could run it from the command line; but, of course, the real goal is to be certain that it's running correctly. It was hard for me to understand what running correctly meant in the case of Snort, so I spent some time reading "Intrusion Detection Systems with Snort" by Rafeeq Ur Rehman to get some kind of idea of how Snort is supposed to be used. Armed with this knowledge I resumed the task of setting up Snort so that it could run as a daemon. The Snort distribution includes an rpm subdirectory which contains several useful files, including an initialization script which I was able to use. The bulk of the rest of the work was Snort specific and involved much more reading of the Snort manual, examining the installation of Snort on my host machine, scrutinizing many example recipes, and applying numerous patches to the initialization script and configuration files. At this point Snort can be started as a daemon and can work together with Barnyard2, an ancillary utility that interprets Snort's output. I also put together a recipe for tcpdump, which will help investigate and verify the packets that Snort is inspecting. The next step is verifying that Snort is actually detecting and logging suspicious packets as it is supposed to do.
Some essential parts of my workflow and observations are the following:
- I got into the habit of simply searching the Yocto mega-manual for keywords or variables. This was often helpful, but it was very hard to know what manual I was in when I found a match to my search string, so that I sometimes lacked context.
- I used the find command a lot, especially combined with grep to, for example, find all the recipe files that inherited from a specific class.
- I struggled with bitbake at first because I had not grasped that the tasks available are dependent on the recipe, that there was a way to list all available tasks (-c listtasks), and that there was a way to get a dump of all environment variables and how they are expanded (-e).
- It took me a while to understand the structure of the work directory, and which subdirectory contained the results of which task.
- I asked questions on the IRC channel when I got really stuck or, recently, just curious.
- I used quilt to generate my patches.
- I avoided writing patches to eliminate warnings due to deprecated features as I expect the upstream maintainers will deal with those problems when they become errors.
- There is no way I can keep up with the volume of posts on the various mailing lists. I just scan the subjects, read anything that looks relevant to what I'm doing, and archive them all.
- The Yocto development process is much more elaborate than the processes that I've been involved with previously as a research assistant. I'm still working on fathoming how it all goes together.