August 17, 2016

Quality Assurance Interns Pave the Way for OpenX’s Broker Software

By: Sandra Ning, QA Intern

Over the past two months, I have been working alongside a fellow Quality Assurance intern, Andrew, to ensure the changes OpenX engineers make to broker software are working as intended.

There are a few challenges to overcome when testing the broker’s real-time bidding auction function. If we want to test new changes to the broker, we must do it without disrupting real-time ad exchanges. That means the code needs to be tested separately from auctions happening in real-time. Furthermore, the broker relies on client data in order to run an auction. If we can’t run real-time auctions to get this data, how can the broker perform its calculations. 


We can circumvent this issue by simulating an auction instance. Old data is fed from previous clients to the broker. This simulation uses broker code to calculate the value of clearing prices and price floors. These numbers are compared against the actual figures from the historical data. If they differ significantly, then the code changes may have disrupted important broker operations like currency conversion or clearing price calculation. This helps engineers identify potential problems in the code early on, which is exactly what testing and quality assurance is all about!

OpenX engineers push changes to the broker development repository several times a day. Therefore a simulation must be run every time a change is made to catch the most errors. Manually running the package after every change is tiresome. Our goal was to develop a precise automated solution. If the system could update the broker, run an auction with old user data, check to see if the broker returned expected values or not, and do it all automatically, testing becomes frequent and easy.

Between Andrew and I, our job was to create this automatic system using GoCD, a continuous delivery server specifically designed to automate workflows like the the process needed to compile and run broker. On GoCD, related tasks that are executed in sequence are called pipelines. Our pipelines needed to begin executing whenever a change is made to the broker code, and:

  1. Build the new version of the broker
  2. Use the new broker version to compile the new simulation package
  3. Run the simulation using data with known input and output
  4. Allow the pipeline to succeed if the broker returns the expected values, or fails if it doesn’t

Without needing to build the architecture for automating these tasks, it seemed simple to build the pipeline. But like any project, various problems cropped up for every step of progress. The first problem is one any intern, or new employee, experiences: overcoming the learning curve. OpenX uses a lot of internal tools to maintain their code and data. Andrew and I ran Hive queries to grab old user data, and used Maven to build the simulation package. We branched, pulled and pushed code from Github and learned how to debug in Eclipse. All of this was in addition to figuring out how to fit the materials, dependencies, tasks and artifacts on GoCD together to make a working pipeline. The auction simulation code was already written when we started working on the project. Problem two was dusting off the code, which hadn’t been updated in months, and editing it to mesh seamlessly with the latest version of broker. Problem three was the trickiest of all—debugging the simulation.

Although the simulation code was written by OpenX’s data science team, it hadn’t been kept up-to-date with the broker’s API. Initial runs of the simulation resulted in errors that made testing impossible. Debugging the problem was tough. The simulation code depends on data files that change daily, and thus must all be from the same date and time for the broker to output correct values. Before making any attempt to comb through the code, we double and triple-checked that these data files were correct, and pulled more files from Hive if they weren’t.

Bugs cropped up from data types not being initialized correctly and from updating the code to fit the new broker API. In the end, Andrew and I were able to completely set up the pipeline to automate the simulation on GoCD. I learned a lot—not just about internal tools or how OpenX’s broker works, but also about asking for help from those around me, and tackling problems from various angles until something gives. I also created something that OpenX engineers will—hopefully—use, long after Andrew and I leave.