When building our A/B experimentation platform at Walmart Global Tech one of the critical features was identifying when a user qualifies for an experiment. A qualified user is defined as someone who was assigned to the experiment and who saw (as best we can tell) the part of the site being experimented on. In order to determine if a user qualified, we need to query data from our site (Walmart.com) or our native app by a set of predetermined criteria. This predetermined criteria is built during the experiment setup phase and is in the form of a query expression (similar to SQL). These expressions can get pretty complex, as anyone who is familiar with building SQL queries or scalar functions can attest. Additionally, the attributes in the expression are defined by an external system, ie. the site/native app, and therefore are subject to change outside of our application’s control.
This presents us with two challenges to address. First, we need to templatize the query expressions in a way that values can be substituted in. And second, we need these templates stored outside codebase in a configuration file.