Initially, we and our customers thought of the product as a series of one-off activities. As we gained customers, both their needs and ours changed. We needed a way to scale our efforts. But, first we needed to understand all of the use cases in order to figure out how to set priorities among the options for modifying widgets. I put together this matrix to structure those use cases, and to help facilitate our system architecture discussion.

Proof of concept

After we nailed down our initial system components, we needed a test. One widget was buried in the middle of a very busy page. We decided to run a proof of concept to find the best font size and font weight.

Variations tested

Results

The algorithm settled on 15px/normal as the best font size and weight. We also saw a trend line that looked promising, and possibly repeatable.

We didn’t notice when we started the test, but the customer embedded this widget inside of a fixed-height iframe, so the system’s ability to work optimize in the face of unknown constraints was extremely exciting to us. One of our main goals was to save us the manual effort of chasing down each widget’s configuration to set up a decent test. So we decided to go ahead and build the full system.

Research interview protocol

While the engineers built out the system, I started working on gathering more info from our customers. We were going to need a way to allow them some level of input in order to make the optimization align with their business goals, and communicate the system’s behavior to them (in a visualization that could scale up to millions or more variations).