Taking Machine Learning from Concept to Product

Executive Summary

• A pharmaceutical applications company wanted a way to easily verify that the right pharmaceutical product was in the hands of the right patient. 
• The idea was to have a proprietary barcode printed on the package that would then be scanned by the customer. Originally, the company doubted that the idea was even technologically feasible. 
• Using machine learning, Big Nerd Ranch was able to create an app that used the customers’ device to scan the barcode to ensure the medication was sent to the correct person. 
• The outcomes include the proven viability of the app and the subsequent release to the market.

Challenge

Now that prescriptions can be ordered online and shipped directly to consumers, it’s more important than ever to ensure that the medicine they receive is correct. A leading pharmaceutical applications company came to Big Nerd Ranch with an idea to use technology to verify the authenticity of a pharmaceutical product and give the patient the power to confirm that the medicine they received is genuine. This would also guard against counterfeiting. The main challenge was a biggie—was the idea even technologically possible?

Big Nerd Ranch Solution

Short answer: it’s certainly possible for the Nerds! Through a rigorous and in-depth process, we helped this client prove the viability of the solution and shape the future of their product. 

The solution revolves around a specially designed barcode that includes proprietary dots appearing in random around the QR code. The barcode pattern is recorded for each package at the factory before being sent to the customer who then verifies it on their end using their device. 

The technical challenge for Big Nerd Ranch was recognizing the dots and their locations around the barcode. The major obstacles from the start included: 

• The need to have the customer’s device recognize the barcode and dot pattern 
• A recognition solution that was effective and quick since users will have less than ideal lighting conditions, won’t hold their phones still and often at different orientations 
• Tiny dots—the QR code is 7.62 mm wide (⅓ of an inch) and the dots are the size of a small piece of glitter and are measured in Microns 

Because of the complexity of the project, we took a three-phase approach. 

Phase 1

We began by developing a machine learning algorithm that could detect the dots in each image. This involved several steps:

• We started by creating training data. We wrote software to generate sample images and manually drew bounding boxes around stars and classified the data 
• Then we used TensorFlow to build and train the object detection model on that sample data 
• Once we had a model that was working well at identifying the dots, we moved the model to the CoreML machine learning library and created a simple iOS app that allowed the model to run on an iPhone

Phase 2

We then began to determine the size and location of the dots that were identified by the model from the 1st phase. 

• We did a few things to improve accuracy, such as normalizing image skew and removing artifacts from images. 
• We had to be very accurate with the locations. For each dot location, we only had a margin of error of 150 microns, and being off by just 1 pixel is 50 microns of difference. For context, a human hair is 70 microns wide. 
• In the end, we were within that tolerance and our measurements were within two human hairs of location accuracy. 
• And keep in mind, the user is not asked to hold their phone completely still. This works as you would expect from any other barcode scanning app.

Phase 3

Finally, we worked to determine the properties of the dots. This is a critical function of the security of this product. Without this, you could just use a photocopy of a valid label to bypass the detection mechanism. During this process, our guidance heavily shaped the company’s approach. 

• We started with a single image and just identifying a raw color value for each dot—this was not enough to verify that there are certain properties as the company’s proprietary dots change colors in different lighting conditions. 
• We suggested a phased image approach which resulted in taking 20 pictures and comparing the color data across the images. Additionally, we incorporated color correction and used known white areas around the QR code to set our white balance. 
• We also experimented with some techniques such as using the tilt sensor to ensure that images are captured at different angles and turning the flash on during the process to improve the lighting. 
• We ended up with a greater than 90% success rate in identifying the dots. We also suggested changes to the manufacturing process from our learnings to improve accuracy and the company is making some of those changes now.

Solution

As the project was somewhat nebulous from the start it lacked a well-defined path—especially considering the client wasn’t even sure if the product was technologically possible. As a result, it enabled us to learn along with our client and produce some exciting and productive results.

One of the biggest results to come from this project is that we were able to prove the viability of the product. We were able to incorporate machine learning to solve a problem and define a solution that makes counterfeiting virtually impossible. This is especially critical in the pharma industry when counterfeit goods can be life-threatening. 

As a result of their partnership with Big Nerd Ranch the product is now viable to the market. We are continuing to work with them as they move this forward in production.