A demo Qeexo has shown at various trade shows that always gets a great response is what we call our Smart Shipping demo that uses a classifier trained in Qeexo AutoML running on a STMicro STWINKT1B device that is able to detect and display what is being done to the box in real time (whether it is falling, stable, being shaken, etc.). A video of the demo can be watched on YouTube here.
This post will explain how the demo is built using “stock” AutoML functionality and didn’t require us at Qeexo to do any internal application changes to support it, so you would be able to implement something like it yourself using our current version available at Qeexo AutoML.
Making the Smart Shipping Demo
Creating this demo requires three steps. The first step is to collect the data. The second step is to train the classifier using AutoML. The third step is to integrate a frontend display (HTML/javascript in this example).
Collecting the Data and Training the Classifier
To collect data and train the classifier we can use built in AutoML functionality to handle data collection. In this case we use Bluetooth data collection application but depending on the application it can also just use the web based data collection application.
For some examples of this we have some excellent videos on YouTube that can be referenced:
Integrating the Frontend Display
Once we have a classifier trained and tested on the AutoML web application, we are ready to integrate it to our frontend application.
By default, AutoML also produces a flashable embedded classifier that will output the results of classification via serial/text. The output (in both Bluetooth and direct USB serial) will look like the following:
PRED: 0.06, 0.22, 0.72 2Where the comma separated list of values are the output probabilities for each class and the final integer number is the selected class. So in this example since class 2 had an output probability of 0.72 the classifier outputs class 2 as its prediction.
A regular expression that will match on the prediction output lines is:
^PRED:[ ,0-9.]*[0-9]$The Smart shipping demo you saw uses the web browser’s built in Bluetooth functionality to connect to the device, and then uses JavaScript and HTML to parse the output and display the proper graphics. The code for this is attached but below I’ll highlight some relevant sections:
Bluetooth Connecting
The bluetooth code we’ve included should be fine to re-use for your application as we are just using standard Google Chrome Bluetooth Device API. The example section that will set up bluetooth connectivity is:
function connectDeviceAndCacheCharacteristics() {
if (bluetoothDevice.gatt.connected && classValueCharacteristic) {
return Promise.resolve();
}
console.log('Connecting to GATT Server...'); // eslint-disable-line no-console
return bluetoothDevice.gatt.connect()
.then(server => {
console.log('Getting Service...'); // eslint-disable-line no-console
return server.getPrimaryService(service_uuid);
})
.then(service => {
console.log('Getting Characteristic...'); // eslint-disable-line no-console
return service.getCharacteristic(characteristic_uuid);
})
.then(characteristic => {
classValueCharacteristic = characteristic;
classValueCharacteristic.addEventListener('characteristicvaluechanged', handleCharacteristicValueChanged);
document.getElementById("pair_button").style.display = "none";
classValueCharacteristic.startNotifications()
.then(_ => {
console.log('> Notifications started'); // eslint-disable-line no-console
})
.catch(error => {
console.log(`StartNotifError: ${ error}`); // eslint-disable-line no-console
});
})
.catch(error => {
console.log(`ConnectError: ${ error}`); // eslint-disable-line no-console
});
}Once we connect to the device we set up our own parsing code to trigger on the “characteristicvaluechangedevent” here:
classValueCharacteristic.addEventListener('characteristicvaluechanged', handleCharacteristicValueChanged);This handleCharacteristicValueChanged function is where we parse the output probabilities and the predicted class. In this function we match on the regex above and then parse the string to determine the predicted class. Once we have this predicted class we update the graphics on the page to display the right image. Inside we grab the line and parse it to get the classification value.
var label = null;
console.log('decoded stuff...');
if (line.endsWith('\0')) {
line = line.slice(0, -1);
}
if (line.match('^PRED:[ ,0-9.]*[0-9]$')) {
console.log('parsing classification...');
console.log(line);
let predictionline = line.replace('PRED: ', '');
let predictionfields = predictionline.split(',');
for (let i = 0; i < predictionfields.length; i++) {
predictionfields[i] = predictionfields[i].trim();
}
let predictionclass = predictionfields[0];
console.log("class num: " + predictionclass);
label = classlabel2str[parseInt(predictionclass)];
let dom_elem = document.getElementById('label');
console.log(`Class label: ${ label}`); // eslint-disable-line no-console
changeDisplayLabel(dom_elem, label);
} else if (line.match('^SCORE: -?[ ,0-9.]*[0-9]$')) {
console.log('Score line, ignoring...');
}Once we’ve done this, we can update our display however we want based on the classification predictions coming in.
Wrapping it Up
Doing the above with your own AutoML libraries and frontend applications will allow you to integrate and web or native application with the output of the classifier. To make that easier, the code for the above demo is linked below.
To discover what you can achieve with Qeexo AutoML register at Qeexo.com. For any questions or comments, or help setting up your own application based on Qeexo AutoML don’t hesitate to reach out and contact us here.