Our approach is to predict Wi-Fi loss 15 seconds ahead of time based on the past 60 seconds. To get the best possible result, we evaluated various ML algorithms, feature vectors and training sets. An overview will be given here.
Feature Vectors
We evaluated three different feature vectors. The first one is the full feature vector, as can be seen below:
- Pressure (delta)
- Absolute Pressure
- Linear Acceleration (all axes separately)
- Linear Acceleration (all three axes together)
- Step Counter
- Power Data (if the phone is charging or not)
- Remaining Battery Capacity
- Gravity (all axes)
- Gyroscope
- Magnetic Field (all axes)
- Absolute Phone Orientation
- Phone Rotation (all axes)
- Wi-Fi Freuency (2.4 GHz or 5 GHz)
- Wi-Fi Speed (IEEE 802.11b, IEEE 802.11n, …)
- Wi-Fi RSSI
Additionally, we used a reduced feature vector, containing the following sensors:
- Pressure (delta)
- Linear Acceleration (all three axes together)
- Step Counter
- Power Data (if the phone is charging or not)
- Gravity (Z-axis)
- Wi-Fi Freuency (2.4 GHz or 5 GHz)
- Wi-Fi Speed (IEEE 802.11b, IEEE 802.11n, …)
- Wi-Fi RSSI
Finally, we also used just the Wi-Fi RSSI for comparison.
Algorithms
Besides using different feature vectors, we also used different ML algorithms and different configurations.
| Algorithm | Configuration |
|---|---|
| Radom Forest | 10 Trees 20 Trees 25 Trees |
| Multi-layer Perceptron Classifier | 1 Layer, 100 Neurons (NN1) 3 Layers, 100, 200, 300 Neurons (NN2) 5 Layers, 400 Neurons each (NN3) |
Training and Test Sets
Data from 5 volunteer testers were collected for this work. Thus, we tried two different Training and Test sets. First, all data was combined and than split randomly in 70:30 relation with 70% for training the Neural Network and 30% for testing. In the second approach the data was split per user, resulting in a model that was trained on a specific user.
Results
In this section the raw results are presented. First we look at the Random Data Split to validate the effectiveness of our approach. A baseline evaluation using a Random Forest is presented, comparing an RSSI-only approach to the Reduced and Full Feature Vectors.
Random Data Split
Random Forest
RSSI-only Feature Vector
10 Trees
precision recall f1-score support
loss 0.90 0.97 0.93 49260
no loss 1.00 0.99 0.99 457966
avg / total 0.99 0.99 0.99 507226
Reduced Feature Vector
10 Trees
precision recall f1-score support
loss 0.88 0.98 0.93 42334
no loss 1.00 0.98 0.99 345799
avg / total 0.98 0.98 0.98 388133
Full Feature Vector
10 Trees
precision recall f1-score support
loss 0.89 0.99 0.94 42107
no loss 1.00 0.98 0.99 343073
avg / total 0.99 0.99 0.99 385180
Neural Network
Reduced Feature Vector
NN1
precision recall f1-score support
loss 0.96 0.93 0.95 51098
no loss 0.99 1.00 0.99 426178
avg / total 0.99 0.99 0.99 477276
NN2
precision recall f1-score support
loss 0.98 0.96 0.97 51098
no loss 1.00 1.00 1.00 426178
avg / total 0.99 0.99 0.99 477276
NN3
precision recall f1-score support
loss 0.98 0.95 0.97 51098
no loss 0.99 1.00 1.00 426178
avg / total 0.99 0.99 0.99 477276
Full Feature Vector
NN1
precision recall f1-score support
loss 0.96 0.95 0.96 50748
no loss 0.99 1.00 0.99 422865
avg / total 0.99 0.99 0.99 473613
NN2
precision recall f1-score support
loss 0.98 0.97 0.98 50748
no loss 1.00 1.00 1.00 422865
avg / total 0.99 0.99 0.99 473613
NN3
precision recall f1-score support
loss 0.99 0.96 0.98 50748
no loss 1.00 1.00 1.00 422865
avg / total 0.99 0.99 0.99 473613
User-based Data Split
To prove how well the trained model can generalize across users, the data is split up based on users, where the user evaluated with is not used for comparison.
Neural Network
RSSI-only Feature Vector
Training rssi_user1_nn-3...
precision recall f1-score support
0.0 0.98 0.97 0.97 25982
1.0 1.00 1.00 1.00 653387
avg / total 1.00 1.00 1.00 679369
Training rssi_user2_nn-3...
precision recall f1-score support
0.0 0.93 0.75 0.83 26192
1.0 0.98 1.00 0.99 354533
avg / total 0.98 0.98 0.98 380725
Training rssi_user3_nn-3...
precision recall f1-score support
0.0 0.97 0.88 0.92 124228
1.0 0.98 1.00 0.99 764701
avg / total 0.98 0.98 0.98 888929
Reduced Feature Vector
Training reduced_user1_nn-3...
precision recall f1-score support
0.0 0.92 0.96 0.94 25772
1.0 1.00 1.00 1.00 439487
avg / total 0.99 0.99 0.99 465259
Training reduced_user2_nn-3...
precision recall f1-score support
0.0 0.79 0.73 0.76 23801
1.0 0.98 0.98 0.98 306163
avg / total 0.97 0.97 0.97 329964
Training reduced_user3_nn-3...
precision recall f1-score support
0.0 0.93 0.89 0.91 104565
1.0 0.98 0.99 0.99 646044
avg / total 0.98 0.98 0.98 750609
Full Feature Vector
Training full_user1_nn-3...
precision recall f1-score support
0.0 0.91 0.69 0.78 25772
1.0 0.98 1.00 0.99 439487
avg / total 0.98 0.98 0.98 465259
Training full_user2_nn-3...
precision recall f1-score support
0.0 0.72 0.54 0.62 23492
1.0 0.96 0.98 0.97 302383
avg / total 0.95 0.95 0.95 325875
Training full_user3_nn-3...
precision recall f1-score support
0.0 0.68 0.69 0.68 103850
1.0 0.95 0.95 0.95 638116
avg / total 0.91 0.91 0.91 741966
For more data on the evaluated models, please look in the available notebooks. Learned models are also available there.
Notebooks
Alls Jupyter Notebooks can be downloaded from our GitHub repo. We provide a Dockerfile so that you can start the Docker container and see all steps done from preparing the data until the learning evaluation directly in a Jupyter Lab. Just check out the mentioned repo, build the container and start it. Please provide the right parameters in the commands below:
docker build .
docker run -p=8888:8888 -v <PATH_TO_SQLITE_TABLES>:/jupyter <CONTAINER_ID>
The output of the container will contain a URL (similar to http://0.0.0.0:8888/?token=9b7e8f0aaec254f72bbed67074deae7bab1ceb6e8473e917). Replace 0.0.0.0 with localhost or 127.0.0.1 and open the URL in your browser. That’ it.