I have come up with some official metrics on how the various moodilyzer schemes are doing, using the f1 score. This is a pretty widely used thing, and once again scikit makes this really easy, so I went with it. Here are the results:
| mood | LSTM | ELMo+LR | ELMo+MLP |
| grateful | 0.13 | 0.19 | – |
| happy | – | – | 0.33 |
| hopeful | 0.19 | 0.20 | 0.29 |
| determined | 0.04 | 0.32 | 0.51 |
| aware | 0.17 | 0.34 | – |
| stable | 0.08 | 0.08 | 0.38 |
| frustrated | – | 0.09 | – |
| overwhelmed | 0.27 | – | – |
| angry | – | – | – |
| guilty | – | – | – |
| lonely | – | – | – |
| scared | – | – | – |
| sad | – | – | 0.33 |
These are pretty low numbers, to be sure. We had a very small dataset. If we average these up for each classifier, we get:
LSTM: 0.07
ELMo+LR: 0.09
ELMo+MLP: 0.14
That’s kind of a nice result! Things improved as we went along and tried different approaches. If we use the best approach for each mood and average those out, it goes up even further to 0.21!
So, I think I am going to call it a day on text classification until I can find another fun dataset.