Social scientists study behaviour by analysing how people speak and listen. Now, TU computer scientists are stepping in with motion sensors. There are some concerns, though.
In the future your smartphone might reveal more about you than you know yourself. (Photo: Sam Rentmeester)
At a recent speed date event, visitors were given an accelerometer to hang around their necks. This motion sensor recorded their movements. Participants had one-on-one conversations first, followed by a mingling session. “We set it up like this for two reasons,” says Dr Ekin Gedik. “We were interested in the way people mingle in an uncontrolled environment. We also decided to use an accelerometer instead of a microphone or camera, because these make people feel self-conscious. We didn’t want to influence their behaviour.”
Gedik, who recently defended his PhD thesis at the Faculty of Electrical Engineering, Mathematics and Computer Science, started the research because he wondered if it were possible to model human behaviour. The processing power of computers is growing, which enables collecting and analysing large amounts of behavioural data. Gedik decided to test this power by building a computer model that can analyse human behaviour. In his first model, he focused on estimating personality traits by tracking people’s movements.
Before the speed dating event, all the participants took a personality test. The results of these tests were put into the model. At the event, the accelerometer tracked the movements of the participants, and the model connected this data to their personality traits. This method could then be reversed. The model now can derive personality from movement. This process of learning from data is called machine learning.
Challenges for the technology
The method is not flawless yet, as interpretation of people’s motion is difficult. A wallflower stays in the same spot and has timid conversations, whereas a social butterfly moves around the room speaking to everybody. The tracked movement may be very different, but they are both speaking.
Furthermore, if someone makes animated arm movements for example, the sensor cannot tell if that person is enthusiastic or aggressive. Machine learning can help with these interpretation challenges. The speed date event was recorded on video. Afterwards, volunteers made notes about participants’ behaviour by watching the material. Putting these notes into the model improved its interpretation capacity.
‘Modelling people’s personalities is not really ethical’
Collecting data about people of different origins is also necessary, as the way people act is culturally determined. An Italian probably speaks a lot more animatedly than a Swede, even though they may be saying the same thing.
Ethical concerns of behavioural analysis
Gedik is looking at how to improve the technology. “We want to use smartphones because the built-in sensors can provide much more data than the basic accelerometers we used. Besides, they enable upscaling the experiment easily.” If smartphones were to be used, it would require adaptations to the computer model.
Should smartphones be used, there would also be privacy and ethics concerns. Getting people’s consent to use their data and ensuring that movement would only be tracked at specific times are both points for consideration.
Not only the collection of data is an ethical question, what to do with that data is something that troubles Gedik as well. “Personally, I believe that modelling people’s personalities is not really ethical. It would largely depend on the purpose. If you’re modelling someone’s personality to enhance their experience, it could be good. If you use it for job applications, then less so.”
Gedik is enthusiastic about the results, but he does not think the model will replace social scientists just yet. “Right now, I view it as complementary. The model might help scientists see patterns that they overlook.” In the future, however, your smartphone might reveal more about you than you know yourself.
- Ekin Gedik, Capturing human behaviour through wearables by computational analysis of social dynamics, PhD supervisors Prof. Marcel Reinders and Dr Hayley Hung at the Faculty of Electrical Engineering, Mathematics and Computer Science TU Delft, 4 December 2018.
Nina van Wijk / science editor trainee
Comments are closed.