Use case: Social ethnodroids for pretotyping in language learning

In this two-part essay, in Using Pretotype vs Prototype in hypothesis-based entrepreneurship, we first introduced the concepts of pretotyping and hypothesis-driven entrepreneurship.  In the second part, we are going to apply this approach to pretotyping in language learning using an ethnographic robot called a social ethnodroid. See also another pretotyping experiment Hypothesis-driven: Nordstrom’s ‘flash build’ for app development

This case was inspired by Wiles, J., Worthy, P., Hensby, K., Boden, M., Heath, S., Pounds, P., … Weigel, J. (2016). Social cardboard: pretotyping a social ethnodroid in the wild. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction (pp. 531–532). IEEE Press. Retrieved from http://dl.acm.org/citation.cfm?id=2906962.

Note: Review and discussion questions for classroom discussion appear at the end.

Janet Wiles, Professor of Complex and Intelligent Systems at The University of Queensland, pulled together a team of researchers from a wide range of disciplines, from designers and developmental psychologists to engineers and cognitive scientists. Why? The brief was simple, the team wanted to study the interaction of technology and language by pretotyping in language learning . They sought to develop a child-friendly robot that could evaluate young children’s interactions with tablet-based language learning tasks and games. Would the children be willing to learn language from a robot?  Would it be possible eventually to design a teacher’s assistant that promoted learning of aboriginal languages in remote Australian regions? These were some of the questions the researchers wanted to answer. ^[1]

To that end, they first created a social ethnodroid – a robot that functions as an ethnographer – that could measure tablet-based learning progress and interactions with children ages one to six years old. Rather than build an expensive prototype, they developed a cardboard robot with tablet computers in its face and torso. Using A/B testing, they tested it in a relatively structured environment of an early-learning centre and in a relatively unconstrained setting of a science fair. This was a rapid, inexpensive experiment to examine the children’s attitudes and behaviors in the two user contexts and provided insights into form, sensors and analyses for further design. Their idea was to do a ‘quick and dirty’ test before they launched into full product design.

Ethnodroids and the Wizard of Oz technique

In design terminology, ‘prototyping’ covers all aspects of development. [[See previous essay]].  A prototype what you produce just before you manufacture. It costs a lot of money and shows all the features the product will have. Prototyping seeks to answer questions such as: ‘Can we build it at all?’, ‘Will it work as expected?’, ‘How cheaply can we build it?’, ‘How fast can we make it?’.

Pretotyping, in contrast, involves building a scaled-down version of a product, known as the ‘minimum viable product’ (MVP), which is the smallest and least expensive solution that delivers and captures customer value’.[2] Pretotyping is one of the tools of ‘lean start-up’ where innovators and inventors can address questions about what product to build. Its research question are different: ‘Would people be interested in it at all?’, ‘Would they buy it if we build it?’, ‘Will they use it as expected?’, ‘Will they continue to use it?’. A prototype of a real functioning robots would normally require expensive hardware and software and would entail substantial, perhaps prohibitive, investment.

Now to the case, Janet Wiles is Professor of Complex and Intelligent Systems at The University of Queensland. Her research seeks to understand complex systems with particular applications in biology, neuroscience and cognition. Wiles pulled together a team from a wide range of disciplines from designers and developmental psychologists to engineers and cognitive scientists to explore pretotyping in language learning. They were keen to test how (and whether) their main customer, children, would interact with a robot.

Using pretotyping methodology, they constructed a minimum viable social ethnodroid that only simulated interaction with children, who were none the wiser about the internal workings of the bot. Wiles and collaborators used the Wizard of Oz test to pretotype their robot. Just as in the scenes in Frank Baum’s famous novel, Dorothy, Scarecrow, Tin Man, and Cowardly Lion paid no attention to the man behind the curtain.[3] Only when Dorothy’s dog Toto pulls back the curtain, do they discover that a meek, middle-aged man was projecting a fearsome, disembodied head surrounded by fire on to the curtain.

Team develops a child-friendly robot

For Wiles and her team there was no sense in spending millions to test the robot. As Savoia, the father of pretotyping, says, it helps you to ‘make sure you are building the right ‘it’ before you build it right.’[4] The team’s hypothesis-based approach sought to discover if they were even building the right ‘it’.

To begin, they built a social ethnodroid.  An ethnodroid is a research robot that functions as an ethnographer to observe and collect information about human behaviors. It is a social robot that provides an embodied presence for research into social interaction with the target group.

Using design thinking techniques, the team developed a child-friendly ethnodroid called Opie to evaluate social components of children’s interactions with tablet-based learning tasks and games. The key requirements for the MVP were that it be both child-safe and socially engaging. The team sought to answer these questions:

A range of robots were developed (see top) to explore pretotyping in language learning.[5] The general form was a semi-anthropomorphic robot with stylized elliptical head, a torso constructed of sheets of medium-density fiberboard secured to a wooden frame, and arms constructed from foam tubes ending in paddle-shaped hands. The frame and equipment behind the robot were covered by a black curtain. A tablet computer in the head showed a face, and another tablet in the torso was used for games.

The robot was positioned in the corner of a small, child-safe room. A dissembler-facilitator (the Wizard) hid behind the curtain and interacted with children ages one to six years in groups and individually. Children were invited to interact with the robot and a facilitator for 10 minutes. Testing was an ongoing process that enabled many small refinements of the design. Other experiments took place at a roped-off corner of a science fair. The key indices collected included massive amounts of ‘touch data’ and visualizations captured by sensors in the tablets and by observers. Savoia’s pretotyping questions were adapted to children’s interactions with the robot.

Using Savoia’s pretotyping questions

• Were the customers (children) at all interested in a minimally viable robot?  Few of the research team expected that it would engage children’s interest for a substantial period of time. To their surprise, most children were strongly attracted to the robot in both the classroom and science fair settings. They engaged for up to twenty-three minutes with the robot and most played through all four available games. The children were also tolerant of the robot’s occasional malfunctions. The team concluded that children were sufficiently interested even in the robot’s minimally viable pretotype form to support its research functions.
• Would the children’s carers and parents buy it if the project built a real, function robot? Savoia’s second question applies to parents and teachers, the actual gatekeepers to the children’s time and attention, and to researchers who deploy the robot as a context for interaction. Key considerations in the studies concern ethics, data collection, and storage.[6] Both the classroom and science fair settings involved voluntary visits with the robot, so it was not surprising that all parents were positive about the research aspects of the study. However, the team were surprised by the enthusiasm shown by parents, despite the robot’s limited functionality and form.
• Will children, the robot’s primary customers, use the robot as expected? Testing showed some unexpected behaviours between the children and the robot with respect to the tablet, voice, eye contact and physical touch. The team had anticipated interest in the robot’s hands and had implanted touch sensors. However, children in the lab rarely touched the robot except for the tablet and the touch sensors were not triggered. At the fair, participants touched the robot on the hands, arms and torso, and some used the frame for support. These touches were quantified through manual hand coding. The cameras and microphones recorded the interactions between participants and robot in a relatively constrained field of view and showed that sensor placement needed adjustment. The key finding for ethnography was the massive amounts of data generated from every trial. Coding of touch data resulted in rapid innovation in sensors to automate the touch analyses and visualizations to interpret the data, and prompted the design of automated analysis tools much earlier than planned.[7]

Key findings of the use of pretotyping in language learning

The key finding of this hypothesis-based pretotyping experiment is just how little a robot actually needs to be social and engaging: Minimum Viable Product = A cardboard frame + foam arms + two tablets – such a cheap MVP can actually be socially engaging, given the right context and Wizard.

The social interactions were not just between child and robot, but were structured by the settings, the parents, facilitator, the Wizard of Oz technique, and the robot’s autonomous communication. In its ethnographic role, it became obvious that the robot could generate more low level sensor data than could be stored for analysis, and that interpreting the data deluge had to be designed from the start.

In the end, the team concluded that pretotyping could not only be used for product development but also as a research tool. The process proved useful in assisting the researchers to know what to look for and identifying future needs for functionality, demonstrating the value in extending the pretotyping concept beyond its current setting in the commercial world.

For some of the team this process led to a ‘pivot moment’ when they realized that the robot could also be used for other purposes, for example, to teach children from a remote Aboriginal community their traditional languages. The technology has been deployed in classrooms in the southeast Arnhem Land community, programmed to help teach heritage languages. The robot was programmed with interactive language activities and memory games that encourage the children to identify, sound out, and repeat words back so that teachers can track their progress.[8]\

The major learning from the ‘droid experiment was ‘Fail fast, pretotype often, validate relentlessly, and pivot quickly’.

Questions

1. Can you describe the difference between a prototype and a prototype by brainstorming a new product or service and validating it?
2. What are the A/B testing experiments that the team carried out?
3. Name five ‘Wizard of Oz’ tests that you can imagine for other products or services?
4. Can you think of any even cheaper way to test children’s learning through robots? (Hint: Think ‘Mechanical Turk’ in Chapter 7.
5. The team discovered how little a robot actually needs to be social and engaging.  Are there any features on the ethnodroid that could have been left out and still had a strong validation?

---

[1] This case was inspired by Wiles, J., Worthy, P., Hensby, K., Boden, M., Heath, S., Pounds, P., … Weigel, J. (2016). Social cardboard: pretotyping a social ethnodroid in the wild. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction (pp. 531–532). IEEE Press. Retrieved from http://dl.acm.org/citation.cfm?id=2906962.

[2] Maurya, A. (2012a). Running Lean: Iterate from Plan A to a Plan That Works (2 edition). Sebastopol, CA: O’Reilly Media; Maurya, A. (2012b). Running lean: iterate from plan A to a plan that works. O’Reilly Media. Retrieved from https://goo.gl/cLgGVh; Maurya, A. (2013). What is a Minimum Viable Product (MVP). Retrieved from https://www.youtube.com/watch?v=MHJn_SubN4E; Maurya, A. (2016). Scaling Lean: Mastering the Key Metrics for Startup Growth. New York: Portfolio.

^[3] Akers, D. (2006). Wizard of Oz for Participatory Design: Inventing a Gestural Interface for 3D Selection of Neural Pathway Estimates. In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (pp. 454–459). New York, NY, USA: ACM; Höysniemi, J., Hämäläinen, P., & Turkki, L. (2004). Wizard of Oz Prototyping of Computer Vision Based Action Games for Children. In Proceedings of the 2004 Conference on Interaction Design and Children: Building a Community (pp. 27–34). New York, NY, USA: ACM.

[4] Savoia, Alberto. 2014. ‘Pretotype It.’ Jama Software. https://www.jamasoftware.com/blog/pretotype/.

[5] Worthy, P., Boden, M., Karimi, A., Weigel, J., Matthews, B., Hensby, K., ... & Viller, S. (2015, December). Children's expectations and strategies in interacting with a wizard of oz robot. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (pp. 608-612). ACM.

[6] Savoia, A. (2016). How ethical is pretotyping? Retrieved May 15, 2017, from https://www.quora.com/How-ethical-is-pretotyping

[7] Hensby, K., Wiles, J., Boden, M., Heath, S., Nielsen, M., Pounds, P., ... & Smith, M. (2016, March). Hand in Hand: Tools and techniques for understanding children's touch with a social robot. In Human-Robot Interaction (HRI), 2016 11th ACM/IEEE International Conference on (pp. 437-438). IEEE; Rogers, K., Wiles, J., Heath, S., Hensby, K., & Taufatofua, J. (2016, March). Discovering patterns of touch: a case study for visualization-driven analysis in human-robot interaction. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction (pp. 499-500). IEEE Press; Evans, M., Kerlin, L., & Jay, C. (2015, April). I Woke Up as a Newspaper: Designing-in Interaction Analytics. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 477-488). ACM.

[8] Mounter, Brendan (2017). Opie the Robot helping preserve ancient languages in remote Aboriginal Australia. Australian Broadcasting System.