A Case Study on Interdisciplinary Collaboration in Unfamiliar Domains

This paper reports a case study in which two designers [one experienced and one inexperienced] and two experienced educators were observed while designing two chemistry educational interfaces for topics with different levels of complexity. The aim was to observe how the designers and educators would collaborate in unfamiliar domains. Variables related to collaboration and topic misunderstandings were measured. The variables related to topic misunderstanding did not point to an effect of topic complexity. Observations suggest an effect of personality interplay on collaborative construction, regardless of the experience of the professionals and the topic under consideration.


Introduction
Before reviewing research focused on interdisciplinary projects, it is necessary to define the meaning of "interdisciplinary design".Because of the lack of [precise] definitions, Tress et al. (2005) defined interdisciplinary as projects that involve several unrelated academic disciplines in a way that forces them to cross subject boundaries to create new knowledge and theory and solve a common research goal.This is the case of educational interfaces projects.
Designing software interfaces, in general, depends on knowledge about the task domain, and the lack of this knowledge affect the problem designing process.Ball et al. (2010) confirmed that when dealing with complex requirements, designers will employ a mixed-strategy mode, solving simpler problems early in the process in a "first-pass", developed in a breadth-first manner, then addressing more complex requirements in a depth-first manner, and spending more time on them.Baker and van der Hoeker (2010), observing the behaviour of software designers, also noticed a change in the strategy used through the design session: designers were trying to find key ideas, from which they would develop secondary concepts.A consequence of this is a dense network of crossed references throughout the protocols, with a variety of requirements being mentioned several times.Ball and Ormerod (1995) found that experienced designers would employ a breadth-first strategy.In these cases, the complexity of the requirements might have triggered changes in the strategy employed by the designers.Visser (1994) proposed that design activity is "opportunistic", in the sense that designers will adopt either breadth-first or top-down strategy as they see fit.
However, as Ball et al. (2010) and Tang et al. (2010) noticed, in such scenarios even expert designers could fail managing broader requirements, i.e. requirements related to user interaction and usability.Christiaans and Almendra (2010) specifically identified that meeting requirements in an integrated manner is a key characteristic of a well balanced design.This need for integrative solutions is pointed by Wallace and Anderson (1993) long ago as a challenge for design teams.They conclude that proper methodological approaches to interface design can reduce the dependence on talent and greatly improve the chances of success.Managing multiple competencies and designers with different backgrounds was also one of the concerns of Détienne (2006), which developed a framework for collaborative design.The author argues that in co-design activities -where actors or teams share an identical goal and contribute in order to reach it -two process are of major importance: establishing a shared frame of reference and establishing mechanisms for clarification and convergence [of ideas and concept] (Détienne, 2006, p.11).When it comes to educational software, the educator usually becomes part of the team, introducing particularities to the design process.

Educational Interface Projects: Managing Interdisciplinarity
Managing teams with professionals from such diverse backgrounds poses a challenge.Back in 1995, van der Mast proposed a method to integrate a team of teachers and students.His method had a broad perspective, including not only costs and team management tools, but also preparation of a software product at the delivery phase.In a recent work in the same area, van Aalst and van der Mast (2003, p.41) identified six recurrent difficulties: project management; poor communication between professionals from different disciplines; depth of skill level; technical issues; psychological issues; and user issues.Out of these, "poor communication between professionals from different disciplines" is unique to interdisciplinary projects.This finding was one of the motivations for the present research.
Despite the difficulty in managing interdisciplinary teams, such integration is highly desirable (Verstegen et al., 2009;Crosier et al., 2002;Siozos et al., 2009;Triantafyllakos et al., 2011;Winters & Mor, 2008).This scenario is likely to be a response to the critique Squires made long ago (1999, p.463), that "workers in these areas (design, programming and teaching) seldom speak to each other or take note of each others' work".In the same context, Zaharias and Poylymenakou (2009, p. 76) mentioned an "ellipsis of research validated usability evaluation methods that address the user as a learner in a holistic fashion, which includes taking into consideration cognitive and affective learning factors".Hinostroza and Mellar (2001, p.27) asserted that "in addition to having knowledge of learning theories, educational software designers should take teaching practices into consideration".Guralnick (2006) expresses the same concenrn, when he says that the design of e-learning interfaces should be determined by how people learn and the tasks they need to perform in the program.He proposes and approach to educational interface design that focuses on the goals, audience, and learning objectives.
The concerns raised by these researchers that true collaboration is not often exercised during the design process was the key element of concern when structuring the designers' and teachers' environment described herein.

Design Research: Methodological Aspects
The method employed for this research was Chi´s Verbal Analyses (Chi, 1997), a variation of the think aloud method -one of the most popular research methods in design research, according to Chai and Xiao (2011).Chi (1997) argues that, regarding ill-structured problems, it is not possible to know in which states of the problem space a problem-solver could be.For this reason, she contrast her verbal analysis method with Ericsson and Simon's (1983) protocol analysis, proposing that verbal analysis should aim "to capture the representation of knowledge that a learner has and how that representation changes with acquisition" (p.3)the emphasis on analysing representations is string on her paper.Chi states that her proposal differs from protocol-analysis regarding [p.4]: "the instruction, the goal or focus, the analysis, the validation, and the conclusion".
The non-optional steps are: [1] to segment the transcripts; [2] to develop/choose a coding scheme; [3] to seek pattern(s); and [4] to interpret them.Chi also points out that the triangulation between quantitative and qualitative analyses strengthens the results.
The first step in Chi's method was achieved by employing the concept of a "design move", defined by Goldschmidt (1992, p.73) as "an act of reasoning that presents a coherent proposition pertaining to an entity that is being designed".As moves are usually made up of few words, we grouped moves into "frames of reference", hereafter referenced only as "frames".We evaluated intra and inter-rater agreement for segmenting the protocols with the "design move" concept, using αu (Krippendorff, 1995).There are no procedures -known to the authors -to evaluate the reliability of grouping moves into frames, as each frame would belong to a different category.
The second step in Chi's method is to develop a code to classify the unitized transcript.The variables in this study are non-categorical, to enable straightforward coding.Some of these variable are related to "design moves", others to "frames".We did not verify the agreement for the coding of these variables because they are a matter of counting.
Observations of an educational interface design assignment was the procedure selected to investigate how designers work in unfamiliar domains.The aim of this research effort was to recreate a real design environment via a case study, where close collaboration between educators and designers was required.This paper presents results from the analysis of variables related to collaboration -measured as the amount of verbal segments where designers and educator discuss about the project -and topic misunderstanding -measured as the amount of verbal segments where designers or educator explicitly manifest doubts about the topic under discussion -of designers and educators working in unfamiliar domains.

Material and Methods
The subjects of our observations were: • A graduate designer with eight years experience as a digital/graphic designer, but no previous experience in designing educational interfaces [subject ED]; • An undergraduate design student with no previous professional experience at the time of the experiment [subject ND]; • A teacher who holds a master degree in chemistry education [educator TA]; and • A PhD professor in chemistry education [educator TB].
Educator TB has experience in designing educational software, while educator TA has experience in using digital collaborative learning tools.Both chemists were co-workers from 2008 to 2010 in the same research department working under the supervision of the same professor, although in different assignments.The subjects were split into two pairs: ED+TA and ND+TB.The authors are aware that having few subjects and varying two parameters at the same time [experience level and chemistry topic] does not allow for conclusions through statistic tests to be drawn.However, this is not the aim of this research: our overall approach is to triangulate quantitative/descriptive data in a qualitative analysis with the understanding that the results are representative of the particular situation described in this case study.In doing so, it allows for the influence of two factors that are common in the routine practice of educational interface design teamwork to be observed: different levels of design expertise and different levels of topic complexity.It is important to stress that it is usual to have students in educational software teams (van der Mast, 1995;van Aalst & van der Mast, 2003), reason for which an inexperienced designer was included in the experiment -and this is the case at the university where the authors work.A better understanding of the student's reactions and needs will guide future research on educational interface design.
Having few subjects does not undermine the research design, as this can be described as a case study (Yin, 2009), a well suited research approach to design problems, according to Langrish (1993) and Roth (1999).Examples of similar studies [close investigation of few subjects] can be found in Suwa, Gero and Purcell (2000) -the analysis of the same protocol of one architect; Goldschmidt (1995Goldschmidt ( , 2003) )   The briefing stated that the audience was secondary school students and that the software could be used in either distance or presence education.The teams were asked to provide drawings that could be used as a starting point for the development of the interface.A4 paper and pencils of different types and colours were available, but books or computers could not be used to prevent searches for images, texts or other types of data, as this would take the focus away from the interface design.
The use of the think-aloud method with more than one subject, although not the rule in design research, is not uncommon.Observations of team work dates back to the 1980s (e.g.Valkenburg & Dorst, 1988;Cross & Clayburn Cross, 1995).Austin (2001) designed a fairly similar setting to investigate multidisciplinary teams of 5 designers, whose assignment was related to architecture.

Observed Variables
Observations consisted in tracking variables that would relate to the main premises considered herein: collaboration and influence of topic complexity.Détienne ( 2006) accounted that sharing a local context is a characteristic of grounding activities in co-design.Misunderstandings relate to the clarification of the subject matter/subject domain, and point to doubts or incorrect/incomplete comprehension.
• EDUCATOR_FRAMES: percentage of frames and time in which only the educator spoke.
• DESIGNER_FRAMES: percentage of frames and time in which only the designer spoke.
• SHARED_FRAMES: percentage of frames and time in which both the designer and the educator spoke.
• EDUCATOR_MISUNDERSTANDINGS: percentage of moves in which the educator expresses doubts or demonstrates misunderstandings.
• DESIGNER_MISUNDERSTANDINGS: percentage of moves in which the designer expresses doubts or demonstrates misunderstandings.

Unitizing and Classifying Design Moves and Frames
The verbalizations were unitized using the concept of "design moves", proposed by Goldschmidt (1991Goldschmidt ( , 1992Goldschmidt ( , 1995Goldschmidt ( , 1997)), because it reflects intentions of the designers regarding the process, which is suited for the objectives of the present research.Goldschmidt defines "design moves" as "the smallest coherent operation detectable in design activity" (1991,1992); "an act of reasoning that features a coherent proposition pertaining to an entity that is being designed" (1991,1992); "a step, an act, an operation, which transforms the design situation relative to the state in which it was prior to that move" (1995); "moves in the problem space are the small steps in which reasoning proceeds" (1997).Design moves have been used as a rule for unitizing verbalizations in several researches such as Kan & Gero (2008); Gero and Tang (2001) and Suwa et al. (2000).

Analyses of the Variables Related to Collaboration and Misunderstandings
The results related to the variables EDUCATOR_FRAMES, DESIGNER_FRAMES and SHARED_FRAMES are presented in Table 1.Each cell in this table shows the percentage of frames per session and length (percentage of session total time).The results for this set of variables show that for pair ND+TB the value of EDUCATOR_FRAMES was higher and they lasted longer in their sessions.For this pair, the amount of DESIGNER_FRAMES increased slightly across sessions [from 5% to 7%] while the time the designer spoke alone in moves remained the same [4%].The value of SHARED_FRAMES decreased slightly across sessions, both in amount and length.This is consistent with the observations reported in the previous section: designer ND did not shared decisions regarding pedagogical and chemistry related topics.As a consequence, educator TB spoke alone for a longer time.
Different conclusions were obtained for pair ED+TA.EDUCATOR_FRAMES increased across sessions [from 2% to 9%], but the time remained about the same [from 1% to 2%].For this pair, the count of DESIGNER_FRAMES decreased across sessions [from 12% to 3%], while the length showed no significant variation [from 10% to 8%].This indicates that EDUCATOR_FRAME and DESIGNER_FRAME were shorter in the second session.Similar to the previous pair, the value of SHARED_FRAMES changed slightly across sessions.This evidence has not been considered as an indication of topic complexity, since there was no consistent increase in amount and time of EDUCATOR_FRAMES.However, the values of these three variables were very different between the pairs -different enough to point to an effect of the pair.This conclusion is consistent with was observed during the design session.However, we cannot be certain because there are no variables related to personality characteristics.We do not consider this difference in the amount of time each educator spoke as a function of knowledge of the subject matter [science and chemistry education; computers and education] as we consider both educators master these subjects.
To complement this analysis, Table 2 shows the content of EDUCATOR_FRAMES and DESIGNER_FRAMES.The transcriptions of pair ND+TB show that their collaborative design did not have the same characteristic, reflecting in their higher values of EDUCATOR_FRAMES and lower values of SHARED_FRAMES.This might have happened because educator TB treats both subjects [SM and SW] using sophisticated argumentation and non trivial approaches, e.g., the very odd state transformation of iodine [session SM], a conceptual map of chemistry topics, and using a tripod of representation prototypes -semantic, numeric and pictographic -to explain chemistry concepts.Educator TB said, in session SW, that those are not common ways to approach those chemistry topics in high school.These were frames mostly related to science education, a topic that is unfamiliar to designers, and this was the reason why the values of EDUCATOR_FRAMES were higher for educator TB.Educator TA addressed many of the chemistry topics educator TB addressed, but did not use such non-trivial approaches.
Complementing this analysis, the values of the variables related to misunderstandings are shown in Table 3.This table shows the percentage of moves where designers or educators expresses doubts or demonstrates misunderstandings.The results were unexpected because the Solubility in Water topic is more complex than States of Matter, and there was an inherent assumption that the amount of misunderstandings would be higher in the second topic (SW).These results again suggest that the topic under consideration had no major influence on collaboration.A possible explanation is that the designers did not question the educators in depth on chemistry concepts, relying on the teachers' experience and knowledge.The next step in this analysis is having a closer look at occurrences of misunderstandings.
Starting with session SM_ED+TA, Table 4 has example of moves classified as misunderstandings in this session.This example shows educator TA explaining to designer ED the importance of a phenomena using different representation models.This specific case under consideration was pointed out by both educators, and is used to indicate that the ability to translate between different types of representations is a characteristic consistently demonstrated by experts and absent in novices (Kozma & Russel, 1997).

Representations in Chemistry Education -session SM_ED+TA
Then it could be about the aggregation (sic) of the matter… Then it is up to you.
Yes, then it will be about representational levels.
Conceptual? Like, concepts and representations?

…
Then it had to be a simulation, showing those structures.

Would it be an illustration or an image of those…
We do not have images of atoms… Those would be illustrations that would represent this…

Screen comparing water + oil and water + sugar -session SW_ED+TA
We could make a box comparing… Water and sugar, they have a greater interaction…

And the other [oil] does not… They repel each other…
No, they do not repel… They do not interact.
Yes, so… We could make this comparative box of the water interacting with sugar…And the oil does not [interact]… Tables 4 shows that even complex issues related to science education and chemistry were quickly resolved by designer ED.When the topic domain was unfamiliar, designer ED accepted what educator TA suggested.Educator TA, on the other hand, does not try to explain those concepts in depth to the designer.While this might be an efficient strategy for dealing with non-familiar domains, it can hinder the designer from making valuable contributions.The picture is different for designer ND, since he had fewer misunderstandings than designer ED (1% in both sessions).In session SM, the misunderstandings were scarce, and were related mostly to chemistry concepts.In Table 5 there are examples of moves classified as misunderstandings for the pair Designer ND + Educator TB, for both sessions.In our evaluation, Designer's ND misunderstandings were more severe because, as pointed out before, educator TA uses a sophisticated argumentation line.In this example, educator TA could have just said that the "representational level" is made out of formulae and abbreviations.Instead, educator TA explained why do chemistry concepts have to be depicted in those representational levels.At that time, educator TA was not talking about "theories".For designer ND, the location pattern of misunderstandings also changed across sessions.In session SW, they were concentrated in frames related to the effects of temperature and screen elements.As with designer ED in session SM, only one of these topics is related to chemistry.

DESIGNER ND EDUCATOR TB Representations in Chemistry education -session SM_ND+TB
And another very important issue is that of language: the characteristics that are going to enunciate the behavior.
And in this enunciation of language (sic) there is a whole description of chemistry which is carried out through formulae, abbreviations… So we call it macroscopic, submicroscopic and representational language.So those three ideas have to be present every time we explain any chemistry concept.

And what about theories?
Effects of temperature -session SW_ND+TB And there is also this debate with the student, about the effects of temperature on solubility of kitchen salt… Then we can discuss this with a macroscopic model too… Like a video or something like that.

And I added examples of referenced waters (sic) [with different solutes], that could be in this unit…
Of the effect of temperature?But then what?We could have this "macro" (sic) session….
Like if you would return to a previous subject?[starts reviewing the sheets of paper] This content is much more complex that the previous, huh (sic)?It is much denser, huh (sic)?
In this example, designer ND is not following the conversation with educator TB.He is trying to make a contribution to the effects of temperature in solubility, but he gets confused.
Our conclusion, regarding these variables, is that designer ED did not try to understand chemistry concepts in depth, and educator TA did not try to explain them in depth.On the other hand, educator TB explains the topics using a sophisticated argument, which [1] does not allow much time for designer ND to participate, as educator TB's arguments were lengthy, and [2] they were not entirely clear for designer ND.This evidence points to an influence of personality interplay, an effect that would be difficult -if possible -to control or measure.

Conclusion
The aim of this case study was to observe the collaboration of professionals in unfamiliar domains, since this is a common situation in the design of educational interfaces.Few studies have been reported on the design process in interdisciplinary fields, and for this reason an explorative case study was conceived to better understand the design process.
Five variables were analyzed under collaboration and misunderstanding, and they indicate differences between pairs ED+TA and ND+TB.The values of SHARED_FRAMES for pair ND+TB were lower than those of pair ED+TA.The difference is due to EDUCATOR_FRAMES which was higher for pair ND+TB in both sessions.
For pair ED+TA, while the value of DESIGNER_FRAMES lowered and EDUCATOR_FRAMES increased throughout the sessions, the length in time of these variables remained similar in both sessions [about 1% of the total session time for EDUCATOR_FRAMES and 10% of the total session time for DESIGNER_FRAMES; see Table 1].The same was observed for pair ND+TB.Since the values of those three collaboration variables remained similar for both pairs in both sessions, it has been concluded that the complexity of the topic did not influence collaboration.However, since the values of these three variables were different between the pairs and similar within the pair, it is likely that this is an effect of personality interplay.
This observation is reinforced by the analysis of the non-shared frames.Educator TB's frames reflect the sophisticated conceptual understanding of chemistry throughout both sessions through lengthy and relatively complex argumentation.In session SW, the educator recognizes that his approach to education was not standard when teaching states of matter and solubility in water.In contrast, educator TA refrained from complex arguments, allowing designer ED to speak about education and even chemistry, being pro-active in critical decisions throughout the design process.
As for the analysis of the variables related to misunderstandings, it was expected that the rather complex concepts on Solubility in Water would impose more difficulties on designers -which in turn would be reflected by the amount of misunderstandings the designers would express.However, this was not observed, despite the fact that the Solubility in Water topic is more difficult than States of Matter -as agreed by both educators.This evidence is partially explained by the designers' response when confronted with a new subject: they did not try to understand the subject in depth by questioning the educators, but accepted the educators' arguments.The misunderstandings of designer ND were more severe than those of designer ED, which is, in our assessment, due to the non-trivial argumentation line of thought of educator TB.
As a general conclusion, some of the results reported in the present case study were not expected: the weak effect of topic and the strong effect of personality interplay.There is sufficient evidence to indicate that attention should be paid, in future researches, to personality interplay and how it affects team performance.The variables tracked in this study may provide useful guidance for future observations.
Regarding the collaboration and connectivity variables, higher scores should be seen as an indication of teamwork and optimized design process.One of the requirements for improving collaboration is to enhance the designers' knowledge on education, pedagogy and cognitive sciences, which might not only improve communication with educators, but also improve the designers' strategies towards interactiveness in educational software.
Figure 1.V s rewarded 0,00.We c e experimen ons related t Because moves are usually made up of

Table 1 .
Values for the variables related to authorship.

Table 2 .
Content of the frames of pair ED+TA and ND+TB.

States of Matter (SM) Session Solubility in Water (SW)
By examining the non-shared frames of the four sessions [Table 2], it is possible to see that most of them were specific to each professional's practice.The exceptions are the DESIGNER_FRAMES of session SM, where designer ED argues about the need to have examples other than water; the pedagogical strategy the pair should follow and what features the screen with exercises should have.

Table 3 .
Content of the frames of pair ED+TA and ND+TB.