Friday, September 11, 2009

Some thought questions for Sept 16

Craig talked about content analyses of process as one way to study and understand design. This week we have 2 studies (Atman et al, Mosborg et al) - one uses verbal protocol analysis to understand what designers do and how this relates to the quality of their final designs, the other explores the ways practicing professional designers think about design processes (one of the tasks should look very familiar!). The third paper is a meta-analysis around "good design".

Some questions to get you thinking:
What can you learn through studying design processes?
What are some relationships between "good process" and "good final product"? How do these ideas relate to your perceptions of what distinguishes more novice and more expert designers?

In class I'll bring some of the "hard data" from these studies for us to push on these ideas :)

Click on the link for "comments" below to post your thoughts


SpngRoy SqrPants said...

The Atman paper results seem to match with the Mosborg and Mehalik papers, effectively saying that the beginning stages of the design process are important for quality designs (unless you are a freshman), but I have some questions on the Atman paper. Was the design activity of designing a playground inherently discipline biased? All the freshman had no engineering experience but the seniors they selected, whom formed a much less diverse (ethnic and sex) crowd than the freshman, were civil, industrial, and mechanical engineers. Would it have been “better” to select all seniors with no domain bias like electrical, computer, and chemical engineers? One of the interesting contradictions between the papers is that the Atman paper reports “Those seniors who considered a greater number of alternative solutions generated a higher quality design.” In the Mosborg paper ROGER makes a great case for not developing numerous alternative solutions, instead making revisions to an initial design mainly because of time and money constraints. But then in the Mehalik meta-analysis paper it, Search the space (explore alternatives), is said to be significant for good design. I do like the insistence on communication in the Mosborg paper though. It underlines the importance of making user-centered designs and of course continuous teamwork. These studies on the design process allow us as future teachers to know which area of the design process to emphasis to students. But how exactly do we do that without causing fixation!?!

Eric Holt Design said...

When a novice begins the design process, they are taught to trust the model – it has worked for countless students and professionals. As they grow in knowledge and experience, they learn to trust in themselves. I would expect seniors to better than freshman, and season design professionals to do better than seniors. Its part of the designers learning process. To paraphrase Jim Horning - Good design comes from experience, and a lot of that experience comes from bad design.

As I read these papers, it challenges me to think through the design process “model” that has developed in me and to look at that process from a different perspective. Because I built my home design business on a fixed fee model, I am constantly looking at ways to make the design process more efficient, so I make more money. A good process produces a good final product efficiently. A bad process might produce a good final product, but the inefficiency of the process will destroy all the joy of the design journey and financial motivation for the designer.

Kevin said...

Two things caught my attention when reading the Atman et al. (1999) study:

1) Many of the same/similar words used to describe design in the previous class exercise appeared in this paper (and other assigned readings), including: “Adaptable”, “Flexible”, “Iterative”, “Methodological/process”, “Many Alternatives/Solutions”, “Constraints”, “Optimization”, “Experience”, “non-linear”, etc. I did not notice any new or unique concepts, compared to our class discussion, used to describe design, therefore I feel we were able to develop a pretty comprehensive list of what design is/involves.

2) As a more quantitative-minded person who helps review papers for journals, I was impressed with (a) the data statistics generated out of qualitative data, (b) level of detail regarding study methodology, and (c) the full-disclosure of potential biases/assumptions and their respective consequences. This latter point has become a rarity in the transportation papers I have reviewed lately . . . but that is a subject for another day.

With some more observations, I think a statistical/econometric model could be calibrated allowing for even more meaningful conclusions. Particularly with a comprehensive set of data statistics, the authors should already have a good idea of what variables will be significant to model/test. More specifically, the data and primary hypothesis (freshmen and seniors have significantly different design behaviors) seem nicely set up for a three-stage least squares (3SLS) regression and likelihood ratio test.

The authors could take the ‘Quality score’ as the dependent variable with independent variables including: the amount of time for each design stage/step, and transition behavior, the number of transitions and information gathering requests/categories, self-assessment ratings, and demographics/socio-economic characteristics (e.g., sex, age, ethnicity, GPA, major, number of credits completed) of subjects. Considering correlation/endogeneity problems, instrument variables are appropriate, and hence a 3SLS is recommended. To statistically verify the difference between seniors and freshmen, a simple likelihood ratio test can reveal the confidence level an author has in that such a difference exists. The parameters (including sign and magnitude) and t-statistics would also provide information on the relative strength of variables in determining the final ‘quality score’. Finally, various statistical/probabilistic techniques (i.e., random effects, mixed models, addition of heterogeneity-capturing term(s), etc.) could be applied to help reduce the uncertainties discussed in the previous class.

All-in-all, the Atman et al. (1999) paper falls far more closely to what I was expecting of a think-aloud study, and I think their conclusions could be further validated/explored with a robust econometric model in the future.

Andrew O said...

The Atman et al. paper was an interesting read for me as I deal with design skill progression from soph. to senior in my courses and notice similar trends in awareness and process. A couple observations for discussions arose in reading. As noted by Roy, there is a significant demographic and specialization shift between the freshmen and senior groups. As with Purdue ENGR there is a retention issue that skews the students in senior courses to those with certain skill sets and proclivities toward design. Does this effect what we can determine about out learning strategies? How would the date look if the same set of students were measured in a longitudinal study? The design step of Communication (last step) had an interesting pattern in the timelines which indicated to me a possibility that the students who had no formal design experience were less confident in speaking about their process due to a lack of specific language and understanding of their own process. How does this potential limitation impact the VPA methodology in such developmental studies?
Mosborg et al. raised a critical question for me. Although their study deals with the question of how do the processes of practicing design engineers differ or agree with those of novice / student designers and with that which is being taught, their focus on one designer in specific raised a second question for me. Do those engineers identified as disruptive innovators (create entirely novel solutions to problems / needs) have a significantly different design process than those classified as incremental innovators (who create designs that are improved over other known designs)? Or is it a difference only in degree and emphasis on certain design steps?
And which type of design (incremental or disruptive) is valued as good design? Mehalik and Schunn talk about quality of design in their paper with a different set of items (processes) than Atman et al. (meeting constraints). Mehalik and Schunn also appear to contrast with the emphasis of Mosborg et al. on the significance of exploring alternative designs. These make me wonder is innovation weighted differently than meeting design criteria in valuing good design or design processes.

michihcim said...

The Mosborg article mentioned that novices tend to think of the “creative” side of design process (although often neglecting the evaluation and iteration part of it). It reminded me that I used to think like that. However the training in the area of circuit design that I went through never had a component of creativity. It trained students to be master craftsmen and not artists. Therefore I wonder how different levels of creativity (analytical ability, etc.) are required in different jobs with titles of design. It could differ in context- whether people do conceptual design, detail design, or design implementation, or depending on the product.

When I began reading Atman’s paper, I had the question of how people judge the quality of design. Another is the technical question of how to segment the transcripts, since I saw some studies done in the similar manner that talked about the frequencies of occurring codes as the conclusion. How researchers segment the data might have an effect on the finding. Both questions are answered in the result section and I am happy. (The second question is bypassed since Atman et al. were interested in the time spent and the transition rate, not so much the frequency of occurring codes.)

One last note, although in class last time we had some disagreements on whether design is always for others, I just want to point out that in the Mosberg article, “Roger’s” comments on communication that involves in the process of design is what Diego mentioned that designers are never alone during the process.

AD said...

After this week’s reading, I feel like we could design an entirely new model of design process based just on this set of literature.
That is probably purposeful, but I find it rather frustrating.
What I found most interesting from this literature were the references to how more experienced/educated designers go about design differently. From my multiple internships with the USDA NRCS, this is something which has always made me jealous of my trainers. My mentors (many of whom have had 15-30 years of experience) were able to visit a job site and rather accurately estimate material quantities and cost on the spot. While reading Roger’s comments in the Mosburg paper about “ability to respond to the inquiries of others” and “estimating fluently,” I reflected on these experiences I had as the novice engineer working with more experienced mentors. In addition to these particular qualities, I was also able to relate to Atman’s discussion of how freshmen engineers spent less time on a design and “did not have a good understanding of their ability to produce a quality design.” As a younger engineer, there can be: 1) pressure to perform a task quickly in order to impress/compete with your superiors/elders, and 2) frustration which develops over not knowing how to complete a design or where to find the information you need, so you rush to what is perceived as “the end” to get the pain of designing over with.

Bethany Fralick said...

What can you learn through the design process? GREAT question, one I think is hard to answer. Is it important to understand how a student defines the process of design? I think individuals learn different concepts and skills from the design process because each uses a different set of design steps. Atman et al researched and found that novice and expert designers have different quality designs, but why?

I am wondering what influenced these students through their academic careers to develop their personalized design process. Was it an internship? course? conference? journal paper? professor?

tforin said...

I'm starting to see a common thread that good design requires a good amount of time spent.

Thinking about the time spent on a design, reminds me of the breadth of experiences a designer has.

A novice designer would not have any significant experiences or at least be aware of such experiences. That in turn allows them to be fixated on one route and that means they spend less time reflecting on the design and the design falls short in the end.

An expert designer would be able to see and use the connections between experiences and the design. Thhe expert's experiences allows for more design routes and now the designer has to rationalize why the experience should be used in the design. So experiences allow for more reflection and iteration in a design. These cycles of reflection and iteration allows the expert designer to spend more time designing and it allows the expert to understand what can be done to make a better design.

I feel that this can be seen very clearly when you observe a designer in the visual arts.

Kat said...

I found the "process" readings quite enlightening. Being an engineer I have been through the design process multiple times in my past. I also was asked some times to describe the "how" question, but I guess since i never had to really "analyze" the process to its full potential, i had never put any thought on some of the steps/activities described. So when it comes to these two articles, i found the results interesting (although some times unexpected, since i always thought that the more experienced you are the less thought transitions you make), but what was really helpful was that now all the smaller steps have an existence and title, i.e. i would never think of including ideas like "abstraction" or "trade-offs" when describing what i am doing/thinking

When it come to the "good design" reading, i would definitely say that although the correlations between the process and the quality have been made kind of clear, I am not at all closer to the definition of what the "good design" finally is. I actually think that going back to the bias discussion, this definition attempt is where all our biases can really be exposed.

Laura said...

Reading the papers this week essentially led me to a similar conclusion as last week. Last week, I thought "design is impossible to define!" and now I'm thinking "design is impossible to teach!"

I came to this conclusion mostly after reading the Mosburg et al. paper. One thing we all seem to agree on is that design is non-linear - however, the most common introduction to engineering design is a block model. And based on the studies described, it seems that engineers sort of agree that this linear-flow model isn't necessarily adequate, whether that's based on the words they use to describe design or considering the steps that they take as engineers.

So essentially the linear block model isn't a sufficient representation of design - but if we don't start teaching that method of design to freshmen, where would we start? Freshmen undergrads don't have the experience or knowledge that practicing engineers or even senior undergrads have. Is it possible to adequately teach design from the beginning, or does the teaching of design have to be evolutionary and nonlinear, just as design is?