From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0002.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0001.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0001.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0003.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0003.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0002.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0001.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0004.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0004.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0003.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0002.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0002.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0005.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0005.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0004.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0003.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0003.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0001.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0006.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0006.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0005.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0004.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0004.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0002.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0007.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0007.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0006.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0005.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0005.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0003.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment-0001.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0008.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0008.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0007.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0006.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0006.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0004.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment-0002.htm From digitas at panix.com Fri Mar 12 19:22:35 2010 From: digitas at panix.com (Randal) Date: Fri, 12 Mar 2010 13:22:35 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: >Jose's title was not clearly formulated in line with his intention: >"Who is in favour of empirical research approach?" I doubt somebody in >this group could not be in favour. > Personally, as a designer, I have never ever seen ANY empirical research results that didn't either: 1. Merely state the obvious. 2. Indicate preference for a design direction that any half decent designer could have directed you to in five minutes. 3. Create results that are essentially useless or at worst completely misleading. Empirical design at its best only prevents really bad and stupid design decisions at about ten times the cost of a good designer who could do exactly the same thing. -- Randal From marconi2006 at googlemail.com Fri Mar 12 19:57:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 12 Mar 2010 15:57:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: Randal, I appreciate your sincerity. I think that there are many others who think just like you. However, how do you test if your design ideas are really working? Jos?. On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody in > >this group could not be in favour. > > > > Personally, as a designer, I have never ever seen ANY empirical research > results that didn't either: > > 1. Merely state the obvious. > > 2. Indicate preference for a design direction that any half decent designer > could have directed you to in five minutes. > > 3. Create results that are essentially useless or at worst completely > misleading. > > Empirical design at its best only prevents really bad and stupid design > decisions at about ten times the cost of a good designer who could do > exactly the same thing. > > -- Randal > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100312/38ae7cf1/attachment.htm From Mateja.Jamnik at cl.cam.ac.uk Sat Mar 13 00:16:32 2010 From: Mateja.Jamnik at cl.cam.ac.uk (Mateja Jamnik) Date: Fri, 12 Mar 2010 23:16:32 +0000 Subject: InfoD-Cafe: CFP: Diagrams in Education Workshop Message-ID: **************************************************************** Diagrams in Education August 9, 2010 Portland, OR, USA http://www.diagrams-conference.org/2010/content/workshops diagrams2010 at diagrams-conference.org A Workshop at the Sixth International Conference on the Theory and Application of Diagrams http://www.diagrams-conference.org/2010/ **************************************************************** Diagrams are ubiquitous in human problem-solving and communication. The success of diagrammatic representations over symbolic/linguistic representations in certain domains is due to the fact that diagrams capture some of the underlying spatiality of the domain, either because the structure of the domain is itself spatial or because it is homomorphic to 2-D space. In either case there is a mapping from aspects of the domain to the domain of diagrams. This mapping, in conjunction with efficient routines that extract information from diagrams as well as modify them, makes diagrammatic representations a powerful tool. Education or instruction is a special case of communication with challenging demands on the instructor. In this setting, communication generally consists of the transfer of information about the structures and processes of a physical or abstract domain to the student. From an instructional perspective then, it is important to understand why and how diagrammatic representations can facilitate this communication. Additionally, given the various differences between diagrammatic and linguistic representations, including their dimensionality, holistic nature, varying levels of abstraction and iconicity, it is also important to understand how these differences, either in combination or individually, enable diagrammatic representations to be a better medium for instruction in appropriate domains. The scope of this workshop covers all areas of diagram use in education. Relevant issues include (but are not limited to) the differences between diagram use in education vs their use in reasoning, aspects of effective diagram construction for instructional purposes, the influence of the domain (expert-to-novice transfer of information) on diagram construction and the spatial features (and temporal features in the case of animated diagrams) that are important in this domain. This will be a half-day workshop. There will be two sessions of an hour and a half each with a break in between. The format will include an invited talk from a distinguished researcher in the field but consist mostly of short presentations from the authors of the accepted papers. Each session will also have a panel (of the session?s presenters) that will answer questions from the audience. *Submissions * ************** We invite contributions on recent work (including work in progress) in this area. Interested participants must submit a 2-page abstract by the submission deadline (April 9, 2010) using the EasyChair conference system (see workshop web page for more information). Accept/Reject notifications will go out May 7, 2010. *Important Dates * ****************** Submission: April 9, 2010 Notification: May 7, 2010 Workshop: Aug 9, 2010 *Workshop Organizers * ********************** Unmesh Kurup, Rensselaer Polytechnic Institute, USA B. Chandrasekaran, The Ohio State University, USA Program Committee Dave Barker-Plummer, Stanford University, USA Andrew Fish, University of Brighton, UK Richard Lowe, Curtin University of Technology, Australia For more information, contact diagrams2010 at diagrams-conference.org **************************************************************** From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0009.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0009.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0008.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0007.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0007.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0006.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment-0004.htm From digitas at panix.com Fri Mar 12 19:22:35 2010 From: digitas at panix.com (Randal) Date: Fri, 12 Mar 2010 13:22:35 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: >Jose's title was not clearly formulated in line with his intention: >"Who is in favour of empirical research approach?" I doubt somebody in >this group could not be in favour. > Personally, as a designer, I have never ever seen ANY empirical research results that didn't either: 1. Merely state the obvious. 2. Indicate preference for a design direction that any half decent designer could have directed you to in five minutes. 3. Create results that are essentially useless or at worst completely misleading. Empirical design at its best only prevents really bad and stupid design decisions at about ten times the cost of a good designer who could do exactly the same thing. -- Randal From marconi2006 at googlemail.com Fri Mar 12 19:57:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 12 Mar 2010 15:57:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: Randal, I appreciate your sincerity. I think that there are many others who think just like you. However, how do you test if your design ideas are really working? Jos?. On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody in > >this group could not be in favour. > > > > Personally, as a designer, I have never ever seen ANY empirical research > results that didn't either: > > 1. Merely state the obvious. > > 2. Indicate preference for a design direction that any half decent designer > could have directed you to in five minutes. > > 3. Create results that are essentially useless or at worst completely > misleading. > > Empirical design at its best only prevents really bad and stupid design > decisions at about ten times the cost of a good designer who could do > exactly the same thing. > > -- Randal > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100312/38ae7cf1/attachment-0001.htm From Mateja.Jamnik at cl.cam.ac.uk Sat Mar 13 00:16:32 2010 From: Mateja.Jamnik at cl.cam.ac.uk (Mateja Jamnik) Date: Fri, 12 Mar 2010 23:16:32 +0000 Subject: InfoD-Cafe: CFP: Diagrams in Education Workshop Message-ID: **************************************************************** Diagrams in Education August 9, 2010 Portland, OR, USA http://www.diagrams-conference.org/2010/content/workshops diagrams2010 at diagrams-conference.org A Workshop at the Sixth International Conference on the Theory and Application of Diagrams http://www.diagrams-conference.org/2010/ **************************************************************** Diagrams are ubiquitous in human problem-solving and communication. The success of diagrammatic representations over symbolic/linguistic representations in certain domains is due to the fact that diagrams capture some of the underlying spatiality of the domain, either because the structure of the domain is itself spatial or because it is homomorphic to 2-D space. In either case there is a mapping from aspects of the domain to the domain of diagrams. This mapping, in conjunction with efficient routines that extract information from diagrams as well as modify them, makes diagrammatic representations a powerful tool. Education or instruction is a special case of communication with challenging demands on the instructor. In this setting, communication generally consists of the transfer of information about the structures and processes of a physical or abstract domain to the student. From an instructional perspective then, it is important to understand why and how diagrammatic representations can facilitate this communication. Additionally, given the various differences between diagrammatic and linguistic representations, including their dimensionality, holistic nature, varying levels of abstraction and iconicity, it is also important to understand how these differences, either in combination or individually, enable diagrammatic representations to be a better medium for instruction in appropriate domains. The scope of this workshop covers all areas of diagram use in education. Relevant issues include (but are not limited to) the differences between diagram use in education vs their use in reasoning, aspects of effective diagram construction for instructional purposes, the influence of the domain (expert-to-novice transfer of information) on diagram construction and the spatial features (and temporal features in the case of animated diagrams) that are important in this domain. This will be a half-day workshop. There will be two sessions of an hour and a half each with a break in between. The format will include an invited talk from a distinguished researcher in the field but consist mostly of short presentations from the authors of the accepted papers. Each session will also have a panel (of the session?s presenters) that will answer questions from the audience. *Submissions * ************** We invite contributions on recent work (including work in progress) in this area. Interested participants must submit a 2-page abstract by the submission deadline (April 9, 2010) using the EasyChair conference system (see workshop web page for more information). Accept/Reject notifications will go out May 7, 2010. *Important Dates * ****************** Submission: April 9, 2010 Notification: May 7, 2010 Workshop: Aug 9, 2010 *Workshop Organizers * ********************** Unmesh Kurup, Rensselaer Polytechnic Institute, USA B. Chandrasekaran, The Ohio State University, USA Program Committee Dave Barker-Plummer, Stanford University, USA Andrew Fish, University of Brighton, UK Richard Lowe, Curtin University of Technology, Australia For more information, contact diagrams2010 at diagrams-conference.org **************************************************************** From gunnar at gunnarswanson.com Sat Mar 13 18:56:51 2010 From: gunnar at gunnarswanson.com (Gunnar Swanson) Date: Sat, 13 Mar 2010 12:56:51 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: <9FFE181F-A2E7-49D1-9DA1-22FE9C35152B@gunnarswanson.com> I fall somewhere in the middle of this. My clients are often small enough that the cost of finding out much about what would be effective would exceed the value of knowing. My experience with "researchers" has not all been good. For example, my students are currently working on a project with a "research" project here at East Carolina University. I believe that I understand the research in their field better than they do. They hide behind being "scientific" even when they wander away from anything backed by actual research. When pressed for reasons for their decisions and demands they resort to "we like it" even though they are working with my students because they say that research shows that designers who are closer to the target audience are more effective. They are among the least strategic people I have ever met. Most of the time when marketing people have told me "research says x is better" I have discovered that they either misunderstand the research or are speaking in third hand generalities learned in class somewhere. It is easy to understand designers taking the position that research is an impediment to good design. That said, I strongly believe that certain (not all) research can inspire creative and effective design and move it forward. Even if that were not true, the research faction has a more effective bit of rhetoric than the traditional design faction so clients will believe them, thus research is inevitable. If the research is controlled by designers, it will be inspirational at best and harmless at worst. If it is controlled by others, it will be a hindrance to design unless we educate researchers. One way or another, designers need to participate in design-related research, if only for self-preservation. As much as I am not a Jakob Nielsen fan, he is right about how much information you can get out of very little testing. It is worthwhile to see if people understand what we assume they understand. That doesn't lead me to believe that A/B testing every design decision or trying to find or follow a set of rules for design is a worthwhile future. A lot of worthwhile empirical information can be cheap and casual, comparable to making dummies and quick prototypes of various sorts. I try to get my students to think in terms of what they could learn that would be helpful as designers. I discourage them from asking friends and roommates questions like "Which of these do you like?" and instead try to get them to be at once more specific and more general--"What do you think the coffee in each of these tastes like?" or "Tell me what you feel about the company behind each of these." Gunnar ---------- Gunnar Swanson Design Office 1901 East 6th Street Greenville NC 27858 USA gunnar at gunnarswanson.com +1 252 258 7006 all new website: http://www.gunnarswanson.com On Mar 12, 2010, at 1:57 PM, Jose Marconi Bezerra de Souza wrote: > Randal, > > I appreciate your sincerity. > I think that there are many others who think just like you. > > However, how do you test if your design ideas are really working? > > Jos?. > > On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody > in > >this group could not be in favour. > > > > Personally, as a designer, I have never ever seen ANY empirical > research results that didn't either: > > 1. Merely state the obvious. > > 2. Indicate preference for a design direction that any half decent > designer could have directed you to in five minutes. > > 3. Create results that are essentially useless or at worst > completely misleading. > > Empirical design at its best only prevents really bad and stupid > design decisions at about ten times the cost of a good designer who > could do exactly the same thing. > > -- Randal > gunnar at gunnarswanson.com +1 252 258 7006 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100313/e4e86ae8/attachment.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0010.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0010.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0009.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0008.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0008.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0007.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment-0005.htm From digitas at panix.com Fri Mar 12 19:22:35 2010 From: digitas at panix.com (Randal) Date: Fri, 12 Mar 2010 13:22:35 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: >Jose's title was not clearly formulated in line with his intention: >"Who is in favour of empirical research approach?" I doubt somebody in >this group could not be in favour. > Personally, as a designer, I have never ever seen ANY empirical research results that didn't either: 1. Merely state the obvious. 2. Indicate preference for a design direction that any half decent designer could have directed you to in five minutes. 3. Create results that are essentially useless or at worst completely misleading. Empirical design at its best only prevents really bad and stupid design decisions at about ten times the cost of a good designer who could do exactly the same thing. -- Randal From marconi2006 at googlemail.com Fri Mar 12 19:57:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 12 Mar 2010 15:57:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: Randal, I appreciate your sincerity. I think that there are many others who think just like you. However, how do you test if your design ideas are really working? Jos?. On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody in > >this group could not be in favour. > > > > Personally, as a designer, I have never ever seen ANY empirical research > results that didn't either: > > 1. Merely state the obvious. > > 2. Indicate preference for a design direction that any half decent designer > could have directed you to in five minutes. > > 3. Create results that are essentially useless or at worst completely > misleading. > > Empirical design at its best only prevents really bad and stupid design > decisions at about ten times the cost of a good designer who could do > exactly the same thing. > > -- Randal > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100312/38ae7cf1/attachment-0002.htm From Mateja.Jamnik at cl.cam.ac.uk Sat Mar 13 00:16:32 2010 From: Mateja.Jamnik at cl.cam.ac.uk (Mateja Jamnik) Date: Fri, 12 Mar 2010 23:16:32 +0000 Subject: InfoD-Cafe: CFP: Diagrams in Education Workshop Message-ID: **************************************************************** Diagrams in Education August 9, 2010 Portland, OR, USA http://www.diagrams-conference.org/2010/content/workshops diagrams2010 at diagrams-conference.org A Workshop at the Sixth International Conference on the Theory and Application of Diagrams http://www.diagrams-conference.org/2010/ **************************************************************** Diagrams are ubiquitous in human problem-solving and communication. The success of diagrammatic representations over symbolic/linguistic representations in certain domains is due to the fact that diagrams capture some of the underlying spatiality of the domain, either because the structure of the domain is itself spatial or because it is homomorphic to 2-D space. In either case there is a mapping from aspects of the domain to the domain of diagrams. This mapping, in conjunction with efficient routines that extract information from diagrams as well as modify them, makes diagrammatic representations a powerful tool. Education or instruction is a special case of communication with challenging demands on the instructor. In this setting, communication generally consists of the transfer of information about the structures and processes of a physical or abstract domain to the student. From an instructional perspective then, it is important to understand why and how diagrammatic representations can facilitate this communication. Additionally, given the various differences between diagrammatic and linguistic representations, including their dimensionality, holistic nature, varying levels of abstraction and iconicity, it is also important to understand how these differences, either in combination or individually, enable diagrammatic representations to be a better medium for instruction in appropriate domains. The scope of this workshop covers all areas of diagram use in education. Relevant issues include (but are not limited to) the differences between diagram use in education vs their use in reasoning, aspects of effective diagram construction for instructional purposes, the influence of the domain (expert-to-novice transfer of information) on diagram construction and the spatial features (and temporal features in the case of animated diagrams) that are important in this domain. This will be a half-day workshop. There will be two sessions of an hour and a half each with a break in between. The format will include an invited talk from a distinguished researcher in the field but consist mostly of short presentations from the authors of the accepted papers. Each session will also have a panel (of the session?s presenters) that will answer questions from the audience. *Submissions * ************** We invite contributions on recent work (including work in progress) in this area. Interested participants must submit a 2-page abstract by the submission deadline (April 9, 2010) using the EasyChair conference system (see workshop web page for more information). Accept/Reject notifications will go out May 7, 2010. *Important Dates * ****************** Submission: April 9, 2010 Notification: May 7, 2010 Workshop: Aug 9, 2010 *Workshop Organizers * ********************** Unmesh Kurup, Rensselaer Polytechnic Institute, USA B. Chandrasekaran, The Ohio State University, USA Program Committee Dave Barker-Plummer, Stanford University, USA Andrew Fish, University of Brighton, UK Richard Lowe, Curtin University of Technology, Australia For more information, contact diagrams2010 at diagrams-conference.org **************************************************************** From gunnar at gunnarswanson.com Sat Mar 13 18:56:51 2010 From: gunnar at gunnarswanson.com (Gunnar Swanson) Date: Sat, 13 Mar 2010 12:56:51 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: <9FFE181F-A2E7-49D1-9DA1-22FE9C35152B@gunnarswanson.com> I fall somewhere in the middle of this. My clients are often small enough that the cost of finding out much about what would be effective would exceed the value of knowing. My experience with "researchers" has not all been good. For example, my students are currently working on a project with a "research" project here at East Carolina University. I believe that I understand the research in their field better than they do. They hide behind being "scientific" even when they wander away from anything backed by actual research. When pressed for reasons for their decisions and demands they resort to "we like it" even though they are working with my students because they say that research shows that designers who are closer to the target audience are more effective. They are among the least strategic people I have ever met. Most of the time when marketing people have told me "research says x is better" I have discovered that they either misunderstand the research or are speaking in third hand generalities learned in class somewhere. It is easy to understand designers taking the position that research is an impediment to good design. That said, I strongly believe that certain (not all) research can inspire creative and effective design and move it forward. Even if that were not true, the research faction has a more effective bit of rhetoric than the traditional design faction so clients will believe them, thus research is inevitable. If the research is controlled by designers, it will be inspirational at best and harmless at worst. If it is controlled by others, it will be a hindrance to design unless we educate researchers. One way or another, designers need to participate in design-related research, if only for self-preservation. As much as I am not a Jakob Nielsen fan, he is right about how much information you can get out of very little testing. It is worthwhile to see if people understand what we assume they understand. That doesn't lead me to believe that A/B testing every design decision or trying to find or follow a set of rules for design is a worthwhile future. A lot of worthwhile empirical information can be cheap and casual, comparable to making dummies and quick prototypes of various sorts. I try to get my students to think in terms of what they could learn that would be helpful as designers. I discourage them from asking friends and roommates questions like "Which of these do you like?" and instead try to get them to be at once more specific and more general--"What do you think the coffee in each of these tastes like?" or "Tell me what you feel about the company behind each of these." Gunnar ---------- Gunnar Swanson Design Office 1901 East 6th Street Greenville NC 27858 USA gunnar at gunnarswanson.com +1 252 258 7006 all new website: http://www.gunnarswanson.com On Mar 12, 2010, at 1:57 PM, Jose Marconi Bezerra de Souza wrote: > Randal, > > I appreciate your sincerity. > I think that there are many others who think just like you. > > However, how do you test if your design ideas are really working? > > Jos?. > > On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody > in > >this group could not be in favour. > > > > Personally, as a designer, I have never ever seen ANY empirical > research results that didn't either: > > 1. Merely state the obvious. > > 2. Indicate preference for a design direction that any half decent > designer could have directed you to in five minutes. > > 3. Create results that are essentially useless or at worst > completely misleading. > > Empirical design at its best only prevents really bad and stupid > design decisions at about ten times the cost of a good designer who > could do exactly the same thing. > > -- Randal > gunnar at gunnarswanson.com +1 252 258 7006 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100313/e4e86ae8/attachment-0001.htm From marconi2006 at googlemail.com Fri Mar 5 15:01:56 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 11:01:56 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? Message-ID: Colleagues, As far I know, there is only one article targeted to designers that presents clear arguments in favour of empirical research (direct observation followed by statistical analysis) as a way to guide design practice: "To picture or not to picture: how to decide" by Judith E. Sims-Knight. Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 * Do you know any other one?* I am sure that your suggestion will help all academic researchers that follow our discussions. Many thanks. -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/f7860496/attachment-0011.htm From frascara at ualberta.ca Fri Mar 5 17:36:12 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Fri, 05 Mar 2010 09:36:12 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: Message-ID: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Dear Jos?, Please look at all the colection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis. Cheers Jorge. Quoting "Jose Marconi Bezerra de Souza" : > Colleagues, > > As far I know, there is only one article targeted to designers > that presents clear arguments in favour of empirical research > (direct observation followed by statistical analysis) > as a way to guide design practice: > "To picture or not to picture: how to decide" > by Judith E. Sims-Knight. > Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 > * > Do you know any other one?* > > I am sure that your suggestion will help > all academic researchers that follow our discussions. > > Many thanks. > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The University of > Reading (UK) > From marconi2006 at googlemail.com Fri Mar 5 21:00:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 5 Mar 2010 17:00:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Dear Jorge, I will try to make myself clear: I am developing a list of arguments in favour of (or against, if you like) empirical research. But, I am focused on empirical research that involves statistical analyses only. I am not looking for examples of design practice informed by empirical research. I agree with you in relation to the "Information Design Journal". I made a contribution myself. How about now? Any specific sugestion? Many thanks for asking. Jos?. On 5 March 2010 13:36, wrote: > Dear Jos?, > > Please look at all the colection of Information Design Journal, you will > find lots of articles about empirical research, or do I not understand your > point? > Also the book "Information design" easterby and Zwaga is full of that, and > there is more, of course. Attention: empirical research is not only > quantitative research, nor statistical analysis. > > Cheers > > Jorge. > > > Quoting "Jose Marconi Bezerra de Souza" : > > Colleagues, >> >> As far I know, there is only one article targeted to designers >> that presents clear arguments in favour of empirical research >> (direct observation followed by statistical analysis) >> as a way to guide design practice: >> "To picture or not to picture: how to decide" >> by Judith E. Sims-Knight. >> Visible Language, v26 n3-4 p324-87 Sum-Fall 1992 >> * >> Do you know any other one?* >> >> I am sure that your suggestion will help >> all academic researchers that follow our discussions. >> >> Many thanks. >> >> >> >> -- >> Jos? Marconi Bezerra de Souza >> Visiting lecturer of Paran? Federal University, Brazil >> PhD - Department of Typography & Graphic Communication, The University of >> Reading (UK) >> >> > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100305/674bad9d/attachment-0011.htm From caroline.jarrett at effortmark.co.uk Fri Mar 5 23:03:23 2010 From: caroline.jarrett at effortmark.co.uk (Caroline Jarrett) Date: Fri, 5 Mar 2010 22:03:23 -0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com From frascara at ualberta.ca Sat Mar 6 11:40:14 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Sat, 06 Mar 2010 03:40:14 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <20100306034014.847324bqva6dcsn4@webmail.ualberta.ca> Dear Jose, Sorry I cannot think now about possible references, but I would only like to say one thing: statistical research is good to tell you the frequency with which something happens, but it says nothing about why it happens. If you do not know why something happens, as a designer, you are left empty handed. Therefore it helps me find a problem, and assess part of its significance, but it does not help me solve it. When I use statistical research, I always mix it with other kinds of research methods. Best regards Jorge Quoting "Caroline Jarrett" : > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From miles.kimball at ttu.edu Sat Mar 6 20:20:57 2010 From: miles.kimball at ttu.edu (Kimball, Miles) Date: Sat, 6 Mar 2010 13:20:57 -0600 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> Message-ID: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> You might look at Davida Charney's article, "Empiricism Is Not a Four Letter Word," in this collection: http://www.amazon.com/Central-Technical-Communication-Johndan-Johnson-Eilola/dp/0195157052/ref=sr_1_fkmr0_1?ie=UTF8&qid=1267902418&sr=8-1-fkmr0 This is also good general text on research methods in technical communication, but much will apply for research in design: Hughes, Michael A. and George F. Hayhoe. A Research Primer for Technical Communication. LEA (http://www.routledge.com/books/A-Research-Primer-for-Technical-Communication-isbn9780805863352) Also check out this general bibliography on communication research methods compiled by a colleague: http://www.faculty.english.ttu.edu/rickly/5363/bib.html For my part, I find that quantitative methods have a tendency to oversimplify the research questions that we feel we can ask -- for example, Nielsen-type usability testing, which focuses so much on finding errors that it potentially leaves out other serious user experience issues. Not that there's anything wrong with quantification -- it just needs to be combined with other methods to get the full picture. I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well. Miles Kimball Texas Tech University -----Original Message----- From: infodesign-cafe-bounces at list.informationdesign.org [mailto:infodesign-cafe-bounces at list.informationdesign.org] On Behalf Of Caroline Jarrett Sent: Friday, March 05, 2010 4:03 PM To: 'Discussions about information design' Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? Jos? explained: > I am developing a list of arguments in favour of (or against, if you like) empirical research. > But, I am focused on empirical research that involves statistical analyses only. Not sure if these examples are what you mean, but here goes anyway. 1. The Direct Marketing industry has focuses solely on A/B testing for ages. Send out a mail shot with half of the pieces done in style A and half in style B: whichever gets the best response wins. I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely available second hand. (Actually, I may buy one: I lent my copy to someone who failed to return it). The results tend to be pieces that are remarkably ugly but effective. 2. More recently, a client of mine decided that he wanted to do A/B testing, which was easy for him to arrange, rather than anything else. We did end up with a better form. I wrote it up: Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of the Usability Professionals' Association Conference, Minneapolis, Minnesota, USA. Article is available as a .pdf from here: http://www.formsthatwork.com/Articles 3. Quite a few of us have been using statistical and numerical methods in usability, usually as adjunct or complementary techniques. For example, I do quite a lot of search log analysis for the Open University and we use other complementary numerical methods e.g. they do a lot of traffic analysis. However, like the thoughtful people they are, they regard these as part of an overall package that includes qual. methods as well. I extracted some examples of the ways that we use quantitative methods in this paper: Jarrett, C. et al " Using Measurements from Usability Testing, Search Log Analysis and Web Traffic Analysis to Inform Development of a Complex Web Site Used for Complex Tasks" http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF ID=80667064&CFTOKEN=87243617 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative methods with qualitative methods, and also sometimes uses quant methods on their own. His approach is written up in: Tullis, T and Albert, W. "Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics" Morgan Kaufmann 5. There is a sort of fashion for numerical-only justification of design decisions going on in parts of the web design community. For example, just in the last few days, there's been a bit of discussion around the "madlibs" style form used by http://huffduffer.com/signup Luke Wroblewski justified his design decisions to use this style with conversion statistics: http://www.lukew.com/ff/entry.asp?1007 but others have fought back with some further statistics: http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o wn-tests/ and scepticism about the way the stats have been generated: http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver sion-by-30-well-maybe/ Best Caroline Jarrett www.formsthatwork.com ___________________________________________________________________ Use the following address to post a message to all subscribers: infodesign-cafe at list.informationdesign.org To subscribe, unsubscribe or change your options, visit: http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe For all Information Design matters: http://InformationDesign.org Problems? Write to: InfoDesign-Cafe-Admin at list.InformationDesign.org ___________________________________________________________________ From r.waller at reading.ac.uk Sat Mar 6 22:35:09 2010 From: r.waller at reading.ac.uk (Rob Waller) Date: Sat, 6 Mar 2010 21:35:09 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: The debate about empirical research rather easily descends into a fight between "it's just your opinion" (scientists about designers) and "you don't understand the real world" (designers about scientists). A version of this debate has been going on as long as I can remember, and before. From 1965: ?Before scientific research, printers and type designers were concerned mainly with the esthetic appearance of the printed page. This preoccupation with esthetics, together with considerations of economy and tradition, dominated all typography until about 1920. As a result of these obstructive emphases, a scientific typography has been slow in developing. Indeed, the printing industry continues to resist procedural changes suggested by experimental findings.? Miles Tinker (1965) Bases for Effective Reading, University of Minneapolis Press, page 115. From 1982: ?[the technology of text] exists as a counterpoint to the artistic and unsystematic approach to text design and layout that has prevailed since petroglyphs were first inscribed on walls?. David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology Publications, page x. On the other hand... From 1931: ?[the univariate research design] is good experimental technique. It is an article of faith among investigators. Yet it won?t work in the way it has been applied to typography unless one is prepared to go to very unusual lengths with it.? Buckingham BR (1931) ?New data on the typography of textbooks? , Yearbook of the National Society for the Study of Education, 30, page 104. From 1975: ?No one, in our opinion, can possibly predict all the variables that might interact with the one chosen for experimentation. And even if a comprehensive list of variables could (by some magic) be drawn up, the permutations would generate millions of experiments ? far more than could ever be executed? The same mental set that predisposes researchers to choose problems that can be neatly handled in a laboratory also inhibits them from going outside their circle to see the problems that typographers face in practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and tests: a framework for improving typography, Programmed Learning & Educational Technology, 12, page 76 __________________________________ Rob Waller Department of Typography & Graphic Communication University of Reading T +44 (0) 118 378 6411 M +44 (0) 7850 665933 On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you > like) empirical research. > But, I am focused on empirical research that involves statistical > analyses only. > > I am not looking for examples of design practice informed by > empirical research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100306/43133476/attachment-0010.htm From marconi2006 at googlemail.com Sun Mar 7 11:35:44 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Sun, 7 Mar 2010 07:35:44 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: Wow!! EXCELLENT feedback. Thank you all for the generosity. -------------------------------------- I suspect that there is another (maybe is the same) interesting debate involving: (1) information designers who know how to design materials that deserves to be tested, but do not (or reject to) know how to design statistically based experiments to test them, and (2) empirical researchers who know how to calculate if there is statistically significant difference between materials, but who does not know how to design (or select) materials that can be considered as good enough by designers, My personal challenge as a researcher (and teacher) is to find a reasonable compromise between both competencies. I am doing my best so... I have learned from my experience as a PhD researcher that the so called "design thinking" should not be restricted to the design of information design artefacts. The real challenge is to apply design thinking to the of design experiments that compares those artefacts. One possible path is (1) to compare "design artifacts" that are recognised (or celebrated) as good responses to a specific problem, and, if "statistically significant" difference is found (2) to Iook for theories that explain those results. It can be the other way around so. As Jorge has pointed out: what really matters is to find WHY something seems to be better in the artificial (and rather limited) experimental context. But, I can advance that to every reasonable explanation in favour of "Design A", I can find an opposite (and quite reasonable too) explanation in favour of "Design B". These contrasting views teach us about the possible scenarios in which A would be better than B and vice-versa. For me, this continuous re-thinking and re-contextualization of apparently effective designs IS the core of the so called design research. Jos?. On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a fight > between "it's just your opinion" (scientists about designers) and "you don't > understand the real world" (designers about scientists). > > A version of this debate has been going on as long as I can remember, and > before. > > From 1965: ?Before scientific research, printers and type designers were > concerned mainly with the esthetic appearance of the printed page. This > preoccupation with esthetics, together with considerations of economy and > tradition, dominated all typography until about 1920. As a result of these > obstructive emphases, a scientific typography has been slow in developing. > Indeed, the printing industry continues to resist procedural changes > suggested by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that has > prevailed since petroglyphs were first inscribed on walls?. David Jonassen > (1982) The Technology of Text, Englewood Cliffs, NJ: Educational Technology > Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it won?t work > in the way it has been applied to typography unless one is prepared to go to > very unusual lengths with it.? Buckingham BR (1931) ?New data on the > typography of textbooks? , Yearbook of the National Society for the Study of > Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the variables > that might interact with the one chosen for experimentation. And even if a > comprehensive list of variables could (by some magic) be drawn up, the > permutations would generate millions of experiments ? far more than could > ever be executed? The same mental set that predisposes researchers to choose > problems that can be neatly handled in a laboratory also inhibits them from > going outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, alternatives and > tests: a framework for improving typography, Programmed Learning & > Educational Technology, 12, page 76 > > __________________________________ > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > > > Dear Jorge, > > I will try to make myself clear: > > I am developing a list of arguments in favour of (or against, if you like) > empirical research. > But, I am focused on empirical research that involves statistical analyses > only. > > I am not looking for examples of design practice informed by empirical > research. > I agree with you in relation to the "Information Design Journal". > I made a contribution myself. > > How about now? Any specific sugestion? > > Many thanks for asking. > > Jos?. > > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > -- Jos? Marconi Bezerra de Souza Visiting lecturer of Paran? Federal University, Brazil PhD - Department of Typography & Graphic Communication, The University of Reading (UK) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/d085181a/attachment-0009.htm From W.Stahl-Timmins at exeter.ac.uk Sun Mar 7 15:35:53 2010 From: W.Stahl-Timmins at exeter.ac.uk (Will Stahl-Timmins) Date: Sun, 7 Mar 2010 14:35:53 +0000 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> Message-ID: <4C759FB1-EEF3-4378-8713-BFAA928695BE@exeter.ac.uk> I'd like to echo Jos?'s appreciation of all the useful links provided over the last few days. I currently work within a health policy research setting, although my background is in graphic design. For my PhD, I'm looking at the use of information graphics in the area. As you might expect, I come across a lot of research that fits into Jos?'s type (2): > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, You may wish to look at some of the following for examples of sophisticated statistical analysis. And very poor designs being tested... Benbasat, I. & Dexter, A. S. (1985). An Experimental Evaluation Of Graphical And Color-enhanced Information Presentation. Management Science, 31, 1348-1364. Elting, L. S., Martin, C. G., Cantor, S. B., & Rubinstein, E. B. (1999). Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures. BMJ, 318, 1527-1531. Feldman-Stewart, D., Brundage, M. D., & Zotov, V. (2007). Further Insight into the Perception of Quantitative Information: Judgments of Gist in Treatment Decisions. Medical Decision Making, 27, 34-43. Frownfelter-Lohrke, C. (1998). The Effects of Differing Information Presentations of General Purpose Financial Statements on Users' Decisions. Journal of Information Systems, 12, 99-107. Lim, K. H. & Benbasat, I. (2000). The Effect Of Multimedia On Perceived Equivocality And Perceived Usefulness Of Information Systems. MIS Quarterly, 24, 449-471. Remus, W. (1984). An Empirical Investigation Of The Impact Of Graphical And Tabular Data Presentations On Decision Making. Management Science, 30, 533-542. Remus, W. (1987). A Study Of Graphical And Tabular Displays And Their Interaction With Environmental Complexity. Management Science, 33, 1200-1204. Hope your research goes well Jos?. It's an interesting area to look at. I do share the general view expressed in this thread that qualitative research is mostly the way to go in terms of evaluating information design. However, I do find a few statistical significance calculations invaluable for explaining the benefits of good information design to people that don't have a design background. I suspect it's not really necessary for designers to quote quantitative metrics when we discuss information design. Our design experience and time spent developing our visual communication skills seem to give us our own ways of measuring the success of our (and others') designs. But when I need to show the differences in comparative information presentations to those with no design backgrounds, I find that statistical analysis of quantitative metrics can be very useful. I think David Sless of the Communication Research Institute in Australia has come to similar conclusions, and I heartily recommend his paper in the Information Design Journal: Sless, D. (2008). Measuring information design. Information Design Journal, 16, 250-258. Cheers all, and keep up the good discussion and intellectual generosity :) --Will. ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG ............................................... Will Stahl-Timmins B.A., M.A. PhD Researcher: Information Graphics in Health Technology Assessment. T: +44 (0) 1392 726 081 M: +44 (0) 7941 865 196 E: w.stahl-timmins at exeter.ac.uk www.pms.ac.uk/infographics/ www.pms.ac.uk/pentag/ www.willstahl.com PenTAG (Peninsula Technology Assessment Group) Peninsula College of Medicine and Dentistry Veysey Building Salmon Pool Lane Exeter EX2 4SG On 7 Mar 2010, at 10:35, Jose Marconi Bezerra de Souza wrote: > Wow!! > EXCELLENT feedback. > Thank you all for the generosity. > > -------------------------------------- > > I suspect that there is another (maybe is the same) interesting > debate involving: > > (1) information designers who know how to design materials that > deserves to be tested, > but do not (or reject to) know how to design statistically based > experiments to test them, > > and > > (2) empirical researchers who know how to calculate if there is > statistically significant difference between materials, > but who does not know how to design (or select) materials that can > be considered as good enough by designers, > > My personal challenge as a researcher (and teacher) is to find a > reasonable compromise between > both competencies. I am doing my best so... > > I have learned from my experience as a PhD researcher that the > so called "design thinking" should not be restricted to > the design of information design artefacts. > The real challenge is to apply design thinking to the > of design experiments that compares those artefacts. > > One possible path is > (1) to compare "design artifacts" that are recognised (or > celebrated) as good responses to a specific problem, > and, if "statistically significant" difference is found > (2) to Iook for theories that explain those results. > It can be the other way around so. > As Jorge has pointed out: > what really matters is to find WHY something seems to be better > in the artificial (and rather limited) experimental context. > > But, I can advance that to every reasonable explanation in favour of > "Design A", > I can find an opposite (and quite reasonable too) explanation in > favour of "Design B". > These contrasting views teach us about the > possible scenarios in which A would be better than B and vice-versa. > For me, this continuous re-thinking and re-contextualization of > apparently effective designs > IS the core of the so called design research. > > Jos?. > > > On 6 March 2010 18:35, Rob Waller wrote: > The debate about empirical research rather easily descends into a > fight between "it's just your opinion" (scientists about designers) > and "you don't understand the real world" (designers about > scientists). > > A version of this debate has been going on as long as I can > remember, and before. > > > > From 1965: ?Before scientific research, printers and type designers > were concerned mainly with the esthetic appearance of the printed > page. This preoccupation with esthetics, together with > considerations of economy and tradition, dominated all typography > until about 1920. As a result of these obstructive emphases, a > scientific typography has been slow in developing. Indeed, the > printing industry continues to resist procedural changes suggested > by experimental findings.? Miles Tinker (1965) Bases for Effective > Reading, University of Minneapolis Press, page 115. > > From 1982: ?[the technology of text] exists as a counterpoint to the > artistic and unsystematic approach to text design and layout that > has prevailed since petroglyphs were first inscribed on walls?. > David Jonassen (1982) The Technology of Text, Englewood Cliffs, NJ: > Educational Technology Publications, page x. > > On the other hand... > > From 1931: ?[the univariate research design] is good experimental > technique. It is an article of faith among investigators. Yet it > won?t work in the way it has been applied to typography unless one > is prepared to go to very unusual lengths with it.? Buckingham BR > (1931) ?New data on the typography of textbooks? , Yearbook of the > National Society for the Study of Education, 30, page 104. > > From 1975: ?No one, in our opinion, can possibly predict all the > variables that might interact with the one chosen for > experimentation. And even if a comprehensive list of variables could > (by some magic) be drawn up, the permutations would generate > millions of experiments ? far more than could ever be executed? The > same mental set that predisposes researchers to choose problems that > can be neatly handled in a laboratory also inhibits them from going > outside their circle to see the problems that typographers face in > practice.? Macdonald-Ross M & Waller RHW (1975) Criticisms, > alternatives and tests: a framework for improving typography, > Programmed Learning & Educational Technology, 12, page 76 > > __________________________________ > > Rob Waller > Department of Typography & Graphic Communication > University of Reading > T +44 (0) 118 378 6411 > M +44 (0) 7850 665933 > > On 5 Mar 2010, at 20:00, Jose Marconi Bezerra de Souza wrote: > >> Dear Jorge, >> >> I will try to make myself clear: >> >> I am developing a list of arguments in favour of (or against, if >> you like) empirical research. >> But, I am focused on empirical research that involves statistical >> analyses only. >> >> I am not looking for examples of design practice informed by >> empirical research. >> I agree with you in relation to the "Information Design Journal". >> I made a contribution myself. >> >> How about now? Any specific sugestion? >> >> Many thanks for asking. >> >> Jos?. > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > > > > -- > Jos? Marconi Bezerra de Souza > Visiting lecturer of Paran? Federal University, Brazil > PhD - Department of Typography & Graphic Communication, The > University of Reading (UK) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100307/77ecd031/attachment-0009.htm From frascara at ualberta.ca Mon Mar 8 10:35:38 2010 From: frascara at ualberta.ca (frascara at ualberta.ca) Date: Mon, 08 Mar 2010 02:35:38 -0700 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> Message-ID: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Dear Miles, You say: "I like Jos?'s distinction between empirical research and its subset, quantitative research. Not all empirical research is statistical -- qualitative methods can be empirical as well." Jose's title was not clearly formulated in line with his intention: "Who is in favour of empirical research approach?" I doubt somebody in this group could not be in favour. Jos?'s distinction came in response to my observation: On Friday March 5, I wrote: "Dear Jos?, Please look at all the collection of Information Design Journal, you will find lots of articles about empirical research, or do I not understand your point? Also the book "Information design" Easterby and Zwaga is full of that, and there is more, of course. Attention: empirical research is not only quantitative research, nor statistical analysis." I use empirical research all the time, including focus groups, user trips, etnographic observation, long interviews, and user testing. I also use statistical analysis, but far more seldom. I think this might be a common pattern for evidence-based work in visual communication design. Cheers Jorge > > > -----Original Message----- > From: infodesign-cafe-bounces at list.informationdesign.org > [mailto:infodesign-cafe-bounces at list.informationdesign.org] On > Behalf Of Caroline Jarrett > Sent: Friday, March 05, 2010 4:03 PM > To: 'Discussions about information design' > Subject: Re: InfoD-Cafe: Who is in favour of empirical research approach? > > Jos? explained: > >> I am developing a list of arguments in favour of (or against, if you like) > empirical research. >> But, I am focused on empirical research that involves statistical analyses > only. > > Not sure if these examples are what you mean, but here goes anyway. > > 1. The Direct Marketing industry has focuses solely on A/B testing for ages. > Send out a mail shot with half of the pieces done in style A and half in > style B: whichever gets the best response wins. > > I learned about this approach from Rene Gnam's "Rene Gnam's Direct Mail > Workshop" Prentice Hall (April 1990) - out of print, but cheap copies widely > available second hand. (Actually, I may buy one: I lent my copy to someone > who failed to return it). > > The results tend to be pieces that are remarkably ugly but effective. > > 2. More recently, a client of mine decided that he wanted to do A/B testing, > which was easy for him to arrange, rather than anything else. We did end up > with a better form. I wrote it up: > Jarrett, C. and Minott, C. (2004) "Making a better web form" Proceedings of > the Usability Professionals' Association Conference, Minneapolis, Minnesota, > USA. > Article is available as a .pdf from here: > http://www.formsthatwork.com/Articles > > 3. Quite a few of us have been using statistical and numerical methods in > usability, usually as adjunct or complementary techniques. For example, I do > quite a lot of search log analysis for the Open University and we use other > complementary numerical methods e.g. they do a lot of traffic analysis. > However, like the thoughtful people they are, they regard these as part of > an overall package that includes qual. methods as well. I extracted some > examples of the ways that we use quantitative methods in this paper: > Jarrett, C. et al " Using Measurements from Usability Testing, Search Log > Analysis and Web Traffic Analysis to Inform Development of a Complex Web > Site Used for Complex Tasks" > http://portal.acm.org/citation.cfm?id=1601632.1601722&coll=GUIDE&dl=GUIDE&CF > ID=80667064&CFTOKEN=87243617 > > 4. Similarly, Tom Tullis at Fidelity has combined a variety of quantitative > methods with qualitative methods, and also sometimes uses quant methods on > their own. His approach is written up in: > > Tullis, T and Albert, W. "Measuring the User Experience: Collecting, > Analyzing, and Presenting Usability Metrics" Morgan Kaufmann > > 5. There is a sort of fashion for numerical-only justification of design > decisions going on in parts of the web design community. For example, just > in the last few days, there's been a bit of discussion around the "madlibs" > style form used by http://huffduffer.com/signup > Luke Wroblewski justified his design decisions to use this style with > conversion statistics: > http://www.lukew.com/ff/entry.asp?1007 > but others have fought back with some further statistics: > http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-o > wn-tests/ > and scepticism about the way the stats have been generated: > http://www.sitepoint.com/blogs/2010/03/03/madlib-style-forms-increase-conver > sion-by-30-well-maybe/ > > > Best > Caroline Jarrett > www.formsthatwork.com > > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > ___________________________________________________________________ > > Use the following address to post a message to all subscribers: > infodesign-cafe at list.informationdesign.org > > To subscribe, unsubscribe or change your options, visit: > http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe > > For all Information Design matters: > http://InformationDesign.org > > Problems? Write to: > InfoDesign-Cafe-Admin at list.InformationDesign.org > ___________________________________________________________________ > > From kschriver at earthlink.net Tue Mar 9 04:50:36 2010 From: kschriver at earthlink.net (Karen Schriver) Date: Mon, 8 Mar 2010 22:50:36 -0500 Subject: InfoD-Cafe: Research Award Announcement In-Reply-To: References: Message-ID: <5043F6B1-B823-4934-964B-3EC360EE8FF5@earthlink.net> RESEARCH AWARD ANNOUNCEMENT We are pleased to announce this year?s winner of the ?John R. Hayes Award for Excellence in Writing Research.? This award, aimed at recognizing outstanding quantitative or qualitative empirical research in writing, is awarded annually to an author or authors of an article appearing in the journal Written Communication. This year?s award will go to Lynda Walsh from the University of Nevada (Reno) for ?Marking Territory: Legislated Genres, Stakeholder Beliefs, and the Possibilities for Common Ground in the Mexican Wolf Blue Range Reintroduction Project? (WC 26[2], 115-153). The selection committee for this year?s award included Peter Smagorinsky (chair), Davida Charney, Ulla Connor, and David Fleming. Lynda will be recognized at CCCC on Friday March 19 at the beginning of paper session H.42, ?Re- considering/Re-visioning Audience.? Please join us in Louisville, Kentucky at the Conference on College Composition and Communication (CCCC) to congratulate Lynda on her excellent work: When: Friday March 19, 11:00 a.m. Where: Louisville Convention Center, Room 218, Level 2 We encourage you or your students to submit to WC to be part of the eligible pool for next year. Both seasoned and new scholars are welcome to enter. Articles will be evaluated for quality of empirical scholarship. Winners will be announced in the journal and will be recognized at a meeting of writing researchers, for example, at the Conference on College Composition and Communication (CCCC) or the American Educational Research Association (AERA). Recipients of the award will receive a custom-designed plaque and a $1000 prize. Karen Schriver KSA Communication Design & Research Pittsburgh , PA -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100308/2b13d24c/attachment-0008.htm From alison at alisonblack.co.uk Wed Mar 10 21:38:02 2010 From: alison at alisonblack.co.uk (Alison Black) Date: Wed, 10 Mar 2010 20:38:02 -0000 Subject: InfoD-Cafe: Design, public policy and behaviour Message-ID: <00a901cac091$94f1dcb0$bed59610$@co.uk> Followers of Deborah Taylor-Pearce?s comments on information design and its connection to food choices might be interested in the UK Cabinet Office publication ?Mindspace: Influencing Behaviour Through Public Policy? by LSE behavioural economists. Mindspace is an acronym for: Messenger we are heavily influenced by who communicates information Incentives our responses to incentives are shaped by predictable mental shortcuts such as strongly avoiding losses Norms we are strongly influenced by what others do Defaults we ?go with the flow? of pre-set options Salience our attention is drawn to what is novel and seems relevant to us Priming our acts are often influenced by sub-conscious cues Affect our emotional associations can powerfully shape our actions Commitments we seek to be consistent with our public promises, and reciprocate acts Ego Pretty depressing on the impact of information (p.15) it has a tiny and (to me) rather vague annexe on the role of design as a ?latest development? in influencing behaviour. Alison -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.webtic.nl/pipermail/infodesign-cafe/attachments/20100310/dae3a516/attachment-0006.htm From digitas at panix.com Fri Mar 12 19:22:35 2010 From: digitas at panix.com (Randal) Date: Fri, 12 Mar 2010 13:22:35 -0500 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <012301cabcaf$acbaea70$0630bf50$@jarrett@effortmark.co.uk> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: >Jose's title was not clearly formulated in line with his intention: >"Who is in favour of empirical research approach?" I doubt somebody in >this group could not be in favour. > Personally, as a designer, I have never ever seen ANY empirical research results that didn't either: 1. Merely state the obvious. 2. Indicate preference for a design direction that any half decent designer could have directed you to in five minutes. 3. Create results that are essentially useless or at worst completely misleading. Empirical design at its best only prevents really bad and stupid design decisions at about ten times the cost of a good designer who could do exactly the same thing. -- Randal From marconi2006 at googlemail.com Fri Mar 12 19:57:15 2010 From: marconi2006 at googlemail.com (Jose Marconi Bezerra de Souza) Date: Fri, 12 Mar 2010 15:57:15 -0300 Subject: InfoD-Cafe: Who is in favour of empirical research approach? In-Reply-To: References: <20100305093612.91936kxk91jos34s@webmail.ualberta.ca> <2CF1E24240363D498E52E28F54F2F208FA8F236887@CRATUS.ttu.edu> <20100308023538.12035m1wetb6uhc8@webmail.ualberta.ca> Message-ID: Randal, I appreciate your sincerity. I think that there are many others who think just like you. However, how do you test if your design ideas are really working? Jos?. On 12 March 2010 15:22, Randal wrote: > At 2:35 AM -0700 3/8/10, frascara at ualberta.ca wrote: > >Jose's title was not clearly formulated in line with his intention: > >"Who is in favour of empirical research approach?" I doubt somebody in > >this group could