Measuring Hacking Ability Using a Conceptual Expertise Task

Justin Scott Giboney, Jeffrey Gainer Proudfoot, Sanjay Goel, Joseph S. Valacich


Hackers pose a continuous and unrelenting threat to organizations. Industry and academic researchers alike can benefit from a greater understanding of how hackers engage in criminal behavior. A limiting factor of hacker research is the inability to verify that self-proclaimed hackers participating in research actually possess their purported knowledge and skills. This paper presents current work in developing and validating a conceptual-expertise based tool that can be used to discriminate between novice and expert hackers. The implications of this work are promising since behavioral information systems researchers operating in the information security space will directly benefit from the validation of this tool.


hacker ability, conceptual expertise, skill measurement

Full Text:



Verizon, “2012 data breach investigations report,” 2012.

D. Dey, A. Lahiri, and G. Zhang, “Hacker behavior, network effects, and the security software market,” J. Manag. Inf. Syst., vol. 29, no. 2, pp. 77–108, Oct. 2012.

M. A. Mahmood, M. Siponen, D. Straub, H. R. Rao, and T. S. Raghu, “Moving toward black hat research in Information Systems security: An editorial introduction to the special issue,” MIS Q., vol. 34, no. 3, pp. 431–433, 2010.

Z. Xu, Q. Hu, and C. Zhang, “Why computer talents become computer hackers,” Commun. ACM, vol. 56, no. 4, p. 64, Apr. 2013.

J. S. Giboney, A. Durcikova, and R. W. Zmud, “What motivates hackers? Insights from the Awareness-Motivation-Capability Framework and the General Theory of Crime,” in Dewald Roode Information Security Research Workshop, 2013, pp. 1–40.

M. K. Rogers, “A two-dimensional circumplex approach to the development of a hacker taxonomy,” Digit. Investig., vol. 3, no. 2, pp. 97–102, Jun. 2006.

R. Chiesa and S. Ducci, Profiling Hackers: The Science of Criminal Profiling as Applied to the World of Hacking. Boca Raton, FL: Auerbach Publications, 2009.

A. E. Voiskounsky and O. V Smyslova, “Flow-based model of computer hackers’ motivation.,” CyberPsychology Behav., vol. 6, no. 2, pp. 171–180, Apr. 2003.

S. B. Mackenzie, P. M. Podsakoff, and N. P. Podsakoff, “Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques,” MIS Q., vol. 35, no. 2, pp. 293–334, 2011.

M. T. H. Chi and P. J. Feltovich, “Categorization and representation of physics problems by experts and novices,” Cogn. Sci., vol. 5, no. 2, pp. 121–152, 1981.

J. I. Smith, E. D. Combs, P. H. Nagami, V. M. Alto, H. G. Goh, M. A. A. Gourdet, C. M. Hough, A. E. Nickell, A. G. Peer, J. D. Coley, and K. D. Tanner, “Development of the Biology Card Sorting Task to Measure Conceptual Expertise in Biology,” CBE-Life Sci. Educ., vol. 12, no. 4, pp. 628–644, Dec. 2013.

M. T. H. Chi, “Laboratory methods for assessing experts’ and novices' knowledge,” in The Cambridge Handbook of Expertise and Expert Performance, K. A. Ericsson, N. Charness,

Feltovich, and R. R. Hoffman, Eds. Cambridge University Press, 2006, pp. 167–184.

D. R. Compeau and C. A. Higgins, “Computer self-efficacy: Development of a measure and initial test,” MIS Q., vol. 19, no. 2, pp. 189–211, 1995.

P. H. Cheney and R. R. Nelson, “A tool for measuring and analyzing end user computing abilities,” Inf. Process. Manag., vol. 24, no. 2, pp. 199–203, 1988.

T. J. Holt, “Subcultural evolution? Examining the influence of on- and off-line experiences on deviant subcultures,” Deviant Behav., vol. 28, no. 2, pp. 171–198, Feb. 2007.

A. Bandura, “Social cognitive theory: An agentic perspective,” Annu. Rev. Psychol., vol. 52, no. 1, pp. 1–26, 2001.

A. Bandura, “Self-efficacy mechanism in human agency,” Am. Psychol., vol. 37, no. 2, pp. 122–147, 1982.

A. Newell and H. A. Simon, Human Problem Solving. Englewood Cliffs, NJ: Prentice Hall, 1972.

A. Chandra and R. Krovi, “Representational congruence and information retrieval: Towards an extended model of cognitive fit,” Decis. Support Syst., vol. 25, pp. 271–288, 1999.

H. A. Simon and J. R. Hayes, “Understanding written problem instructions.,” in Knowledge and Cognition, L. W. Gregg, Ed. Potomac, MD: Lawrence Erlbaum Associates, 1974, pp. 165–200.

J. R. Hayes and H. A. Simon, “Psychological differences among problem isomorphs,” in Cognitive Theory, 2nd ed., J. N. Castellan Jr., D. B. Pisoni, and G. R. Potts, Eds. Hillsdale, NJ: Lawrence Erlbaum Associates, 1977, pp. 21–41.

M. Weiser and J. Shertz, “Programming problem representation in novice and expert programmers,” Int. J. Man. Mach. Stud., vol. 19, no. 4, pp. 391–398, 1983.

C. P. Pfleeger and S. L. Pfleeger, Security in Computing, 4th ed. Upper Saddle River, NJ, USA: Prentice Hall, 2006.

M. T. Goodrich and R. Tamassia, Introduction to Computer Security. Boston, MA: Pearson Education, Inc., 2011.

T. Jordan, “Mapping Hacktivism: Mass Virtual Direct Action (MVDA), Individual Virtual Direct Action (IVDA) And Cyber-wars,” Comput. Fraud Secur., vol. 4, no. 1, pp. 8–11, 2001.

T. Jordan and P. Taylor, “A sociology of hackers,” Sociol. Rev., vol. 46, no. 4, pp. 757–780, Nov. 1998.

S. M. Furnell and M. J. Warren, “Computer hacking and cyber terrorism: The real threats in the new millennium?,” Comput. Secur., vol. 18, no. 1, pp. 28–34, 1999.

O. Turgeman-Goldschmidt, “Hackers’ Accounts: Hacking as a Social Entertainment,” Soc. Sci. Comput. Rev., vol. 23, no. 1, pp. 8–23, Feb. 2005.

V. Mookerjee, R. Mookerjee, A. Bensoussan, and W. T. Yue, “When hackers talk: Managing information security under variable attack rates and knowledge dissemination,” Inf. Syst. Res., vol. 22, no. 3, pp. 606–623, 2011.

D. P. Twitchell, “Augmenting detection of social engineering attacks using deception detection technology,” in International Conference on i-Warfare and Security, 2006.

R. E. Bell, “The prosecution of computer crime,” J. Financ. Crime, vol. 9, no. 4, pp. 308–325, 2002.

H. Liang and Y. Xue, “Avoidance of information technology threats: A theoretical perspective,” MIS Q., vol. 33, no. 1, pp. 71–90, 2009.

G. B. Magklaras and S. M. Furnell, “Insider threat prediction tool: Evaluating the probability of IT misuse,” Comput. Secur., vol. 21, no. 1, pp. 62–73, 2002.

Symantec Corporation, “Internet Security Threat Report,” Mountain View, California, 2013.

L. Holmlund, D. Mucisko, K. Kimberland, and J. Freyre, “2010 cybersecurity watch survey: Cybercrime increasing faster than some company defenses,” 2010.

CyberEdge Group, “2014 Cyberthreat Defense Report,” 2014.

R. T. Wright and K. Marett, “The influence of experiential and dispositional factors in phishing: An empirical investigation of the deceived,” J. Manag. Inf. Syst., vol. 27, no. 1, pp. 273–303, 2010.

B. Parmar, “Protecting against spear-phishing,” Comput. Fraud Secur., vol. 2012, no. 1, pp. 8–11, Jan. 2012.

S. Goel and H. A. Shawky, “Estimating the market impact of security breach announcements on firm values,” Inf. Manag., vol. 46, no. 7, pp. 404–410, Oct. 2009.

R. Boyle and J. G. Proudfoot, Applied Information Security: A Hands-On Guide to Information Security Software, 2nd ed. New Jersey: Pearson, 2014.

F. N. Kerlinger and H. B. Lee, Foundations of Behavioral Research, 4th ed. New York: Cengage Learning, 1999.

D. W. Straub, M.-C. Boudreau, and D. Gefen, “Validation guidelines for IS positivist research,” Commun. Assoc. Inf. Syst., vol. 13, no. 1, pp. 380–427, 2004.

T. R. Hinkin and J. B. Tracey, “An analysis of variance approach to content validation,” Organ. Res. Methods, vol. 2, no. 2, pp. 175–186, 1999.


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

(c) 2006-2015 Association of Digital Forensics, Security and Law