WP1 – DIRECTION, MANAGEMENT AND DISSEMINATION OF THE PROJECT: Project management has as its main purpose the planning, monitoring and control of the activities and resources involved in the project.

• TASK 1.1: Project management and economic-administrative management
• TASK 1.2: Dissemination and communication
• TASK 1.3: Risk management and quality assurance

WP2 – THEORETICAL FRAMEWORK DEFINITION, HYPOTHESIS AND TECHNICAL REQUIREMENTS: The objective of this work package will be to gather the information necessary to circumscribe the theoretical framework within which the hypotheses of the investigation of this Project should be raised. In addition, there must be a clear and detailed list with the technical and functional requirements of the materials and methods to be used for each of the modules to be developed.

• TASK 2.1: Review of the state of the art. Definition of conceptual framework
• TASK 2.2: Identification of extrinsic and intrinsic factors to the person that modulate decision-making
• TASK 2.3: Statement of hypotheses
• TASK 2.4: Definition and analysis of functional requirements experimental platform.

WP3 – DESIGN OF EXPERIMENTAL PLATFORM FOR THE STUDY OF DECISION-MAKING AND SOCIAL COGNITION: The objective of this package is to develop the experimental platform that will allow to carry out the whole series of behavior analysis studies in situations of decision-making and social cognition framed in mixed realities.

• TASK 3.1: Design and implementation of graphic solutions
• TASK 3.2: Research and implementation of natural man-machine interfaces that allow user interaction within the scenarios with mixed realities proposed.
• TASK 3.3: Final logical development, both at the platform level and at the stage level.
• TASK 3.4: Design and development of internal layer to obtain metrics related to user behavior in each scenario (human behavior tracking: HBT).
• TASK 3.5: Synchronization and integration of physiological and behavioral signals collected with external equipment
• TASK 3.6: Platform management interface and programming experimental designs
• TASK 3.7: Validate final platform at the level of functionality and usability.

WP4 – EXPERIMENTAL EVALUATION: This is the core of the project. In it, a series of experiments must be designed and executed to help evaluate the behavior of users in the different situations posed. In this case, for each study carried out, a series of metrics will be calculated that will be statistically analyzed. Subsequently, it will try to apply more advanced techniques to try to classify, cluster and / or model the behavior of the user in the face of the experiences raised, focusing especially on decision-making and social cognition.

• TASK 4.1: Design and implementation of experimental protocols
• TASK 4.2: Execution of experimental passes
• TASK 4.3: Intracranial records
• TASK 4.4: Pretreatment of data and extraction of final metrics. Data visualization
• TASK 4.5: Statistical analysis and application of classification, clustering and modeling techniques

WP5 – PRACTICAL VALIDATION, CONCLUSIONS AND FINAL DISCUSSION: Thanks to the analyzed results of the experimental studies, an attempt will be made to apply the knowledge generated in the development of some practical examples that, as a use case, show the potential positive use of mixed interfaces in the situations raised related to decision making and social cognition. After the application of these practical examples, we will be in a position to establish a series of conclusions and a final discussion about the change in the decision-making paradigm that is going to involve the emergence of mixed realities and how this impact can be modulated so that translate positive.

• TASK 5.1: Design and implementation of practical validation examples
• TASK 5.2: Analysis of results, practical examples
• TASK 5.3: Conclusions of the experimental and validation phase. Final discussion

WP6 – SCIENTIFIC DISSEMINATION: The objective of this package is to scientifically disseminate the project and establish relationships with other research and innovation centers, as well as with potential final clients, which will ensure the continuity of the research line proposed by the project and will ensure its profitability with the development of future R + D + i actions.

• TASK 6.1: Carrying out workshops / seminars to disseminate the results and knowledge extracted from the project
• TASK 6.2: Preparation of national and international conference presentations
• TASK 6.3: Writing scientific articles in journals of national and international relevance
• TASK 6.4: Direction of thesis and master’s thesis within the Project
• TASK 6.5: Exchange of researchers between external groups

UPV

Mariano Alcañiz: University professor with 3 six-year terms recognized by the CNEAI and director of LabLENI. Principal investigator of the project. He has participated and participates as principal investigator in 2 H2020 projects and in more than 10 European projects in the FP4-FP7 calls. Holder of 5 chairs with companies and several contracts with major companies. He is also the principal investigator of several R&D projects of basic and competitive science and collaborative with companies. For all this, and for his technical qualities (more than 300 publications in the ICT area), prof. Alcañiz will be the person responsible for the scientific-technical and economic-financial coordination of the project.

Manuel Contero: University professor with 4 six-year terms recognized by the CNEAI and professor at the UPV. Prof. Contero has among his merits the direction of various doctoral theses in the area, the direction in the last 8 years of three national plans in the field of design and education applying immersive technologies and a multitude of publications in high-impact journals and national and international congresses. It is also listed as the holder of several patents.

José Antonio Diego: Industrial Engineer and Professor of the Higher Technical School of Design Engineering of the Polytechnic University of Valencia with 4 six-year terms recognized by the CNEAI. He received his doctorate in 2006 for his work on the application of Artificial Intelligence to Engineering Processes. Since then he has focused his research on the application of these techniques to human-oriented design, both in the field of industrial design and ergonomics, lines in which he continues his investigations. He has directed various doctoral theses and numerous final degree projects on these topics. He has been a principal investigator in several competitive research projects, holds an international patent, and has published multiple articles in high-impact scientific journals and at international conferences. He has been responsible for technology transfer agreements and software development currently in operation and is co-director of the on-line platform Ergonautas with more than 200,000 registered users worldwide.

Jorge Alcaide: Industrial Engineer and Professor of the Higher Technical School of Design Engineering of the Polytechnic University of Valencia. With 3 six-year research periods recognized by the CNEAI. Doctorate in 2004 with a work on innovation processes in product development. His research focuses on the application of new technologies in innovation in product development, analysis of user-product interaction and creativity. He has supervised 4 doctoral theses and more than 70 final degree projects and master’s thesis on these topics. In addition, he has participated in several research projects, being IP in two of them of the Call for proposals-research of MINECO and has published various articles in scientific journals, as well as in international conferences.

UVEG

Enrique Bigné: University Professor with 5 six-year terms recognized by the CNEAI, and one six-year research transfer, he is a professor in the Department of Marketing and Market Research at the University of Valencia. His main line of research is communication, tourism and neuromarketing. He directs the research group on Digital Marketing and Communications. He has led several projects on digital communications in social networks, two of them from the National R&D Plan (2015-2017; 2019-2022), has directed 30 doctoral theses and 90 indexed publications and participated in 120 in national and international conferences. He has been visiting scholar at the universities of Maryland, Berkeley, and Regent’s London. He is editor of the European Journal of Management & Business Economics and belongs to a dozen editorial boards of magazines and is associate editor of Frontiers in Psychology, Journal of Modeling in Management, Journal of Global Marketing and PASOS, Cultural Tourism magazine.

Carla Ruiz: University holder at the University of Valencia with 2 six-year terms recognized by the CNEAI, she has an H index of 29 in Google scholar and 18 in WOS. Her main lines of research are online consumer behavior, neuromarketing and tourism, being since 2017 the director of the research group in Marketing and Electronic Commerce of the University of Valencia (UV2017-367). She has led 5 projects on digital communications in social networks and mobile marketing, two of them being the National R&D Plan (2015-2017) and (2020-ongoing), and she is currently participating in a H2020 project on Neuromarketing and Mixed Reality. She has supervised 6 doctoral theses. Its international visibility is evidenced in the 6 research stays carried out in the USA and UK and in the co-authorship of 90 articles, 32 of them indexed in Journal Citation Reports, 139 contributions in conferences (102 international), and the edition of 2 international research books. She is Associate Editor of Electronic Commerce Research and Applications journal and serves on the editorial board of Online Information Review and Journal of Product and Brand Management; the 3 journals are indexed in SSCI.

Isabel Sánchez: University holder with 2 six-year terms recognized by the CNEAI and professor at the University of Valencia. Her main lines of research focus on consumer behavior in tourism, new technologies and service marketing. She has participated as a researcher in a national project of the National R + D + i Plan, related to digital communication in tourism. She is co-author of more than 35 publications in indexed journals and of these, 9 publications in Journal Citation Reports. Highlight the article Bigné, E .; Sánchez, I. and Sánchez, J. (2001): “Tourism image, evaluation variables and after purchase behavior: inter-relationship”. Tourism Management, 22: 607 – 616., which has received 2378 citations to date. Likewise, she has participated in 43 prestigious national and international congresses. She is a regular reviewer of journals indexed in the Journal citation reports such as Tourism Management, Journal of Travel and Tourism Marketing or Journal of Business Research, among others.

Luisa Andreu: University holder with 3 six-year terms recognized by the CNEAI and professor at the University of Valencia. Her main lines of research are focused on consumer behavior in tourism and communication in social networks. She is a member of the Association of Scientific Experts in Tourism (AECIT) and of the Research Institute in Psychology of Human Resources, Organizational Development and Quality of Work Life (IDOCAL) of the University of Valencia. She has participated as a researcher in four national projects of the National R + D + i Plan, with topics related to digital communication in tourism. He has directed 3 doctoral theses; co-author of more than 70 publications in indexed journals and of these, 16 publications in Social Science Citation Index; and has participated in 120 national and international congresses. She belongs to the Scientific Committee of journals indexed in SSCI: Journal of Destination Marketing & Management (JDMM) and Journal of Hospitality & Tourism Research.

Rafael Currás: University holder with two six-year terms recognized by the CNEAI and professor at the University of Valencia. Two of his main lines of research are online consumer behavior and tourism marketing. He has been a member of 8 competitive research projects, two of them from the National R&D Plan, and principal investigator of a Project for Emerging Groups of the Department. He has published 54 indexed articles (15 of them in the Journal Citation Reports) and presented 65 papers and papers at national and international conferences. He has directed five doctoral theses. He collaborates as a regular reviewer in several journals of the Journal Citation Reports, such as the Journal of Business Ethics, the European Journal of Marketing or Sustainable Production and Consumption.

UCDAVIS

Ignacio Sáez Martínez: Professor in the Department of Neurosurgery at the University of California, Davis. In his training, he has combined the use of neuroeconomic tasks and models with intracranial records in patients undergoing invasive neurosurgery, collaborating or leading the first projects for invasive measurement of neuronal activity in the field of decision-making. He has extensive international experience and is currently PI in professional development (K01) and new technologies (R21) projects of the National Institutes of Health (NIH) in the USA.

Experimental platform for the study of decision taking and social cognition:  The main objective of the platform is to evaluate the cognitive functions and decision-making processes of users in different environments simulated using immersive Virtual Reality. Moreover, the system allow to simulate environments up to 6×10 m2 where the subjects can navigate and interact, providing a high sense of presence. Meanwhile, the system tracks the behavioural and neuro-physiological responses of the subjects, which will allow to measure and analyse through standardized measures the states and traits of the users, as well as the characteristics of the content displayed, providing valuable information about the cognitive functions and decision-making processes of users. Therefore, this will allow: 1) to be able to characterize the subject, 2) to measure his state during the experience and 3) to establish the influence of certain mediators on his behaviour. This provides us with the possibility of obtaining implicit responses through the analysis of behaviour related to the action of the subject within the virtual environment and not through the direct measurement of physiological variables of the subject that may or may not be related to an action. Some of the current experiment analyses the following traits and states:

Demographics: This study aims to classify consumers inside a VR environment based on demographic features (age and gender). As a stimulus, virtual environment consisted of a store and multiple purchase tasks. A sample of 60 participants balanced in gender (30 females and 30 males) and age (from 18 to 36 years) were recruited for conducting the experiment. A set of demographic questionnaires along with data from virtual reality hardware was used as measures for the study.

BFI: This study aims to classify consumers based on the Big Five personality domains during a virtual purchase session using a head-mounted display. A virtual store was used as the stimulus. The personality recognition was ascertained using behavioural measures received from VR hardware, including eye-tracking, navigation, posture and interaction. Responses from 60 participants were collected while performing free and forced tasks in a virtual hypermarket.

Impulsivity: This study aims to recognize impulsivity using behavioural patterns. For this goal, 60 subjects performed three tasks—one exploration task and two planned tasks—in a virtual market (stimulus). Four non-invasive signals (eye-tracking, navigation, posture, and interactions), which are available in commercial VR devices were recorded, and a set of features were extracted and categorized into zonal, general, kinematic, temporal, and spatial types.

Consumer dimensions prediction using behavioural patterns in VR: This study aims to predict the consumer dimensions like confusion, anxiety, enjoyment, and risk perception using behavioural patterns through the use of virtual reality shopping environment as stimuli. A total of 80 participants (40 females) within the age group 18-35 years were selected for the experiment. This study uses measure from virtual reality hardware including eye-tracking, navigation, posture, and interactions.

Contextual and Social mediators: This study aims to assess the influence of contextual (technical vs emotional information/advertisement, change packaging/colour) and social (proximity of seller to the shopper, personalized advices) mediators in a virtual reality shopping environment. Responses from 100 participants (tentative) of balanced gender and age are to be collected for this study. Measures including eye-tracking, navigation, posture, and interaction as well as electrophysiological measures like GSR and EEG are to be implemented for a small group.

Assessment of risk taking in decision-making: The objective of the study is to analyze which variables influence decision-making in the face of risk and to measure these variables through implicit measurements using virtual reality environments. To do this, users will be asked to carry out a series of tasks within a virtual environment designed and developed in order to measure variables related to risk behavior. Behavioral and physiological measures and questionnaires will be used. 100 healthy subjects will participate in this study.

Virtual-Reality behavioral biomarkers – body movements: Recent advances in neuroscience, technology and data science allow to disentangle specific unconscious cognition processes by the measurement of behavioural biomarkers. In particular in this study, autism spectrum disorder was distinguished from neurotypical development through the recording and detection of children’s body movements in virtual reality. Autism diagnosis based on this cutting-edge method achieved 89.36% of accuracy.

Virtual-Reality sensory processing – electrodermal activity: Behavioural biomarkers represent implicit objective processes that allow to characterize certain unconscious cognition processes. Innovative data science techniques as machine learning algorithms are able to make predictions in this field by the analysis of specific implicit processes in humans. In particular, this study was able to detect autism spectrum disorder in children experiencing two virtual environments while electrodermal activity was recorded. Accuracy in diagnosis achieved the 90.3%.

Augmented shopping: The objective of the study is to analyze how the use of a goggle-type augmented reality solution influences the sales experience. For this, users must carry out a series of tasks related to the purchase of products using a Hololens 2 device. Behavioral measures and questionnaires will be used. In this study 50 subjects will participate (25 control condition and 25 experimental condition).

Body odours and WTP: This study uses olfactory priming to differentiate whether body emotional odours (happiness, fear, and neutral) have an influence over the willingness to pay (WTP) in various products’ purchase (e.g food, technologies, alcohol, tickets flights, etc..). To do this, users must answer a series of questions related to the WTP to certain products, while they are exposed to a series of body odors. Behavioral measures and questionnaires will be collected. In this study, 60 subjects (30 control conditions and 30 experimental conditions) will participate.

Entrevista de trabajo virtual: The objective of this study is to compare the feeling of presence in a job interview between 2 different reality spaces (virtual reality space and reality space) based on the comparison of implicit behaviors. To do this, users will conduct a job interview with a professional interviewer, to whom they must verbally answer a series of questions. Behavioral (eye tracking and voice), physiological and questionnaire measures will be analyzed. In this study, 25 subjects will participate, who will go through both conditions in counterbalanced order.

Hotel choice: The objective of this study is to analyze the choice order when the client makes decisions regarding the reservation of a hotel room and the decision to visit a tourist attraction; analyze visual attention to the attributes (information cues) that the client takes into consideration; and analyze the user experience in the decision-making process of the hotel / tourist attraction reservation. For this, a website with photos will be used versus virtual reality in 360 video in the context of a hotel / tourist attraction, and an assistant sales agent will be included in the web and virtual reality conditions, in order to compare with the web and virtual reality without the assistant.

Build your home: The objective of the study is to analyze how the use of a glasses-type augmented reality solution influences the sales experience in the field of purchasing household equipment items. For this, users must carry out a series of tasks related to the purchase of products (a computer and a chair) using a Hololens 2 device. Behavioral measures and questionnaires will be used.

Intracranial records and presence: This study is based on the presence model of Skarbez et al. (2017), in which it is established that there are three components of presence: immersion, company and coherence. This study will work with patients from the epilepsy unit of the La Fe hospital in Valencia, who have intracranial implants to collect brain signals. A virtual environment will be developed with different variants of high / low presence with respect to each of these three components and it will be studied if there are patterns of brain response and if there are differences in functional connectivity. The following measures will be taken: user interaction with the virtual environment, eye tracking during the experience, brain records during the experience, and presence questionnaires after the experience. The study is planned with a sample of 10 patients.

de-Juan-Ripoll, C., Soler-Domínguez, J. L., Chicchi Giglioli, I. A., Contero, M., & Alcañiz, M. (2020). The spheres & shield maze task: a virtual reality serious game for the assessment of risk taking in decision making. Cyberpsychology, Behavior, and Social Networking, 23(11), 773-781.

Alcañiz Raya, M., Marín-Morales, J., Minissi, M. E., Teruel Garcia, G., Abad, L., & Chicchi Giglioli, I. A. (2020). Machine learning and virtual reality on body movements’ behaviors to classify children with autism spectrum disorder. Journal of clinical medicine, 9(5), 1260.

Alcañiz Raya, M., Chicchi Giglioli, I. A., Marín-Morales, J., Higuera-Trujillo, J. L., Olmos, E., Minissi, M. E., … & Abad, L. (2020). Application of Supervised Machine Learning for Behavioral Biomarkers of Autism Spectrum Disorder Based on Electrodermal Activity and Virtual Reality. Frontiers in human neuroscience, 14, 90.

de-Juan-Ripoll, C., Llanes-Jurado, J., Giglioli, I. A. C., Marín-Morales, J., & Alcañiz, M. (2021). An Immersive Virtual Reality Game for Predicting Risk Taking through the Use of Implicit Measures. Applied Sciences, 11(2), 825.

Ausin Azofra, José Manuel, et al. “Do you see what I see? Effectiveness of 360-degree vs. 2D video ads using a neuroscience approach.” Frontiers in Psychology 12 (2021): 256.