Evaluating Crowdsourcing Applications with Map-Based Storytelling Capabilities in Cultural Heritage

Abstract

Crowdsourcing applications that integrate storytelling and geotagging capabilities offer new avenues for engaging the public in cultural heritage. However, standardised evaluation frameworks are lacking. This paper presents an applied evaluation methodology involving the analysis of relevant web-based tools. Towards this goal, this paper presents the development of crowdsourcing applications using, as a case study, the collection of myths and legends associated with the monumental heritage site of the Palace of the Grand Master of the Knights of Rhodes in Greece. Additionally, the paper presents an evaluation conducted through a criteria-based approach and user-based survey. The study reviews the concepts of crowdsourcing and digital storytelling within digital heritage, along with current concepts of living heritage and folklore, and examines relevant initiatives. The evaluation follows a four-stage methodology: (i) initial web-based tool selection based on the minimum requirements of web compatibility, crowdsourced data display, and map-based storytelling capability; (ii) application development using the selected web-based tools; (iii) a five-criteria assessment, based on the factors of open access, usability/tool support, participatory content/story creation, metrics provision and metadata model usage; and (iv) a crowd-based survey, indicating the most effective option. Findings from 100 respondents reveal limited exposure to participatory storytelling applications but interest in contributing content. Social media and influential figures serve as key channels for promoting crowdsourcing open calls. The results highlight gaps in understanding user expectations and perceptions, suggesting future research for gaining insights into engagement rates.

1. Introduction

This study describes the design and evaluation plan of a web application that harnesses the collective memory of the public in relation to a cultural heritage site. The application aimed to enhance the experience of the monumental heritage site of the Palace of the Grand Master of the Knights of Rhodes in Greece by collecting myths, legends and other anecdotal information related to local history, art history, literature, living heritage and folklore, with the contribution of local citizens and visitors through crowdsourcing, presented in a participatory storytelling and map-based format. For this purpose, a number of web tools were selected that support the development of cultural heritage applications. Each web tool was used to design and create an application for crowdsourcing myths, legends and other cultural heritage content about the Palace of the Grand Master of the Knights of Rhodes. The evaluation plan assessed the development of the web applications (i) based on five criteria, i.e., free and online access, usability/tool support, participatory content/map-based story creation, provision for metrics in the form of usage statistics and metadata model usage, and (ii) based on a public opinion survey in the form of an online questionnaire. This study analysed the survey results, including demographic, educational and other variables. Moreover, the study aimed to bring a better understanding of cultural heritage documentation by participatory digital means, scoping current initiatives that combine crowdsourcing, storytelling and geotagging.

The research questions follow the study’s workflow. What web tools are available to create a digital identity for local history, living heritage and folklore, with a case study application of the Palace of the Grand Master of the Knights of Rhodes? How can relevant content be recorded in participatory ways and presented digitally? Which available crowdsourcing tool is most suitable for recording and presenting the local history, living heritage and folklore of the monument? Is society willing to contribute to the recording and promotion of the heritage associated with the monument? The study inquires about aspects of human–computer interaction (HCI), aiming at measurable insights in developing and evaluating crowdsourcing applications with storytelling and geotagging capabilities in the cultural heritage context. What set of criteria can evaluate crowdsourcing web tools with these capabilities? Through the methodology formation, five criteria are tailored for evaluating the web applications, including the degree of open access as informed by established principles, definitions, and typologies. What variables can provide insights into usability, user experience and participation in the crowd-based survey? Which platforms and agencies achieve more visibility for inviting participation in crowdsourcing projects? The design of the crowd-based survey and the collected responses are analysed, bringing further insights into HCI aspects in the digital heritage context. In addition, the study questions whether current web tools deliver the digital documentation of cultural heritage adequately with a focus on elements of living heritage and folklore, discussing limitations and prospects.

The study seeks to enhance public engagement with the monumental heritage site, the Palace of the Grand Master of the Knights of Rhodes, gathering relevant cultural content through user contribution. The focus is on collecting stories of local history, living heritage, folklore, art history and literature with anecdotal value. The key objectives are, on the one hand, to use current web tools with crowdsourcing, digital storytelling and geotagging capabilities in order to examine indicators of HCI, such as user experience, usability and crowdsourcing, and on the other hand, to create and apply an evaluation methodology that is measurable and reusable for crowdsourcing applications in a cultural heritage context. To this end, a hybrid four-stage methodology is set: (i) relevant web tools are assessed, (ii) the web tools are used for developing applications, (iii) the applications are evaluated and filtered down by a set of five criteria and (iv) the most effective option is indicated through a crowd-based survey. The concepts of crowdsourcing and digital storytelling are briefly examined in the context of cultural heritage.

The study aims to contribute to the evaluation efforts in the fields of digital heritage and crowdsourcing by providing an applied case of a metrics-based evaluation plan for assessing crowdsourcing applications that display map-based storytelling capabilities. By providing a two-step evaluation approach, which includes a criteria-based evaluation and a public (user-based) survey assessment of the developed application, the study aims to provide a resource in the assessment practices that are limited in the field. The evaluation approach includes a comprehensive description of all four stages, from selecting the web tools for developing the application (first stage) to the design and creation of the applications (second stage), a two-step evaluation approach that encompasses a set of five predetermined criteria (third stage) and a crowd-based survey (fourth stage).

1.1. Public Participation and Crowdsourcing in Cultural Heritage

Digital transformation in cultural heritage (CH) is increasingly led towards participatory models that actively engage users in content creation and curation. This has been encouraged by the “Culture 3.0” paradigm for active cultural participation, which describes the shift of the public from passively absorbing cultural experiences to co-creating the “source code” that generates cultural meaning [1]. European and international strategies prompt public and democratic participation in heritage governance, such as the prominent Council of Europe Framework Convention on the Value of Cultural Heritage for Society (Faro Convention, 2005 [2]), which represented an early effort in promoting public participation with an emphasis on the involvement of “heritage communities” in the process, defined as people who value aspects of cultural heritage which they wish to sustain and transmit [3,4].

The advancement of participatory digital tools and infrastructures in CH promotes democratic participation and enhances CH content with multiple perspectives. To this end, the concept of crowdsourcing is understood in the context of CH as “a form of digitally-enabled participation that promises deeper, more engaged relationships with the public via meaningful tasks with cultural heritage collections” [5]. Broadly defined, crowdsourcing is engaging motivated users in knowledge building through tasks [6], and is regarded as an online, distributed problem-solving and production model that leverages the collective intelligence of online communities for specific purposes [7]. Crowdsourcing was initially described within business settings as actions previously performed by employees and now outsourced to an unspecified (and generally large) network of people. Crowdsourcing solutions can be offered in various fields and settings, including education and academia, in which case the term scientific crowdsourcing is also used, describing volunteers participating in crowdsourced scientific research [8]. Tasks typically involve providing resources, time and concentration (e.g., sorting images), funding (e.g., donating money to develop an initiative), geographical distribution (e.g., observations in large areas), extended access (e.g., providing access to private property such as house yards), ideas and knowledge (e.g., developing new designs or code). Crowdsourcing initiatives rely on open invitations to engage a wide network of potential contributors. Common challenges in crowdsourced data include the effective management of uncertainty and reliability in contributions from members of the public. The quality, veracity and credibility of volunteer contributions, especially in scholarly settings, may be often questioned [9], due to factors such as lack of expertise, sampling bias, and lack of standardisation in following the same processes among all volunteers. In addition, the extent of participation in crowdsourcing initiatives may vary, and challenges in attaining substantial participation or sustained engagement in several crowdsourcing projects have been observed, often leading to failed crowdsourcing infrastructures due to lack of user participation [10]. Nichesourcing, which targets niche groups (i.e., experts in a particular field), may effectively address these challenges by optimising the outcomes of crowdsourcing with substantial contributions by professionals who can engage with the content in meaningful and knowledgeable ways while enhancing the quality of the content, minimising risks and reinforcing engagement [11,12].

Crowdsourcing in the Social Sciences and Humanities (SSH) has rendered SSH scholarship more accessible to the public. However, crowdsourcing has been less deployed within SSH as opposed to fields such as life or natural sciences [13]. Other terms that describe crowdsourcing processes in the SSH include citizen humanities [14] and citizen social science [15]. Within the SSH, public participation has been particularly prevalent in the field of cultural heritage. Several policy documents and programmes have supported the development of participatory processes in cultural heritage in the past decades [2,16,17]. In this context, Dobreva et al. have examined the development and usage of digital infrastructures for crowdsourcing in digital cultural heritage [18]. Crowdsourcing field-specific tasks typically include transcribing, georeferencing, tagging, and co-curating. Digital infrastructures in the form of crowdsourcing platforms can be developed and maintained by GLAMs (Galleries, Libraries, Archives, and Museums), such as the platform of the National Library of Spain for the development of participatory projects. In addition, crowdsourcing platforms such as CrowdHeritage host campaigns by various institutions and projects, allowing users to improve the metadata quality of digital heritage content displayed on Europeana [19]. Generic and field-agnostic crowdsourcing web platforms such as Zooniverse can host heritage-related projects. Additionally, digital content management systems can display crowdsourcing functionalities. Other platforms, such as Spotteron, can support mobile-first initiatives with custom solutions tailored specifically for mobile accessibility.

1.2. Digital Heritage, Living Heritage and Folklore

Public participation can contribute substantially to a better and more representative understanding of cultural heritage, bringing in diverse narratives and collective experiences embedded within communities or groups that would otherwise get lost over time. This often goes beyond object-centred CH and material culture to more performative and event-based manifestations, which may fall within the fields of intangible cultural heritage and folklore. Intangible Cultural Heritage (ICH) has been ratified under the UNESCO Convention [16] to span mainly across five domains: oral traditions (including language), performing arts, social practices and rituals/festive events, knowledge and practices concerning nature and the universe, and traditional craftsmanship. Its key value lies in the intergenerational transmission of knowledge and skills. A gradual transition from the term intangible cultural heritage to Living Heritage (LH) is observed in response to critiques such as the problematic dichotomous view of tangible and intangible assets since “real world” cultural entities typically incorporate both elements to various proportions [20].

Folklore is a related field that often intersects with LH, however, unlike LH, which includes only elements currently performed and safeguarded by communities, folklore also encompasses historical manifestations that are no longer actively transmitted. Folklore has been promoted through the UNESCO recommendation on the Safeguarding of Traditional Culture and Folklore in 1989 [21] and other frameworks such as the Model Provisions for National Laws on the Protection of Expressions of Folklore against Illicit Exploitation and Other Prejudicial Actions [22]. However, concerns over the term “folklore” and its negative connotations have been raised within the field of anthropology, general social sciences and beyond [23,24]. Particularly for Indigenous communities and marginalised groups, there have been concerns regarding past questionable practices in folklore research where issues such as representation, agency, authorship, identity and ownership have not always been properly acknowledged and respected.

LH is further linked to digital heritage through its documentation, representation and preservation by digital means. Challenges in LH’s conceptualisation and digital documentation have been broadly noted [25,26,27], including multi-authorship in rights statements, non-standardised data models and high heterogeneity of data. LH has been largely interpreted on the basis of an object-centred approach in both traditional cataloguing practices and digital collections management. Although LH can also be connected to tangible artefacts, it often deviates from material culture and is more commonly expressed through event and process-based manifestations. LH multimodal manifestations can be challenging to document from an information perspective that transforms features of “real world” entities into formal models and symbol structures. The documentation of LH lacks a commonly accepted approach in the form of data standards or digital recording. Related approaches are supported by a highly diverse number of ICT tools, including experimental ontological frameworks [28], semantic-based intelligent systems [29] and web-based annotation tools for movement data [30].

1.3. Digital Storytelling

Digital storytelling, in the context of cultural heritage and beyond, refers to the use of digital tools and technologies to convey narratives and preserve stories, while digital storytelling draws from narrative theories (e.g., structuralism, folktale morphology), computer-mediated communication and the expanding digital narratives necessitated a radical rethinking of narrative theory [31] that is interactive, transmedia, participatory and inclusive (e.g., interactive fiction systems, fanfiction through online participatory processes, computer-assisted analysis of digitised texts). In digital storytelling, stories are presented using diverse digital media, including videos, interactive websites, and multimedia exhibits, to engage the public in exploring cultural heritage. A three-part categorisation scheme for classifying digital storytelling works has been proposed by Trichopoulos, including digital storytelling authoring tools, theoretic frameworks, and systems and applications [32]; (a) Authoring tools facilitate storytelling by automating content generation, (b) systems and applications enhance user experience by combining software, databases or other special technologies, and (c) theoretic frameworks may include methods, models and designs that apply digital storytelling in various fields.

Crowdsourcing is often combined with digital storytelling through users contributing their personal reflections and testimonies (in any format such as text, video, photo or audio), creating multimodal interactive narratives [32]. This way, the crowdsourced CH-related content comprises a diverse set of resources, often in the form of anecdotes, other individual observations and sharing ephemera. Various collaboration levels can be implemented in creating digital stories with public participation, from mere web contributions of texts and photos to collaborative cases where participants are highly involved. Participatory digital storytelling has been enhanced by integrating geotagging, geographical mapping, and georeferencing technologies.

Digital storytelling has been used as a research method to co-create place-based immersive digital stories with the participation of homeless people, with the most significant findings in the potential for the formation of co-creative relationships and empathy [33]. The field represents a dynamic and interdisciplinary approach to preserving and communicating cultural narratives, with new applications including the use of computational and emergent digital storytelling techniques that integrate advanced and innovative technologies, such as artificial intelligence, augmented reality, and data visualisation into the storytelling process [34,35]. Towards this direction, an innovative mapping methodology has been implemented in the fields of CH and tourism, utilising geotagged photos within a defined local region as “proxy tools” for creating multi-thematic tourism storyways, such as an intangible cultural heritage route, an industrial tourism route, nature observation and more [36].

1.4. Relevant Work

Web-based crowdsourcing initiatives and infrastructures for cultural heritage have been proliferating in the last decade, contributing to the documentation of existing cultural content and the collection of new cultural resources. A selection of international crowdsourcing initiatives for LH and folklore is reviewed below, with a focus on aspects of HCI; this includes the developed crowdsourcing infrastructure and the platform-user interaction. The selection is based on desktop research conducted in the context of this study and on additional studies [37,38,39]. The selected initiatives offer an indicative view of the digital tools and projects using crowdsourcing for cultural heritage encompassing aspects of LH and folklore, with varied degrees of storytelling and geotagging capabilities.

“Meitheal Dúchas.ie” [40] is a crowdsourcing initiative that aims at advancing the National Folklore Collection of Irish tradition through community transcription. The project invites users to transcribe Ireland’s oral tradition through approx. 750,000 pages of handwritten texts that include oral history, folktales and legends, riddles and proverbs, games, trades, crafts, and topographical information. The archive is part of the Schools’ Collection and has been recorded by over 50,000 schoolchildren between 1937 and 1939 who interviewed their parents, grandparents and neighbours. Contributory engagement for participating users is performed through a custom digital infrastructure, which includes a user interface for volunteers to perform transcription processes in a structured setting that automatically generates transcription information as machine-readable data in XML format. Trying Optical Character Recognition (OCR) might not yield good results in the current state because of several hindering factors (e.g., a mix of Irish and English, different handwriting styles, unusual spelling and the use of both Roman and Gaelic scripts) [41]. The system is also heavily based on human resources for content assessment and approval through an editorial team that reviews new members’ input before publication.

Dodiom [42] and Dialettibot [43] are map-based chatbot crowdsourcing systems for the collective documentation of linguistic resources in Italy as intangible cultural heritage. The systems allow people to upload voice recordings of their spoken dialects and idiomatic expressions in Italy, locating their audio recordings on a geographical web map. The application software is developed on the Telegram online messaging service, allowing user-generated content to be uploaded and published after approval. The chatbot systems are created to advance scientific research in voice-based Natural Language Processing (NLP) through pooling resources from people who contribute by providing voice samples. In particular, Dodiom follows a gamified crowdsourcing approach by means of a Game With A Purpose (GWAP), implementing an asynchronous multiplayer game where the crowd is encouraged to create and annotate the language resource and rate annotations [44].

The Landauf, LandApp BW [45], is a participatory data collection project by the National Archive of the state of Baden-Würtemberg in Germany that aims to gather citizen contributions about landmarks and historical buildings in the area. The project is part of the state’s regional portal and information system, LEO-BW. The goal is to actively engage local communities and harness their collective knowledge to enrich the understanding of the region’s history and culture. The application has gathered over 1500 photos and information about places of interest, such as well-known sights, e.g., castles and churches, as well as lesser-known places and local highlights. It is developed as an independent smartphone app on the SPOTTERON mobile app and web platform [46], which supports the development of citizen science projects through customisable digital tools that enable public engagement. The application uses a localisation function for smartphones to locate monuments on a map. User-generated content can be uploaded through the customised smartphone app, located, categorised and described. The posts can be further liked, commented and statistically processed.

The Vault of Industrial Archives (V.I.D.A.) [47] aims to document the industrial heritage in Greece through participatory mapping. Although the project refers to built heritage and technical culture, it connects the more “tangible” form of heritage to collective memory, inviting members to contribute with testimonies complementary to the geotagging task. The initiative includes an open request for users to record the everyday culture of their places. It crowdsources records through a website form and displays the relevant findings through a Google map system for the geographical coordinates. The map contains different colour and icon tags, following a classification of over 20 colour-tagged categories and numerous subcategories that are identified through icons. The categorisation is based on established sources, such as the Statistical Classification of Economic Activities and historical catalogues of industrial heritage. The map displays the elements interactively, while the field coding applied the Dublin Core description standard and the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH).

2. Materials and Methods

Exploratory desktop research examined existing web tools that support heritage content crowdsourcing with storytelling and geotagging capabilities. The study focused on tools that primarily target humanities- or heritage-related content. Insights were obtained from a comprehensive review of approximately 200 initiatives within the broader field of crowdsourcing in CH, as documented by [37,38], and from supplementary findings grounded in the literature review. The web tool selection was based on three minimum requirements: (a) to function on a web-based environment (independent of device type, e.g., personal computer, tablet, mobile phone), (b) to incorporate and display crowdsourced data and (c) to support map-based (location-based) storytelling, for recording CH resources in the form of stories pinned by users on a geographic map. Four crowdsourcing storytelling web tools with geographical mapping capabilities were selected, i.e., Storymap Js, ArcGIS StoryMaps, Historypin and Omeka. Four web applications were correspondingly developed by using the selected web tools, with the aim to record CH crowdsourced content related to the myths and legends around the Palace of the Grand Master of the Knights of Rhodes.

The study further implemented a two-step evaluation approach of map-based crowdsourcing storytelling tools. The first evaluation step includes a set of five criteria for assessing the web tools, which are based on the related literature, the documentation of the tools, tools’ usage, and the technical capabilities for the development of map-based web applications that crowdsource cultural heritage. The evaluation criteria have been assessed and validated by the interdisciplinary scientific board of the Master in Cultural Informatics and Communication at the University of the Aegean [48] and by laboratory members of the Laboratory of Cultural Heritage Management. The five evaluation criteria include the search for web tools that provide: (1) free and online access, (2) usability/tool support, (3) participatory content/story creation, (4) provision for metrics in the form of usage statistics and (5) metadata model usage. The web tools Historypin and Omeka were selected based on the five criteria for further evaluation. The second evaluation step included the assessment of the two final selected tools through a crowd-based survey (i.e., user-based evaluation). Based on the replies of the participating public, Historypin was selected for displaying higher usability and user experience and enhanced capacity in crowdsourcing for the map-based documentation of cultural heritage resources.

2.1. The Web Tools

The study reviews open-source software and freeware with combined crowdsourcing, storytelling and geotagging capabilities, with the aim to (a) inform the web tool selection for the pilot application development that is capable of crowdsourcing CH content, (b) evaluate the application by designing and implementing it in a real setting, i.e., for the recording of myths and legends of the Palace of the Grand Master of the Knights of Rhodes, and (c) create a two-step evaluation approach of the web tools and applications, evaluating results obtained from the pilot through a set of predetermined criteria and a user survey. The selection of web tools is based on the need to support HCI on three levels: to incorporate crowdsourced data or enable user content contribution through a crowdsourcing process, to enable the geotagging of user content through a GIS-supporting web map, and to create a digital cultural route experience through a user interface with storytelling features. Thus, map-based crowdsourced storytelling, i.e., the presentation of user-generated stories in a narrative format, enables the castle visitors and users of the pilot web application to see the location connected to the recorded myth or legend.

StorymapJS (v.0.8.6) [49]: An open-source map-based storytelling tool that allows users to tell stories on the web, highlighting the locations of a series of events. Users can add interactive stories by creating a narrative map, the StoryMap, in two ways, i.e., (a) with maps, and (b) with photos, works of art, historic maps, and other image files. Similarly, TimelineJS is an open-source tool that enables users to build visually rich, interactive timelines. Both tools can integrate media from various sources, such as Twitter, Flickr, YouTube, Wikipedia, etc.

ArcGIS StoryMaps (no versioning available) [50]: A proprietary software that deploys Geographic Information System (GIS) and applies location-based analytics, allowing users to manage, visualise, and analyse geospatial data. The software is developed by Esri (Redlands, CA, USA), a global market leader in GIS software employing location intelligence, which refers to layering location-specific data on interactive maps or dashboards for revealing insights. ArcGIS includes a suite of products (cf., ArcGIS StoryMaps, ArcGIS Earth), which allow combining multiple data sources using scientific analytical tools on 2D, 3D, and 4D data. ArcGIS StoryMaps is promoted as a professional storytelling software, whereas StoryMaps is described as a personal storytelling solution that includes a free plan.

Historypin (no versioning available) [51]: A map-based app and community archiving platform, free to use but based on proprietary software. It was launched jointly by the non-profit organisation Shift and Google in 2010. It is a crowdsourcing tool that allows users to contribute content in the form of text, images, video and audio, pin it to a web map and narrate stories. The platform is used widely by cultural heritage institutions, civic organisations, councils and community groups.

Omeka (v.0.3) [52]: Free-to-use open-source content management system for creating and displaying dynamic online digital collections, developed at George Mason University. Omeka is useful for capturing digital files where users want to use rich metadata capabilities, scheduling, mapping, and crowdsourcing information. It supports libraries, museums, archives and scholarly collections, organising content through objects, collections and exhibits.

2.2. The Application

The “Myths and Legends of the Castle of the Knights of Rhodes” [53] is conceived as a web-based application designed to harness crowdsourcing and geotagging capabilities to preserve and promote CH content within digital storytelling settings. The key objective is to engage citizens in a participatory experience, encouraging content contribution related to the LH, folklore and local history of the monument. Based on the potential for extended use of the application, long-term goals include the availability of multilingual content and the extension of the application to other heritage sites. The design plan favours specifications based on user-friendly criteria, including easy contribution to participatory storytelling by adding or correcting material, offering open access and navigation to users, and curating content based on simple thematic categorisations. Correspondingly, the user access capabilities should include (i) administrator access rights, enabling content management, creation of additional applications within the technical framework and designating co-administrators; (ii) contributor rights, allowing users to insert information and share existing content to social media; and (iii) viewer capabilities, granting access for all digital visitors without login, with the right to share existing content on social media. Additionally, the utilities should include accessibility to social media for content sharing, content searchability and adaptability to mobile and tablet devices.

The navigation interface should comprise key sections facilitating user interaction and content management. The aim is to enhance user experience and promote active participation and content creation. The site map’s minimum sections include (i) the “about” section for providing a succinct summary of the application’s content, (ii) a search box, enabling users to find content through keywords, (iii) social media integration allowing users to publish and disseminate content seamlessly, (iv) a section explicitly inviting users to contribute by adding new content related to existing themes within the application, (v) capability to users to create their own applications and (vi) a direct communication section that is able to provide immediate assistance, catering to contributors, viewers, and administrators. Based on these common specifications for the application design, the selected web tools were developed to customise their functionalities, particularly focusing on user experience, usability, and participatory design.

2.3. Criteria-Based Evaluation of Tools

A set of selection criteria for evaluating the tools was set following a literature review and examination of the case studies. Drawing upon established assessment guidelines, models, methodologies and frameworks for crowdsourcing projects in CH, a heuristics evaluation was conducted to assess the usability and user experience of the web tool. The evaluation was supported by a group of experts with diverse backgrounds in HCI, digital heritage, and crowdsourcing practitioners. The evaluation process was supported by insights and a critical analysis of the usability and user experience aspects from various perspectives.

The evaluation criteria are based on five main pillars: (1) degree of open access, (2) usability/tool support, (3) participatory content/story creation, (4) provision of metrics and (5) metadata model usage. Several typologies and conceptual approaches for evaluating crowdsourcing methodologies and outcomes were considered, which have been presented in more detail in the literature review. These include the following:

1. Degree of open access: The web tool should facilitate open access through open data and open-access cultural content. This criterion is informed by the 9-factor typology for openness scope in participatory CH projects [38] and the “Open Definition” of the Open Knowledge Foundation for defining open data [54]. The tool should also utilise Free/Libre and Open-Source Software (FLOSS) principles related to the “Open-Source Definition” of the Open-Source Initiative [55], ensuring the open-source code and the availability to be modified, copied and distributed. In addition, tools released as freeware (cost-free) are equally evaluated, which may align with certain principles of openness, allowing free software usage, but without necessarily releasing the code as open source. Freemium software is also equally evaluated in the context of the study, referring to software that offers both a free basic plan and paid plans with additional features.

2. Usability/tool support: Usability and tool support refer to the tool’s usage by the creators for developing the web application. The study is informed by good practices, models and usability evaluations in the field of digital heritage for assessing the general navigation experience, particularly the ease of use of web tools [56,57]. Sub-criteria include (i) the presence of documentation for support and guidance in the application development, in the form of tutorials, community forums or help desks for seeking assistance and knowledge sharing, including technical expertise provided by the web tool developers and other professionals, contributing human resources on various levels and (ii) the technological and sensory options, which may include accessibility across devices and platforms, personalisation of the graphic layout with options for the input of multimedia elements (e.g., images, videos and audio).

3. Participatory content/story creation: The web tool should support crowdsourcing storytelling projects; this criterion is evaluating applications that support any form of story creation by engaging citizens in a crowdsourcing experience for content contribution, from more passive contributions to more engaged ones, drawing on several studies [5,58,59]. The effectiveness of user contributions is evaluated by assessing the web tool’s ability to communicate an open process for participation and support a user-friendly participatory content upload or contribution process. This process involves displaying uploaded user input in an interactive format and creating a narrative. This narrative may take the form of a storyway, a map-based storyline or any other textual displayed progression of the story with visual elements.

4. Provision of metrics: Metrics are added as an evaluation criterion for assisting in monitoring performance and assessing the impact of web-based projects in the form of quantitative measures such as user engagement level, number of contributions and project progress. Qualitative indicators can be included in the form of feedback mechanisms such as user commentaries. The criterion considers the metrics-based evaluation framework of the dimensions of success for participatory (citizen) science projects of Cox et al. [60]. It is also based on the prominent study of Ridge et al. for crowdsourcing in CH [5], which raises awareness over content production metrics that may lead to bias towards productivity, overlooking critical factors such as user engagement and transformative change of contributors or related communities.

5. Metadata model usage: This criterion draws from specifications and recommendations for metadata in CH as published by the National Documentation Centre in Greece (2020) and the corresponding data framework of the Europeana digital platform (Europeana Data Model) [61]. The web tool is assessed for the capacity to support (meta)data models that describe digital content uploaded by users. However, the support of the web tool for metadata specifically for LH is not assessed, as currently, there are no LH metadata standards or comprehensive frameworks for LH data documentation analysed in structural, technical, rights and descriptive metadata. Subsequently, no web tool supports metadata input tailored for LH digital content.

Based on the above criteria, the web tools for developing the application “Myths and Legends of the Palace of the Grand Master of the Knights of Rhodes” were selected and compared, as reported in Table 1.

StorymapJs [49] offers a user-friendly platform for free digital application development upon registration, without software installation. It offers a relatively limited range of technological and sensory options to create cultural routes. Map navigation is challenging; however, support and guidance are provided through instructional videos. The web tool is also compatible with mobile devices and tablets. Based on the evaluation criteria, StorymapJs does not support 3 out of 5 criteria, namely, participatory story creation, metrics and metadata insert. Nevertheless, the final sensory result is satisfactory, interactive and engaging for users, based on the user interface as illustrated in Figure 1 and Figure 2.

ArcGIS StoryMaps [50] by Esri also offers a user-friendly platform for free digital application development upon registration via social media or email, without the need for software installation. It provides many technological and sensory options, ensuring accessibility for users of many skill levels. The platform offers support and guidance in creating an application with instructional videos and texts. With Google Maps integration, users can design maps and create cultural routes. Generally, the design process of the application allows for considerable freedom of movement and creativity. The final layout is satisfactory and easy to navigate for end users; see Figure 3. The interface is also easy to navigate for creators, combining maps and embedded content as illustrated in the menu in Figure 4. However, based on the web tool selection criteria, ArcGIS StoryMaps by Esri does not support crowdsourcing and user collaboration to create content, lacks a metadata template and does not provide creators with visibility into application metrics (i.e., traffic).

Historypin [51] operates as freeware but without releasing its code as open-source. It is available for use at no cost to the user upon email or social media registration. While there are limitations regarding the number of uploaded multimedia, Historypin enables participatory content creation among users through map-based storytelling. It offers operational support and guidance for both developers and users, and it also provides visibility into metrics such as traffic, a feature absent in comparable applications. The layout is satisfactory, but it may not possess the same level of interactivity and aesthetic layout as the previous two applications (see Figure 5 and Figure 6). However, it includes metadata boxes for the description of each multimedia entry, although they are not customisable and not in structured formats such as Dublin Core. This may hinder the interoperability of content by means of seamless exchange of data with other platforms that adhere to established standards.

Omeka (Classic) [52] is an open-source content management system designed for showcasing digital collections. It is freely available for download and installation on a web server with a compatible database system. Once installed, users can access it remotely via a web browser with an internet connection. Omeka also offers a hosted service with a free plan that includes basic storage and options, along with various paid options. The tool provides technical support for both developers and users, including an active forum. There are no restrictions on the number of uploaded media files in the installed version, while adding geographic location data to items is not natively supported, free plugins can add geotagging capabilities. Omeka supports metadata schemas such as Dublin Core. Participatory storytelling with users is also facilitated through plugins, albeit in less interactive ways, allowing comments and contributions via forms. The related software Omeka S (Semantic) provides a more flexible and extensible architecture than its predecessor Omeka, including the freedom to tailor metadata schemas or add plugins that facilitate the import and export of metadata in standard formats (e.g., JSON-LD, CSV, or RDF); however, the layout remains not interactive and is less satisfactory than the previous tools (see Figure 7 and Figure 8).

The first phase of the evaluation process assessed the web tools against the five predetermined criteria (see Table 1), which were also utilised for developing the application “Myths and Legends of the Palace of the Grand Master of the Knights of Rhodes”. Following this evaluation, Historypin and Omeka were selected to proceed to the second phase of the evaluation process using the crowd-based evaluation approach, meeting all criteria for supporting participatory map-based storytelling.

2.4. User-Based Evaluation

Crowd-based (v. user-based) assessment was conducted as the second step of the evaluation approach of the selected web tools in the context of developing participatory map-based storytelling applications with cultural heritage content. In order to capture and analyse the experience of the general public in using the application “Myths and Legends of the Palace of the Grand Master of the Knights of Rhodes”, a structured questionnaire was devised. The questionnaire included close-ended and multiple-choice questions to enable a quantifiable analysis of tool usage dynamics and to inform the selection of the web tool that better meets the study’s objectives. The aim was to include the crowd-based dimension in the evaluation framework in a replicable and measurable way. The questionnaire fits the initial study’s research questions and objectives for designing an evaluation plan for HCI in geotagging crowdsourcing applications. The questionnaire included 12 questions, 3 with demographic information (level of education, gender and age group) and 9 with a primary focus on user perceptions, behaviours and engagement while tackling aspects of system usability, such as ease of use, user satisfaction and usability. The questionnaire gauged user preferences and challenges encountered while utilising the web applications. The key aspects assessed included the users’ digital literacy, tool adoption, communication strategies, awareness, content contribution, trust and participation. The questionnaire was designed and shared online through Google Forms. The sampling lasted one month (from 10 December 2021 to 10 January 2022). The participants were informed through social media advertising (Facebook, Instagram) and by email invitations.

3. Results

The crowd-based survey gathered 100 replies from users aged between 18 and 60. Based on the collected demographic data, the participants were fairly evenly split by gender, with women making up 55% of the respondents in a binary gender question (question 1, Q1) with the options of male and female. Question 2 (Q2) inquired about the age group, with individuals aged between 25 and 34 years comprising the majority of participants (55%), while 20% fell within the age range of 18–24 and an equivalent 20% within the age range of 35–54, with 5% belonging to the 55–64 age bracket. Question 3 (Q3) covered the education level of participants; the majority demonstrated a considerable level of education, comprising 78% of the total sample. This combines graduates of higher education (55.4%) and postgraduates (holders of a Master’s degree, 28.6%, and holders of a doctorate, 3.6%), as opposed to high school graduates (1.8%).

The results from the following nine questions aim to assess the user experience, behaviours and usability of the two web applications selected through the criteria-based evaluation. In question 4 (Q4), “how familiar are you with the use of web applications?”, nearly 8 out of 10 participants reported prior experience with web applications in general (Figure 9). The question addresses the concept of digital literacy among respondents, which aligns with the prevalent age group.

However, in question 5 (Q5), “have you used collaborative tools in the past?”, approximately 8 out of 10 respondents reported no prior experience with the use of participatory storytelling tools (Figure 10). The question further explores digital literacy in the context of collaboration.

Question 6 (Q6), “in which of the following ways are you informed about the existence of crowdsourcing applications?”, addressed the concepts of user awareness and communication strategies. Based on the responses, the most prominent source of information for crowdsourcing applications is social media (Figure 11).

Next, participants were asked to identify the tool with easier navigation and usage. Question 7 (Q7), “which of the two applications is more user-friendly?”, addressed the concepts of usability, ease of use and user satisfaction. While both web applications received overall positive feedback, Historypin was preferred over Omeka, with 73% of respondents indicating it as the more user-friendly option (Figure 12).

Question 8 (Q8), “did you contribute to the enrichment of the content of the application?”, assessed user engagement and participation, with only 35% of participants contributing content in both applications. This may be due to unfamiliarity with the content, the application type or other usability factors (Figure 13). In the Historypin web application, 22 stories and media were contributed in total, at least 8 of which were added by the engaged student who developed the application [53]. In the Omeka web application, five stories and media were contributed in total, all of which were added by the engaged student [64]. The responses indicate only the outcome, while further insights are needed to better understand the underlying causes of contribution and non-contribution.

Question 9 (Q9), “in which application was it easier to add more complete content using multimedia?”, addressed the concepts of functionality, usability and user satisfaction. Among the content contributors, there was a near parity in deciding on the application with the easier and more complete multimedia upload (6 out of 10 favoured Historypin). This may suggest that both web tools offer comparable usability and functionality in facilitating multimedia content integration (Figure 14).

Question 10 (Q10), “If you participated in the content contribution, why did you do it?”, relates to the aspects of motivation, participation and user engagement. Based on the responses, the primary motivations for content contribution were the participants’ interest in history and culture (54.8%) and the appeal of the application (38%). A smaller number of contributors cited their personal connection or special relationship with Rhodes as a motivating factor (19%). This question enabled the respondents to choose multiple answers (Figure 15).

Question 11 (Q11), “would you suggest a friend/traveller to use such an application on their journey to Rhodes?”, addressed the aspects of trust, recommendation and user satisfaction. A total of 98% responded positively to recommending this type of application to prospective travellers or friends, demonstrating that a part of the study’s objectives was met (Figure 16).

Finally, in question 12 (Q12), “will you continue to contribute with your knowledge to similar applications in the future?”, there was a split in whether participants would continue contributing to similar applications; nearly half were positive (55%), whereas 45% expressed uncertainty (Figure 17). More insights are needed to better understand the reasons behind the future intentions.

Based on the questionnaire results, the following findings emerge:

1. Participation was nearly evenly split between men and women, with women comprising a slight majority at 55%; in general, participants were primarily aged between 25 and 34 years (55%).

2. Despite their familiarity with other web application types (76.2%), a significant portion of participants (73.8%) had limited familiarity with (crowdsourcing) participatory storytelling applications.

3. Half of the respondents were interested in further contributing to similar applications.

4. Social media and opinion leaders were identified as the primary agencies for promoting such applications.

5. Based on the respondents, the application designed with Historypin emerged as a more user-friendly option, while both web tools displayed a nearly equal level of easiness in multimedia upload.

6. The majority of participants expressed intent to recommend the application further.

In addition, factors that influenced user choice and display qualitative and quantitative characteristics were further evaluated, including the following: (i) familiarity with web applications; (ii) understanding the benefits of crowdsourcing; (iii) the interactivity, speed and aesthetics of the application; (iv) familiarity with the process of adding content; and (v) the recommendation of the application for Rhodes’ promotion.

4. Discussion

The study’s key objectives were fulfilled to varying extents, including the implementation of the case study for the promotion of the monument “Palace of the Grand Master of the Knights of Rhodes” through the development of a crowdsourcing web application for the collection of local history, LH, and folklore content in the form of anecdotal stories, myths and legends. Based on the collected data from 100 respondents in the crowd-based evaluation, a significant interest among participants is evident in further contributing to and recommending such applications in the context of local cultural heritage promotion.

Control of the crowdsourced data in the Historypin web application has been performed in terms of assessing data quality and relevance and preventing malicious content upload. Based on the web tool’s designed features, user contributions of media, text and geotags, as well as user comments, have been reviewed by the moderator, the student developing the application, on a case-by-case basis. In addition, account verification is mandatory for content upload on the platform, which helps maintain a level of accountability and deters malicious activities. A higher number of user contributions may benefit from a more elaborated process for content control, combining strategies such as assigning content review to a team of moderators, allowing users to flag non-fit content, providing clear submission guidelines, using available automated filters for scanning malicious content and links to external websites, and inviting expert reviews for validating the content’s credibility.

The study’s evaluation methodology for crowdsourcing web applications with map-based and storytelling capabilities was based on promoting the monument heritage site “Palace of the Grand Master of the Knights of Rhodes” through crowdsourcing myths, legends, and anecdotal local stories. Although tailored to a single case study with a precise choice of tools (georeferencing, crowdsourcing, storytelling), the study provides a methodology framework and evaluation approach that is relevant and applicable to the broader fields of cultural heritage and SSH. This is corroborated by the scoped concepts of user experience, usability, and participatory design within the interdisciplinary field of digital humanities, including good practices and typologies in open access. However, the development of multiple applications before the actual evaluation process can be skipped when reusing the methodology. Instead, the evaluation can be anticipated at a preliminary stage before the actual implementation, which may save time and resources by ensuring that the most promising tools are considered. In the context of the study, the web applications were evaluated after their development as part of the educational workflow of the students’ learning experience to enhance their contextual understanding and their technical skills in web application development and support the practical application of the methodology in a case study setting.

The study’s findings indicate limited insights in explaining user expectations in relation to participation rates and potential impediments to content contribution (only 35% of participants were already engaged with content contribution). A limited understanding of the underlying factors influencing user perceptions, e.g., related to ease of use and value, is also prevalent. This further supports the need for more evaluation methodologies and nuanced measurable approaches tailored to cultural heritage crowdsourcing projects. In addition to assessing user demographics, perceptions and behaviours, user-based evaluation can be extended to assess the usability of the crowdsourcing system by incorporating a standardised questionnaire, such as the Computer System Usability Questionnaire (CSUQ). In the present questionnaire, a few questions overlapped with the CSUQ themes of overall user experience, ease of use and future engagement (i.e., Q4, Q6, partly Q8 and Q9). However, a more systematic approach could potentially bring more valuable insights for developers and designers to improve UX. In addition, integrating a more contextualised approach to the user experience evaluation with technologies in cultural heritage settings would further enhance insights, such as with the Cultural User Experience (CUX) methodologies, considering classification suggestions [65]. Moreover, evaluation dimensions particularly relevant to cultural heritage contexts should be identified, including a more holistic assessment that shifts thinking about value from simple output and scale to impact and wider benefits [5].

Crowdsourcing is still an unexplored branch in CH digital applications, although it is anticipated to have a significant impact on the future development of data aggregation infrastructures for cultural heritage after Linked Open Data, leaving behind other technologies such as blockchain and content tagging, based on a survey targeting experts, conducted in 2019 within the Europeana Common Culture Project and the Innovation and Networks Executive Agency (INEA) [66].

However, crowdsourcing also poses known challenges regarding data credibility, liability and validation mechanisms that may be prevalent in scientific crowdsourcing. Various approaches have been explored and applied to enhance trustworthiness in user input, including a review process by professionals before publishing the inserted data, such as in the Meitheal Dúchas.ie project, including peer review to assess the credibility of user tags and annotations through an upvoting or downvoting performed by users, such as in the CrowdHeritage platform [12]. In addition, combining crowdsourcing with storytelling can result in ethical and legal issues in the domain of IPR related to the nature of each SSH project. For example, when participants document stories, Intellectual Property Rights (IPRs) might arise, as their contribution (e.g., image, text) might come from a copyrighted source with restricted reuse permission.

While engaging diverse new audiences is a driving aspiration in crowdsourcing, the challenges to achieving substantial user participation and sustaining contribution interest have often been pointed out. Given the availability of robust supporting infrastructures and resources, an approach to address the issue of low participation numbers is to identify and invite already engaged members of the public, targeting audiences with a high interest in contributing. For example, the described relevant works “Landauf, LandApp BW”, “Meitheal Dúchas.ie” and “Dodiom and Dialettibot” successfully engaged crowd members of the local communities, including residents of a region and speakers of a local language and dialect, correspondingly. The Vault of Industrial Archives (V.I.D.A.) also made a successful call by motivating experts and professionals of the field rather than a heterogeneous audience, resulting in nichesourcing. Hence, effective and targeted communication of a crowdsourcing call is pivotal to the success of a crowdsourcing initiative.

Moreover, conceptualising, modelling and documenting LH in machine-readable, interoperable and participatory ways is challenging. LH contains complex concepts such as community agency and collective expressions, which are often authorship-agnostic, pertaining to communities and their members who are carriers of heritages that are interconnected with economic and social aspects of everyday life, which are often undocumented and, therefore, hard to define. LH content collected through participatory processes by digital means results in documentation workflows and data structures that have been scarcely defined, standardised or assessed [67]. In the context of the selected web tools and designed applications, structured data did not support LH-specific documentation. In particular, metadata elements of the commonly used Dublin Core standard, such as date, creator, rights and coverage, may present defining challenges. In general, the documentation of LH inside and outside museum contexts remains underexplored, with the need to establish an inclusive standardised approach. A preliminary report towards the registration of LH in collection management systems has been recently presented [68]. In addition, as many LH practices and knowledge are in an endangered state of being lost due to several factors, such as weakening of practice, environmental degradation and demographic issues, it might be critical to filter and evaluate crowdsourced LH content, even when documented as unstructured data. However, the crowdsourced stories in the context of the present study are collected from already documented sources and comprise mainly folklore and local history stories. In addition, this study is focused on the educational aspects of web application development and evaluation methods for HCI in crowdsourcing, and the outcome is less related to collecting original LH content.

5. Conclusions

This study formulated and applied a reusable and measurable evaluation methodology for crowdsourcing web applications with geotagging and storytelling capabilities in the CH field. The methodology was validated through a case study focused on the “Palace of the Grand Master of the Knights of Rhodes”, highlighting the potential of crowdsourcing to engage the public in preserving and promoting cultural heritage. Four web tools for developing the crowdsourcing applications were selected (StoryMapJS, ArcGIS StoryMaps, Historypin and Omeka), based on the minimum requirements of being a web application, enabling crowdsourced data display and map-based storytelling. Consecutively, four crowdsourcing web applications were developed using the selected web tools for promoting stories of folklore, LH and local history related to the monument “Palace of the Grand Master of the Knights of Rhodes”. Next, the web applications were assessed based on the evaluation methodology consisting of four stages: (i) the identification of web tools; (ii) the development of web applications using the tools, and a two-step evaluation approach; (iii) a five-criteria set evaluation (assessing open access, usability/tool support, participatory content/story creation, metrics and metadata model usage); and (iv) a crowd-based survey of 12 questions, including 3 questions of demographic information and 9 questions inquiring about primarily user experience, behaviours, user engagement and usability. The crowd-based survey received responses from 100 questionnaire respondents after promotion on social media and through email communication. The developed web applications were evaluated using the two-step evaluation approach: in the first step, selecting two out of the four developed web applications that passed the five predetermined criteria (Historypin, Omeka), and in the second step, selecting one out of the two web applications based on user responses on user-friendliness, ease of use, usability and user engagement. The summative evaluation underscored a general interest among participants in further contributing to and recommending such applications in the context of local cultural heritage promotion. The final application collected 22 stories of folklore, LH and local history content from users.

Future research directions beyond this study’s results are identified in establishing metadata models and standards for addressing the characteristics of LH content [68] integrated into crowdsourced web tools and applications. This could result in improved data quality for LH content regarding interoperability, especially if datasets are available for download and reuse as open data. In addition, exploring the benefits of niche crowdsourcing [11] in the digital documentation of LH might yield promising results in optimising the quality of crowdsourced LH content and in sustaining user interest and activity, as expert knowledge in the LH field often lies in specific communities and their members who are dedicated keepers and practitioners of specific practices and knowledge. In this direction, identifying niche networks and designing engagement strategies could drive more engagement and contribute to higher participation numbers with knowledgeable input.