MUSMET – Musical Metaverse made in Europe – EU EiC Pathfinder Open

The MUSMET project proposes a vision and cutting-edge technological innovation for the future classes of Musical Metaverse devices, networking systems, and services, capable of catering to the needs and expectations of musicians and audiences. The implementation of this vision will spur the creation of radically new ecosystems of interoperable devices and communities utilising them, which holds significant benefits for society, economy, and art.

https://musmet.eu/

TIVOM – Tactile Integration for Visually-impaired Orchestra Musicians – POpS Social Impact Project

The project introduces a wearable vibrotactile device that translates the conductor’s gestures and facial expressions into real-time tactile feedback perceivable by musicians with visual impairments. Leveraging motion tracking technologies, multimodal data acquisition, and machine learning algorithms optimized for low-cost embedded platforms (e.g., Raspberry Pi), TIVOM aims to recognize conducting gestures and convert them into structured vibrotactile signals with minimal latency.

By combining accessible hardware design, real-time gesture recognition, and user-centered validation with visually-impaired musicians, the project seeks to remove sensory barriers that limit participation in orchestral contexts such as rehearsals, performances, and auditions.

The resulting system will promote inclusive musical practices, expand educational opportunities for music institutions, and stimulate innovation within the Internet of Musical Things (IoMusT) ecosystem, generating measurable social, scientific, and economic impact.

HiFiReM – High Fidelity Remote Music Platform – FISR Project

The project envisions a web-based and hardware-augmented communication system capable of overcoming the audio quality degradation and synchronization limitations of conventional videoconferencing platforms. By prioritizing pristine audio transmission and minimizing mouth-to-ear latency below perceptual thresholds, HiFiReM enables geographically distributed musicians to interact “as if” they were co-present in the same acoustic space.

The platform combines innovative WebRTC-based software architecture, optimized audio transmission protocols, and a low-cost dedicated hardware unit (“music box”) to achieve high-quality, synchronized audio streams across standard broadband networks. Advanced features include real-time collaborative performance modes, distributed synchronization mechanisms, and remote audio mixing capabilities suitable for concerts, rehearsals, and music education.

HiFiReM opens new opportunities for remote artistic collaboration, expands access to music education, supports musicians with mobility constraints, and fosters the development of innovative digital music services. Its implementation contributes to strengthening the resilience and sustainability of the music sector in both emergency and non-emergency contexts.

Musical Metaverse (PRIN 2022 Project) – Italian MUR PRIN 2022

The project envisions a new generation of interoperable Musical Metaverse ecosystems enabling musicians to compose, perform, and teach in immersive shared XR environments, overcoming geographical, physical, and social barriers. By integrating advanced human–computer interaction, low-latency networking architectures, machine learning–based traffic prediction and packet loss concealment, and ethical-by-design methodologies, the project aims to enable real-time synchronous musical collaboration between geographically distributed performers.

Through an iterative Design–Develop–Evaluate methodology, the project will deliver progressively refined prototypes capable of supporting collaborative composition, live performance, and music education across classical, pop, rock, and experimental genres. Special attention is devoted to inclusivity, accessibility, gender balance, and participation of visually- and motor-impaired musicians.

The implementation of this vision will lay the scientific and technological foundations of the Musical Metaverse field, fostering new socio-technical ecosystems, artistic practices, and digital communities, with significant impact on culture, research, industry, and society.

Real-Time MIDI Error Recovery for Ultra-Low Latency Networked Music Performance (PoC Project)

The project addresses a key challenge in Networked Music Performance (NMP): maintaining synchronous musical interaction between geographically distributed musicians under strict latency constraints (below 25–30 ms). Because such low latency prevents the use of retransmission protocols and large buffers, packet loss must be handled directly at the receiver without introducing additional delay.

The proposed solution integrates a patented real-time MIDI packet loss recovery mechanism that periodically transmits absolute system state information (active MIDI events), enabling reconstruction of missing data without retransmissions or acknowledgment mechanisms.

The prototype will be implemented on a low-cost embedded architecture based on FPGA and microcontroller technology, supporting both MIDI and raw audio streaming via RTP/UDP/IP in peer-to-peer configurations. It will ensure deterministic ultra-low-latency processing (<5 ms internal delay) and include real-time audio mixing capabilities controlled locally or remotely.

The demonstrator will be validated through objective measurements and perceptual testing with musicians under controlled latency and packet loss conditions. The technology opens opportunities for remote rehearsals, distributed live performances, music education, and professional studio collaboration, contributing to the advancement of Networked Music Performance and Internet of Musical Things ecosystems.