Table of contents
- It Started with a Few Instagram Reels
- My Personal Connection to Music and AI
- From AI Music Generation to Real-Time Interaction
- Prediction: Why Musicians Feel What Comes Next
- Feeling the Music
- Musical Experience, Possibilities, and Creativity
- Can AI Reproduce This Behavior?
- Prediction Is Not Enough
- Multimodal AI: Why Listening Is Not Enough
- Music Is More Than Audio
- What Does Multimodal AI Mean?
- We Already Have Some of the Technical Building Blocks
- Combining Information Requires More Than Recognition
- Timing Changes Everything
- Why This Made Me Think Beyond Music
- From Multimodal AI to Humanoid Robots
- From Perception to Action
- The Additional Challenges of Embodied AI
- Research Is Connecting These Capabilities
- Why I Find This Connection Interesting
- Final Thoughts: Everything Is Connected
- References
1. It Started with a Few Instagram Reels
It started with a few Instagram Reels. I saw people asking ChatGPT to sing. Some of the results were funny, and to be honest, the singing was not very good. So I decided to try it myself. I asked ChatGPT to sing something. The result was interesting, but it was far away from what I would call real singing. However, I was not disappointed. I was more interested in the technology behind it.
The original German Instagram Reel.
We already have AI tools like Suno that can generate music, including vocals and instruments. [6] We also have AI systems that can communicate with us using speech in real time. [2]
This made me wonder:
What would happen if we combined these technologies?
- Could an AI become a musical partner instead of just a tool for generating songs?
- And why would we actually want AI to sing or play music with us in real time?
That question led me to something much bigger. For an AI to participate in a real musical session, it would need more than the ability to generate sound.
It would need to listen, observe, anticipate, decide, and adapt in real time.
And while thinking about these requirements, I realized something interesting.
Many of these capabilities are also relevant to humanoid robotics.
Suddenly, a simple Instagram Reel about AI singing became the starting point for exploring real-time interaction, multimodal AI, prediction, and humanoid robots.
This is the journey I want to share in this blog post.
2. My Personal Connection to Music and AI
Music has always been part of my life. For as long as I can remember, I have enjoyed listening to music and playing instruments. Over time, the drums became my main instrument. Today, I play in cover bands, and I enjoy making music together with other musicians.
For me, playing music is more than just playing the correct notes or keeping the rhythm.
It is about interaction.
When musicians play together, they listen to each other. They react to small changes in rhythm, dynamics, and expression.
Sometimes, a simple look, a nod, or a small movement is enough to communicate what should happen next. [1]
I also started experimenting with creating my own music.
Over time, I tried different approaches, including working with other musicians, using loops, and experimenting with AI music generation.
One challenge when creating your own music is finding people who share the same interests and musical direction, and who also have the time to work together.
Tools like Suno offer another possibility. They allow me to turn musical ideas into songs without needing an entire band for every experiment. [6]
It does not replace the experience of playing live with other musicians. But for me, it provides another way to explore my creativity.
I prefer using AI to explore my musical ideas rather than leaving them entirely unexplored.
Today, I experience both worlds: playing live with other musicians and creating my own music with the help of AI.
And this brings me back to my original question.
What if AI could move beyond generating finished music?
What if it could actually participate in a musical session?
What would it need to listen to, understand, and do to become a musical partner?
3. From AI Music Generation to Real-Time Interaction
When we look at AI music generation and real-time speech interaction, we can see two important technological developments. On one side, we have systems that generate music. On the other side, we have systems that communicate with humans in real time. Both technologies have developed considerably. [6] [2]
However, generating a complete song and actively participating in a live musical session present two entirely different technical challenges.
A music generator can process a prompt and produce a complete musical result. A real-time musical partner must continuously listen, react, and adapt to what is happening. Imagine sitting behind your drums while an AI plays bass. You start playing a simple rhythm. The AI follows you. You increase the tempo slightly. The AI adapts. You change the dynamics or introduce a drum fill. The AI recognizes the change and responds musically. This sounds simple, but it requires several capabilities.
- First, the system must process audio with very low latency.
- Second, it must recognize and interpret musical structures, including tempo, rhythm, harmony, and dynamics.
- Third, it must generate appropriate musical responses while the session continues.
- And finally, it must adapt to unexpected changes without interrupting the musical flow. [4]
This is not only a question of computing power. Model architecture, efficient audio processing, synchronization, and prediction are equally important. Another interesting aspect is that technological development often happens in parallel.
We develop speech models, music generation systems, multimodal models, and robotics systems for different purposes. But sometimes, capabilities developed in one area can also become useful in another.
We do not necessarily need to solve every problem with one large technological breakthrough.
Sometimes, progress happens one step at a time, by combining capabilities that already exist.
And this leads to the next question:
How can an AI react quickly enough to play music together with a human?
One possible answer is prediction. Because when musicians play together, they do not simply wait until something happens and then react. They also anticipate what might happen next.
4 – Prediction: Why Musicians Feel What Comes Next
When musicians play together, they do not simply react to notes that have already been played. There is something more happening. As a drummer, I experience this every time I play with other musicians.
I do not really predict what will happen next. I feel what might happen next.
This is an important difference. Musicians are not fortune tellers. We cannot see the future. But when we play music together, we develop a feeling for what might come next. I would describe this as a kind of intuitive prediction. [7]
4.1 – Feeling the Music
When I play drums with my band, I do not consciously calculate every note or every movement. I listen to the music. I watch the other musicians. I feel the rhythm and the energy of the song. Several things influence what I decide to play.
For example, I know the musical style.
If we play a rock song, I already have certain expectations about the rhythm, the structure, and possible transitions.
I also have my own musical experience.
Over the years, I have listened to many songs, learned different rhythms, played with different musicians, and developed my own repertoire. All these experiences influence my playing. Sometimes, I recognize that a transition is coming. Sometimes, I feel that a drum fill would fit. And sometimes, I notice that another musician is preparing to change something.
I do not need to think consciously about every detail. Much of this happens intuitively.
Of course, this does not mean that there is no learning or pattern recognition involved. My experience influences what I expect. But from my personal perspective, it feels much more like intuition than calculation.
4.2 – Musical Experience, Possibilities, and Creativity
There is another important aspect. Even when I feel what might come next, I still need to decide what I can actually play. My musical possibilities depend on several things. I have my own repertoire of rhythms, fills, and techniques. I understand certain musical structures and have expectations based on the musical style and harmony. But I also have physical limitations.
I can only play what my technical abilities and physical coordination allow me to play.
This influences my creativity. When I play a song, I try to find the best musical contribution within my possibilities. Sometimes, this means playing exactly what the musical style suggests. Sometimes, it means doing something different. For example, a certain transition might normally lead to a typical drum fill. But I might decide to play something unexpected instead. Not because I made a mistake, but because I want to create a different musical expression.
This is an important part of creativity.
Music is not always about playing what is most likely to come next. Sometimes, creativity means choosing something unexpected. [9]
And when musicians play together, these decisions influence each other. One musician introduces a new idea. Another musician reacts. The music develops through this interaction.
4.3 – Can AI Reproduce This Behavior?
This brings me back to artificial intelligence. If we want an AI to become a musical partner, it needs more than the ability to recognize a rhythm and generate the next notes. It needs to interpret the musical context. It could use information such as the current tempo, rhythm, harmony, dynamics, and musical style. It could also consider what has happened earlier in the session. Based on this information, a model could estimate what musical events might happen next. [8] For example, it might estimate that a transition is approaching or that the drummer is preparing a fill.
This is where prediction becomes interesting.
An AI does not need to feel music in the same way as a human to attempt to reproduce some of the musical behavior that results from human intuition.
It could use learned patterns and predictive models to generate appropriate musical responses. However, predicting the most likely musical event is not necessarily the same as making a good musical decision. An AI musical partner would also need to handle unexpected changes. And ideally, it should not only follow the human musician. It should also be able to contribute musical ideas. This introduces an interesting balance between prediction and creativity.
- Should the AI always choose the most likely continuation?
- Or should it sometimes introduce something unexpected?
For me, a musical partner should be able to do both.
4.4 – Prediction Is Not Enough
There is one more challenge. Even if an AI can predict what might happen next, it still needs to react at the right moment. In music, timing is essential. A response that arrives too late can interrupt the musical flow.
But there is another aspect that I find particularly interesting.
When I play with other musicians, I do not only listen. I also watch them. I notice their movements, facial expressions, and gestures.
Sometimes, a simple look tells me that something is about to change. A guitarist might move differently before starting a solo. A singer might take a breath before entering the next part of a song.
These are additional signals that help musicians coordinate their playing. [1] They are not guarantees of what will happen next, but they provide useful context.
Musical interaction is not only about sound. It is also about observing and interpreting what other musicians are doing.
And this leads to the next technical challenge. If an AI should participate naturally in a musical session, audio processing alone might not be enough. It could also benefit from visual information, movement recognition, and an understanding of the surrounding situation.
This brings us to another important area of artificial intelligence.
Multimodal AI.
5. Multimodal AI: Why Listening Is Not Enough
In the previous chapter, I described how musicians develop a feeling for what might happen next. As a drummer, I experience this when playing with my band. I listen to the music, feel the rhythm, and use my experience to decide what I should play. But there is another important aspect.
When musicians play together, they do not only listen to each other. They also observe each other.
And this brings me to another interesting area of artificial intelligence: multimodal AI.
5.1 – Music Is More Than Audio
Imagine playing a song with your band. You are sitting behind your drums. The guitarist is playing, the singer is preparing for the next verse, and everyone knows the song. Suddenly, the guitarist looks at you. Perhaps he wants to start a solo. Perhaps he is preparing for a transition. Or perhaps he is simply looking in your direction. Sometimes, a small movement, a facial expression, or a simple nod is enough to communicate something. But the interesting part is that we do not interpret these signals independently. We combine them with what we already know.
We know the song, the musical style, and the other musicians, allowing us to remember what happened earlier and develop a feeling for what might come next. Because the way a guitarist holds an instrument or shifts their weight can provide vital context, we constantly synthesize our ears, eyes, experience, and intuition into a single cohesive response.
We do not simply recognize individual signals. We interpret them within the musical situation.
This is not only my personal observation. Research by Bishop, Cancino-Chacón, and Goebl investigated visual communication between musicians during ensemble performances. Their study found that musicians use visual interaction in different ways, including monitoring one another during periods of temporal instability.
This supports the idea that visual information can contribute to musical coordination. [1] Of course, this does not mean that every movement has a specific musical meaning. A musician can misunderstand another musician. And sometimes, we simply make mistakes. That is part of playing music together.
5.2 – What Does Multimodal AI Mean?
In artificial intelligence, multimodal describes systems that process information from different types of input. For example, these inputs can include audio, images, video, and text. The interesting part is not simply having multiple inputs. It is the possibility of combining information from these different sources to interpret a situation. Let’s return to our imaginary AI bass player. The AI is playing bass while I am playing drums.
It receives the audio from our musical session. From this audio, it could try to identify the tempo, rhythm, dynamics, and musical structure.
Now imagine that the system also has access to a camera. It could potentially observe the movements of the musicians.
Perhaps it recognizes that the guitarist is looking toward the drummer. Perhaps the singer makes a gesture shortly before a transition. A multimodal system could combine these visual observations with the audio information and the current musical context. But that does not automatically mean the AI understands what is happening.
Recognizing a gesture is not the same as understanding the intention behind it.
For example, the guitarist looking toward me could mean several things. The system would need to interpret this observation together with the current musical situation. And even then, its interpretation might be wrong. This is where multimodal AI becomes much more interesting than simply connecting a camera and a microphone to a model.
5.3 – We Already Have Some of the Technical Building Blocks
When I started thinking about these requirements, I wondered whether we already have technologies that could support parts of this interaction. And the answer is interesting. We already have several technologies that address different parts of the problem. For example, OpenAI provides a Realtime API for low-latency speech interaction.
The API supports streaming audio input and output, allowing applications to process spoken interactions without relying on a traditional sequence of separate speech recognition, text generation, and speech synthesis steps. [2] Google also provides multimodal real-time interaction through its Gemini Live API. The documentation describes processing real-time audio together with additional information such as video. [3]
These technologies demonstrate that real-time audio processing and multimodal interaction are already practical areas of AI development. But speech interaction and musical interaction are not identical.
Music introduces additional timing and synchronization requirements. A response that is acceptable in a spoken conversation might arrive too late for a musical performance.
And there is another interesting development. In June 2026, Google introduced Magenta RealTime 2, an interactive music model designed for low-latency music generation on local hardware. [4]
According to Google’s documentation, the model supports interactive control through audio, MIDI, and text. It also supports live accompaniment and integration into music production environments. [4] This is particularly interesting for my original question.
We are no longer talking only about generating a complete song from a prompt. We are also talking about AI models designed for interactive music creation. However, there is an important distinction.
An AI model that generates music in real time is not automatically an AI musician that understands the intentions of other musicians.
For my imaginary AI bass player, we would need to combine several capabilities. The system would need to process musical information, interpret context, synchronize its output, and adapt to what happens during the performance. Adding visual information could provide another useful source of context. But how reliably such a combined system could perform in an actual band remains an open engineering question.
5.4 – Combining Information Requires More Than Recognition
Let’s imagine another situation. We are playing a rock song. The guitarist suddenly changes the way he plays. At the same time, he looks toward me and moves closer to the microphone. As a drummer, I might feel that something is about to change. Perhaps the guitarist wants to start a solo. Perhaps there will be a break. Or perhaps he wants to introduce something unexpected.
My interpretation comes from different sources of information.
- I hear the music.
- I observe the guitarist.
- I know the song.
- I remember how we played it before.
- And I use my musical experience.
An AI system could attempt something similar by combining different inputs.
For example, it could process audio features, visual observations, information about the song structure, and earlier events in the session.
It might then estimate which musical developments are plausible. But an important challenge remains. Not every observation is equally relevant. And some observations might contradict each other.
The music could suggest that the chorus is about to begin, while the guitarist introduces something completely unexpected.
- What should the AI do?
- Should it follow the expected song structure?
- Should it react to the guitarist?
- Or should it contribute a different musical idea?
These are not only technical questions about recognizing audio or video.
They are also questions about musical decision-making.
A useful AI musical partner would need to interpret information, handle uncertainty, and adapt when expectations are wrong.
And that brings us back to the intuitive prediction discussed in Chapter 4. As a human musician, I do not consciously calculate all these possibilities.
I feel the music. An AI would need to approximate aspects of this behavior through computational methods. That does not mean the AI experiences the music as I do. It means that we could try to reproduce some of the observable behavior that results from musical experience and intuition.
5.5 – Timing Changes Everything
There is another important challenge. All these capabilities must work together while the music continues. Imagine that the singer gives me a visual signal just before the next part of the song.
If I recognize the signal too late, I might miss the transition. The same problem would exist for an AI system. Audio and video information must be processed quickly enough to influence the musical response.
But low latency alone is not sufficient. The different information sources must also be synchronized. For example, a camera might capture the guitarist’s movement while a microphone captures the sound of the guitar. The AI must correctly relate these observations to the same musical situation.
It must then decide what to play and generate its musical response at the appropriate time. Prediction could help the system prepare for likely musical events. However, it would still need to respond when something unexpected happens.
This creates an interesting combination of technical requirements:
- Real-time audio and visual processing
- Synchronization between different information sources
- Interpretation of musical context
- Prediction and handling of uncertainty
- Musical decision-making and low-latency response
These requirements are connected.
Improving only one component would not necessarily produce a good musical partner.
The real challenge is making all these capabilities work together as one interactive system.
5.6 – Why This Made Me Think Beyond Music
While thinking about these requirements, I started to recognize something interesting. We are no longer talking only about music generation. We are describing an AI system that interacts with its environment.
It receives information, interprets what is happening, develops expectations, decides how to respond, and adapts when the situation changes.
This sounds familiar because similar requirements appear in another area of artificial intelligence: Embodied AI and robotics.
Consider a humanoid robot working together with a human. It might need to recognize speech, interpret visual information, observe human movements, and anticipate what could happen next.
Of course, an AI musical partner and a humanoid robot are not the same. A robot must also perform physical actions, control its movements, and operate safely in the physical world.
But both systems can benefit from combining perception, context, prediction, and timely responses.
This is what made me think beyond music. Not because an AI that can sing will automatically become a robot, but because exploring one technical challenge can reveal connections to other areas of AI.
What changes when an AI system not only perceives its environment but also needs to act within the physical world?
That brings us to Embodied AI.
6. From Multimodal AI to Humanoid Robots
In the previous chapter, I explored how an AI musical partner could combine audio, visual information, context, and prediction. While thinking about these requirements, I noticed something interesting.
Many of the capabilities needed for an interactive AI musician are also relevant to humanoid robots.
Of course, playing music and controlling a robot are different challenges. But there are similarities worth exploring.
6.1 – From Perception to Action
Imagine our AI bass player again. It listens to the music, observes the other musicians, and interprets what is happening. Based on this information, it decides what to play next. The result is generated audio.
Now imagine a humanoid robot working together with a human. The robot observes its environment, recognizes objects, interprets movements, and decides what action to perform. But there is an important difference.
A robot must turn its decisions into physical movements.
It needs to control motors, coordinate movements, and interact with physical objects. And every physical action can change its environment. The robot must observe these changes and adapt its next actions.
This creates a continuous feedback loop:
Perceive → Interpret → Predict → Decide → Act → Observe
Interestingly, we can recognize a similar interaction loop in our imaginary AI musician. The difference is primarily in the actions performed, the environment, and the additional physical constraints.
6.2 – The Additional Challenges of Embodied AI
In robotics, the term Embodied AI refers to artificial intelligence interacting with the physical world through a physical system. For example, a humanoid robot may need to pick up an object, move through a room, or cooperate with another person. These tasks require more than recognizing objects or understanding spoken instructions.
The robot must consider its physical capabilities, the position of objects, available space, and the possible consequences of its movements.
Timing, balance, coordination, and safety become essential. This reminds me of something I discussed in Chapter 4.
As a drummer, my musical decisions are influenced by what I can physically play. I might imagine a complicated drum fill, but my technical abilities and physical coordination determine whether I can perform it.
A robot faces a related challenge.
It must choose actions that are not only appropriate but also physically possible.
However, a humanoid robot needs additional control systems to execute those actions safely and reliably. This is an important difference between generating a digital response and acting in the physical world.
6.3 – Research Is Connecting These Capabilities
This connection is not only a theoretical idea. Google DeepMind introduced Gemini Robotics in March 2025, describing a family of models designed to connect multimodal AI capabilities with physical actions.
One approach uses Vision-Language-Action (VLA) models to translate visual information and instructions into actions for robotic systems. Another area, called Embodied Reasoning, focuses on understanding physical situations and planning actions.
Google DeepMind’s later Gemini Robotics research also explores how reasoning and action models can work together. [10]
These developments are relevant because they address some of the same general challenges discussed throughout this blog post:
- Perceiving and interpreting a changing environment
- Combining information from different sources
- Planning and adapting actions
- Reacting at the right moment
Of course, this does not mean that an AI musician and a humanoid robot use identical architectures.
And it does not mean that an AI musician can simply be transformed into a robot.
The connection lies in some of the underlying capabilities, not necessarily in their implementations.
6.4 – Why I Find This Connection Interesting
When I started watching people ask ChatGPT to sing, I was not thinking about robotics. I was thinking about music. But the more I explored the idea of an AI playing music together with humans, the more I encountered technical requirements that also appear in robotics.
- Real-time processing.
- Multimodal perception.
- Prediction.
- Decision-making.
- Continuous adaptation.
And, when physical interaction is involved, motor control. For me, this is an interesting example of how developments in different areas of AI can be connected. We do not necessarily need one technology that solves every problem. Different models and specialized components may contribute different capabilities. And sometimes, combining these capabilities can open possibilities that were difficult to imagine when the individual technologies were developed.
What started as a question about AI singing became a way for me to think about how intelligent systems might interact with humans in the future.
That brings me to my final thoughts.
7. Final Thoughts: Everything Is Connected
This journey started with something simple. I watched a few Instagram Reels of people asking ChatGPT to sing. The results were funny, but they made me curious. As a musician and AI engineer, I started wondering what would happen if AI could do more than generate finished songs.
What if AI could become a musical partner?
Thinking about this question led me through several areas of technology. Real-time audio processing, low latency, prediction, multimodal AI, and finally humanoid robotics. At first, these topics might seem unrelated. But while exploring them, I realized that they share some important technical challenges.
An AI system needs to receive information, interpret its context, make decisions, and respond at the right moment. Depending on the application, it might also need to predict what could happen next or perform physical actions.
Of course, the technical implementations are different. An AI music generator is not a humanoid robot, and combining existing technologies does not automatically solve all the challenges. But I find the connections interesting.
For me, the important connection is not that an AI musician and a humanoid robot are the same. Both need to interpret information, make decisions, and respond to changing situations. The difference is what happens next. An AI musician generates a musical response, while a humanoid robot must turn its decisions into physical actions.
Sometimes, progress does not come from one major breakthrough. It comes from combining capabilities developed for different purposes.
There is also something personal I take away from this experience. When I play drums with other musicians, I do not consciously calculate every musical decision. I listen, observe, react, and, most importantly, feel the music. AI does not need to experience music the same way I do to attempt to reproduce some aspects of this interaction.
And perhaps that is what makes this topic so interesting to me.
The goal is not necessarily to make AI feel like a human. It is to explore how AI can interact with humans in more natural and useful ways.
What began with a funny Instagram Reel became a reminder of something I find important in technology:
Stay curious, ask questions, and look for connections between things that initially seem unrelated.
Because sometimes, a simple question can lead to a much bigger idea.
8. References
[1] Bishop, L., Cancino-Chacón, C., & Goebl, W. (2019).
Eye gaze as a means of giving and seeking information during musical interaction.
https://doi.org/10.1016/j.concog.2019.01.002
[2] OpenAI – Realtime API Documentation
https://developers.openai.com/api/docs/guides/realtime
[3] Google – Gemini Live API
https://ai.google.dev/gemini-api/docs/live-api/capabilities
[4] Google Magenta – Magenta RealTime 2 (June 2026)
https://magenta.withgoogle.com/magenta-realtime-2
[5] Google DeepMind – Gemini Robotics (March 2025)
https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/
[6] Suno AI music generator (October 2026)
https://suno.com/tools/ai-music-generator
[7] Novembre, G., & Keller, P. E. (2014). A conceptual review on action-perception coupling in the musicians’ brain: What is it good for? Frontiers in Human Neuroscience, 8, 603. https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2014.00603/full
[8] Rohrmeier, M. A., & Koelsch, S. (2012). Predictive information processing in music cognition: A critical review.International Journal of Psychophysiology, 83(2), 164–175 https://www.sciencedirect.com/science/article/pii/S0167876011003898?via%3Dihub
[9] Loui, P. (2018). Rapid and flexible creativity in musical improvisation: Review and a model. Annals of the New York Academy of Sciences. https://nyaspubs.onlinelibrary.wiley.com/doi/10.1111/nyas.13628
[10] Google DeepMind (September 25, 2025). Gemini Robotics 1.5 brings AI agents into the physical world. https://deepmind.google/blog/gemini-robotics-15-brings-ai-agents-into-the-physical-world
Note: This post reflects my own ideas and experience. AI was used as a writing and thinking aid to structure and check the arguments, not to define them.
#ArtificialIntelligence, #AI, #GenerativeAI, #MultimodalAI, #RealTimeAI, #AIMusic, #AIMusicGeneration, #MusicTechnology, #HumanAIInteraction, #HumanAICollaboration, #AgenticAI, #EmbodiedAI, #HumanoidRobots, #Robotics, #MachineLearning, #MusicCreativity, #MusicalInteraction, #PredictiveAI, #SunoAI, #GeminiRobotics, #AIEngineering
Leave a Reply