Skip to content
Munich Music Labs

ResearchPaper · September 2026

A watermark inside the codec.

Latent Audio Watermarking for Robustness to Neural Codec Resynthesis

Lovro Brulec, Sahil Karawade, Leonard Kinzinger · Munich Music Labs & TUM

Built by MML members, supervised by Leonard Kinzinger (PhD student, TUM)

Read on arXiv PDFCode

In short

An audio watermark is an inaudible, machine-readable code embedded in a recording, so its origin can be checked later. As generated speech and music become hard to tell apart from real recordings, watermarks are one of the main tools proposed for provenance and misuse detection.

Established methods such as AudioSeal and WavMark survive most conventional processing: noise, filtering, MP3. They largely fail after neural codec resynthesis, when audio is encoded and decoded again by a learned codec. Neural codecs sit at the core of today’s generative audio systems, so this is exactly the transformation a watermark is most likely to meet.

The paper tests a deliberately simple alternative, DeltaMark: write the watermark into the latent representation of a frozen, pretrained codec, and let that codec’s own decoder produce the audio.

Payload
16 bits

per three-second clip

Detection after 4 codec passes
89.3%

AudioSeal: 11.2% · WavMark: none

Codec retraining
None

EnCodec stays frozen throughout

Why codecs erase watermarks

Built to discard what you cannot hear.

A neural codec compresses audio into a compact latent representation and rebuilds the waveform from it, keeping what matters for how the audio sounds and dropping the rest. A watermark is designed to be inaudible, which places it in exactly the part of the signal the codec is built to drop.

Waveform domain · AudioSeal, WavMarkAdded to the samples.Resynthesis rebuilds the signal from the codec’s latent, and the mark weakens with every pass.
Latent domain · DeltaMarkAdded to the codec latent.The question becomes whether the offset lies in directions the codec keeps when it re-encodes the audio.

How it works

Embed in the latent, read from the audio.

  1. 01EncodeA frozen, pretrained EnCodec encoder maps a three-second clip to a continuous latent: 128 channels × 225 frames.
  2. 02EmbedA small network turns a 16-bit message into a latent offset Δz. It depends only on the message, never on the audio.
  3. 03DecodeThe unchanged EnCodec decoder renders the shifted latent as audio. No codec retraining, no per-clip optimisation.
  4. 04AttackDuring training the audio is distorted: noise, filtering, echo, resampling and an EnCodec round trip.
  5. 05ExtractA convolutional extractor recovers the 16 bits from the audio alone; a small head decides whether a mark is present.

The embedder and extractor are trained jointly on LibriSpeech speech. The codec never changes.

Results

Degrades gradually, not all at once.

We passed watermarked speech through EnCodec at 24 kbps up to four times in a row and measured full-message accuracy: the share of clips where the mark is detected and all 16 bits come back correct. WavMark is not detected in any of these settings.

1 pass24 kbps

  • DeltaMark84.0%

  • AudioSeal65.0%

  • WavMark0.0%

2 passes24 kbps

  • DeltaMark60.6%

  • AudioSeal20.4%

  • WavMark0.0%

3 passes24 kbps

  • DeltaMark40.4%

  • AudioSeal4.0%

  • WavMark0.0%

4 passes24 kbps

  • DeltaMark24.6%

  • AudioSeal0.6%

  • WavMark0.0%

Full-message accuracy, %. 2,000 LibriSpeech test clips. Paper, Table 2.

Lower bitrates

A single pass at lower bitrates shows the same pattern: 75.8% of messages survive at 12 kbps and 37.0% at 6 kbps, against 19.8% and 0.1% for AudioSeal. At 3 kbps and below, payload recovery largely fails, although at 3 kbps the mark is still detected in 54.6% of clips.

Other codecs

On DAC, which DeltaMark never saw in training, detection stays between 96.5% and 99.5%. AudioSeal recovers more full messages at DAC’s two higher settings, but detects fewer marks. Under MP3 and AAC every method holds up well: 96.5 to 96.6% for DeltaMark, 100% for AudioSeal.

Conventional edits

Detection stays above 99% for every tested distortion except low-pass filtering (95.4%). Full-message recovery depends more on the attack: it is high after resampling, clipping and dropout, lower after noise, gain changes and filtering. The baselines are clearly stronger under smoothing.

Why it works

Directions the codec keeps.

The robustness is not only learned from seeing the codec. A variant trained without any codec attack still keeps 87.0% detection after four passes; codec-aware training improves exact message recovery on top of that.

To see why, we decoded and re-encoded the watermark offset and measured how much of it survives along its original direction. The learned offset survives far better than random noise of the same size. Shuffling it in time costs little; shuffling it across latent channels removes the advantage. What carries the mark is how it is spread over channels, not when it happens.

  • Learned offset0.458

  • Shuffled in time0.416

  • Shuffled across channels0.246

  • Random (Gaussian)0.242

Projection ratio after EnCodec at 24 kbps (0 = gone, 1 = fully kept). Controls are norm-matched. Paper, Table 6.

Limitations

What it does not solve yet.

  1. 01

    Same codec on both sides

    The watermark is embedded with EnCodec and most codec tests also use EnCodec. That is a favourable case.

  2. 02

    Not every codec

    Detection transfers to DAC but not to TiCodec, where it drops to 6.9%. AudioSeal and WavMark are not detected there either.

  3. 03

    Quality set by the carrier

    The output can be no cleaner than EnCodec’s own reconstruction, and objective quality stays below the waveform baselines (PESQ 3.32 against 4.48 for AudioSeal).

  4. 04

    Removable in principle

    Clips carrying the same message share the same offset, which an adversary could estimate and subtract. Real deployment would need keyed or content-dependent embedding.

  5. 05

    Speech only, so far

    Training and evaluation use LibriSpeech. Music has not been tested yet.

For researchers

Next steps are content-dependent embedding, higher-fidelity latent carriers, and understanding which codec representations preserve watermark directions. The code is on GitHub.

Method in detailArchitecture, losses, setup

Carrier

Official pretrained mono EnCodec at 24 kHz. For a 3 s clip the encoder gives a continuous latent z of size 128 × 225. Residual vector quantisation is bypassed during insertion.

Embedder

A three-layer MLP (hidden width 256, GELU) projects the 16-bit payload s to a 128 × 64 tensor, linearly interpolated to 225 frames. The watermarked audio is x_w = D(z + α·F(s)), with α a learned scalar and D the frozen decoder. The perturbation depends on the payload only.

Extractor and detector

A 1D residual CNN (four stages, widths 32, 64, 128, 256, global average pooling, linear head) predicts the payload bits. Detection uses a small binary MLP head trained for 10 epochs on the extractor’s frozen pooled features, with its threshold calibrated on validation data to a 1% attack-weighted false positive rate.

Training

Loss: message BCE on attacked audio, latent energy ‖Δz‖², log-mel loss, AudioSeal’s time-frequency loudness loss, and a masking-inspired multi-resolution STFT loss that penalises residuals more in quiet time-frequency regions. Four stages (bootstrap, generalisation, robustness, refinement) switch on validation bit accuracy. One random waveform attack per example, reweighted toward the weakest; the full model adds a differentiable EnCodec cycle without RVQ. 16,000 LibriSpeech train-clean-100 clips, AdamW, batch 64, lr 1e-3, up to 500 epochs with early stopping.

Evaluation

2,000 LibriSpeech test-clean clips. AudioSeal v0.2.0 and WavMark v0.0.3 use their official models at 16 kHz, resampled around the shared 24 kHz transformations, all with 16-bit payloads. Metrics: true positive rate at the calibrated false positive rate, and full-message accuracy (detected and all 16 bits correct).

Quality

Against the original audio: SI-SNR 8.76 dB, PESQ 3.32, STOI 0.969, ViSQOL 4.29. The unwatermarked EnCodec reconstruction alone reaches 9.07 dB and PESQ 3.89, so most of the gap comes from the codec. Measured against that reconstruction, the watermark itself gives SI-SNR 15.52 dB, PESQ 3.73, STOI 0.985, ViSQOL 4.58. AudioSeal and WavMark reach PESQ 4.48 and 4.26.

Cite this paperBibTeX
@misc{brulec2026latent,
  title         = {Latent Audio Watermarking for Robustness to Neural Codec Resynthesis},
  author        = {Brulec, Lovro and Karawade, Sahil and Kinzinger, Leonard},
  year          = {2026},
  eprint        = {2609.25830},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  doi           = {10.48550/arXiv.2609.25830}
}

Built by student researchers

Write the next one with us.

All researchJoin R&D