Cross-Modal Generation and Modality-Driven Semantic Segmentation with Generative Adversarial Networks

January 4, 2021

For my M.Sc. dissertation at the University of Edinburgh, I worked on cross-modal generation: given one sensory modality, could a model generate a plausible version of another? The concrete version of the problem was audio-visual generation. Given an image, generate a plausible audio representation; given audio, generate a plausible visual representation.

The motivation was simple enough: the world is not uni-modal. We do not experience objects only as images, sounds, or isolated labels. A car can be seen, heard, touched, and expected to behave in certain ways. If artificial systems are going to operate in messy real environments, they need some ability to relate information across modalities and keep working when one stream is incomplete, ambiguous, or missing. Learning this joint representation really interested me at the time (and still does).

The dissertation had three parts:

reimplement and analyse CMCGAN, a cross-modal cycle generative adversarial network from literature;
curate a small proof-of-concept dataset, AudioCOCO, by linking classes from AudioSet and COCO;
adapt the cross-modal generation setup into a modality-driven segmentation model, where audio helps decide which object in an image should be segmented.

Overview of the CMCGAN model architecture — The CMCGAN architecture I reimplemented as the starting point for the dissertation.

The reproduction problem

The first part of the project was supposed to be straightforward: reimplement CMCGAN and reproduce the reported results on the Sub-URMP dataset of musicians playing instruments. In practice, this became the most instructive part of the whole project.

The model was implemented by extending an existing PyTorch CycleGAN codebase. Audio clips were converted into log-mel spectrograms so they could be treated image-like by the network, and the model learned the four cross-modal/cycle paths between real and generated audio and visual samples. There were a lot of moving pieces: custom data loading, paired and wrong-pair sampling, generator and discriminator losses, cycle consistency, hyperparameters, and enough GAN instability to keep things interesting.

The uncomfortable result was that I could not reproduce the original paper's visual quality. The generated audio had some resemblance to the target distribution, and the generated images learned some structure - outlines of performers, poses, and occasionally instruments - but the results were nowhere near as clean as the paper's examples.

Generated audio and visual samples from the CMCGAN reimplementation — Generated CMCGAN samples from my reimplementation. The model learned structure, but not at the quality reported in the original paper.

That failure was frustrating at the time, but it was probably the most honest technical lesson in the project. Some implementation details in the paper were underspecified, especially around the discriminator architecture and certain training settings. GANs are also very sensitive to initialization, update schedules, and small architectural choices. The examiner feedback later made a point I still agree with: the reproduction did not work, but the analysis around why it might not have worked was still valuable.

One small example was debugging vanishing gradients in the discriminator. I added gradient visualizations across layers, then adjusted the implementation to get gradients flowing more sensibly. It did not magically solve the whole model, but it made the debugging process less blind.

Gradient flow visualization before and after correcting a vanishing gradient issue — Gradient-flow debugging for the discriminator before and after correcting a vanishing-gradient issue.

Building AudioCOCO

The second part of the dissertation moved away from the musician dataset. Sub-URMP was useful, but it was also visually simple: one centered performer, one primary object class, and a controlled background. That makes cross-modal generation easier than it would be in a natural scene.

I wanted a small step toward a more realistic setting, so I curated AudioCOCO by manually matching overlapping classes between AudioSet and COCO. AudioSet provided labelled audio events, while COCO provided natural images with pixel-level object masks. The goal was not to create a polished benchmark; it was to create enough of a dataset to test whether the cross-modal idea could be made more scene-aware.

COCO samples with person, dog, and car masks — COCO samples used while constructing AudioCOCO subsets for person, dog, and car classes.

Two initial binary subsets were used: person/dog and person/car. The person/dog subset had 1,849 training samples, while the person/car subset had 8,119. The clustering analysis suggested that person/car had more useful audio structure than person/dog, which later matched the model behaviour. This was not a deep discovery, but it was a useful sanity check: some class pairings simply have more separable audio-visual structure than others.

t-SNE clustering for person and car AudioCOCO features — The person/car subset had more visible structure in the audio modality than the person/dog subset.

There were plenty of limitations. AudioSet labels are not localized, clips can contain multiple events, and the audio was not actually paired with the COCO images in a grounded way. Many masks were also tiny relative to the image area, making the segmentation task unnecessarily difficult. Still, as a proof-of-concept dataset, it was useful enough to expose the next problem.

From generation to segmentation

Directly generating full natural scenes from audio is a hard task, and probably too hard for the setup I had. The pivot was to make the generation problem more focused: instead of generating the entire scene, generate a saliency-like mask over the object relevant to the audio.

That became CMCSSGAN: a cross-modal cycle semantic segmentation GAN. Given a conditioning image and an audio clip, the model tried to generate a mask over the object class associated with the sound. In the reverse direction, given a mask, it tried to generate an associated audio spectrogram. This made the task a form of modality-driven segmentation: the image provides the scene, and the audio provides a cue about what should matter.

Validation samples generated by the CMCSSGAN model — Validation samples from the CMCSSGAN model. The generated masks were imperfect, but the model sometimes changed the highlighted region based on the audio target class.

The results were mixed in exactly the way early research projects often are. The ResNet version tended to overfit and was biased toward person masks. A more CMCGAN-like CNN architecture produced less sharp masks, but generalized better across the two target classes in validation. Mode collapse remained an issue in the audio modality, and the generated masks were far from production-quality segmentation.

But the basic idea did show signs of life. The model could produce sensible masks in some cases, and the target audio class could influence what part of the image was highlighted. That was the important proof-of-concept.

Looking back

The feedback I received on the dissertation was fair. The work was technical, the visualizations and implementation details were strong, and the dataset curation was useful. At the same time, some explanations could have been more formal, the outlook could have gone further, and the results needed to be interpreted more carefully. The project did not produce a polished cross-modal segmentation system. It produced a careful attempt, a failed reproduction, a dataset prototype, and a plausible extension worth testing more seriously.

There are many things I would change now. I would use stronger baselines, clearer quantitative metrics, larger and cleaner audio-visual pairings, and more modern architectures. I would also treat the dataset problem as central rather than supporting work. For this task, the dataset is not just plumbing; it defines whether the model has any real signal to learn from.

Even with those caveats, I still like the core question. How should a system use one modality to guide attention in another? How should it behave when the modalities are weakly aligned, noisy, or incomplete? This dissertation was an early, GAN-heavy pass at those questions, and it left me with a much stronger appreciation for how hard it is to make cross-modal learning work outside of tidy examples.