I don’t think there is an option to not load the audio at all; certainly in ELAN, when a media player is created for a media file which contains both video and audio tracks, the audio will be accessible and the only way to not hear it at all, is to keep it always muted.
I had another look into the VLCJ documentation and this issue of multiple players sharing the same volume controls is mentioned there. It applies to so-called in-process players (the type of players ELAN uses, the video(s) are embedded in ELAN).
Maybe the next versions of VLC and VLCJ will allow different approaches.
Best,
Han