Synchronizing multiple videos & controlling audio playback

ELAN
1

Hello,
I would like to synchronize five videos (2 with 29.97, 3 with 30 FPS) in ELAN 6.6 before annotating the project. There are several steps in which I encounter issues, maybe you can give me a hint regarding the cause for these issues and provide a better solution.

1. Synchronization:
I did not find any automatic possibility to synchronize five videos (or manual possibility to do so by waveform). There appears to exist only the option to synchronize manually by video playback (using media synchronization mode), also limited to four videos at a time. The media synchronization mode also appears not to support displaying the waveforms to synchronize by audio.
My workaround is to synchronize the videos roughly using another tool and then applying a start timestamp to each video using CTRL+G in media synchronization mode. However, the synchronization tool provides me with a frame number at which each video starts and not the synchronization timestamp requested by ELAN. Do you have any suggestions how to ease the synchronization procedure?

2. Validation of synchronization/Controlling audio playback:
To check whether synchronization was successful, I would like to play back the videos and listen to the audio tracks to find out whether there is still a temporal mismatch between the tracks’ audio. However, I fail to mute/unmute the audio tracks reliably: Even if ELAN indicates that all tracks are unmute and at full volume, sometimes only one of the audio tracks is played back. Conversely, muting one of the tracks most times causes the whole application volume to be set to zero in Windows Volume Mixer, effectively muting all tracks even though ELAN indicates that all tracks except one are unmute and at full volume. Is this a known bug and if so, how can I control the audio tracks reliably?

I’m thankful for any help.
Best, ASE

One more off-topic note for the website admin: My browser reports that parts of the websites (images) are not transferred via SSL and thus claims that the website is not secure.

2

Hello,

Yes, the synchronization mode is limited to 4 videos and it doesn’t show the waveform of those videos (only for separate wav files). The synchronization mode was originally introduced to sync two videos, later extended to four and to also support wav and timeseries files, but it is far from ideal.
If you have the offset per video from another tool, like in your case, it might be best to enter them in the table in the Edit->Linked Files… window (the Offset column in that table is editable nowadays). You’ll have to convert the frame number to a value in milliseconds, based on the FPS of each video.

A quick test with three videos didn’t immediately allow me to replicate the problem with the volume mixing, but I’ll try some more. If there is indeed a bug in the handling of the volumes per player, I don’t think there is an alternative approach inside ELAN (maybe checking four videos one-by-one with the first video, if that works, is the best solution?).

Best,
Han

PS thanks for mentioning the website, we’ll have a look at that.

3

One more off-topic note for the website admin: My browser reports that parts of the websites (images) are not transferred via SSL and thus claims that the website is not secure.

We have fixed the http issue now.

4

Hello Han,
thank you for your reply and for clarifying what is possible within ELAN.
I do have the same issue also when checking only two videos at a time so that workaround seems not to work for me.

Here some more information which hopefully help reproducing the behavior I encounter:
ELAN 6.6 is running on Windows 10.
In Preferences->Platform/OS, I use “VLC Player Library” as it is the only framework in which all of my videos are played back correctly (i.e., no black screen).

I will follow up with an example shortly.

PS @tla_admin Thanks for investigating the SSL topic. The warning still appears in my browser, maybe there is some more mixed content being loaded from the website:

mixed_content
mixed_content877×377 26.6 KB

If you need more information, let me know.

5

@hasloe Here more details for reproducing the volume behavior (please find a screenshot for steps 5-9 at the end of this post - I’m only allowed to upload one image per post):
Example:
Preparations:

  1. I downloaded a WEBM 1080p public domain video file from wikimedia (find it at commons.wikimedia [dot] org/wiki/File:Trail_Camera.webm) and created 4 copies of it.
  2. I opened ELAN and dragged all 5 video files into it.
  3. I added an offset of 1000 ms between each of the video files (via Edit->Linked Files).
  4. I opened volume mixer to see changes in the volume immediately.
  5. I double checked that the ELAN application volume in volume mixer is at maximum position and not muted. Within the application, the master volume as well as the first track’s volume is at maximum position, the other tracks’ volumes are zero but not muted.
    Note: In the steps 2 & 3 you may also just use two videos and leave the offset out, I just did these steps to represent my specific workflow accurately and to have the attached screenshots match the instructions.
    Test:
  6. I mute the second track. The ELAN application volume in volume mixer is instantly adjusted to zero. That is, at playback, I will not hear any of the tracks.
  7. I unmute the second track. No effect on volume mixer.
  8. I mute the first track. No effect on volume mixer (volume is already zero from step 6).
  9. I unmute the first track. The ELAN application volume in volume mixer is instantly adjusted to maximum position. That is, at playback, I will hear the first track at full volume.

The only option to override this behavior appears to set the desired volume settings within ELAN and “correct” the volume control in the volume mixer manually afterwards. Please note that the use of the volume control slider elements of a single track affects the ELAN application volume in volume mixer similarly, that is, the whole application’s volume will be adjusted.

Do you have any more ideas what I can do about this behavior? And if it is reproducible for you, do you think there will be a fix?

Thanks again for your help!

Best, ASE

Screenshots of the steps described above:

collation_gr
collation_gr940×2304 253 KB

6

Hi, we had to revert a setting that enabled full ssl, as this caused an issue with the login for users. This will hopefully be fixed very soon.

7

Dear Ase,

Thank you for the additional information and screenshot illustrating the volume behavior. This allowed me to better understand and reproduce the problem. On my Windows 11 system I see the same or similar behavior you describe, with this addition that after step 6, hitting the Play button in ELAN sets the system volume to the volume of the first player (I tried with just two video files). I think the cause is the same as what you already describe: the last volume set on any player (be it with the volume slider or programmatically when starting playback) will change the application’s volume in the system’s volume mixer.

I don’t think we see similar behavior with other players available on Windows, this seems unique or the VLC player in ELAN. From the top of my head I don’t know if the integrated VLCJ library allows to change this behavior, e.g. by setting a flag that setting a player’s volume shouldn’t change the application’s volume or if a separate VLC audio mixing component should be instantiated. We’ll have to look into that, for now I don’t see how this can be overcome in the current version of ELAN.

Best,
Han

8

Dear Han,
thank you for reproducing and confirming the behavior. I expected that I could overcome those limitations by muting/unmuting tracks in the right order and/or correcting the volume in the volume mixer afterwards. However, I just realized that I’m not able to mute or unmute single tracks reliably, it appears like it is really all-or-nothing. Is there a separate switch (besides the volume control) to not load audio of specific video files at all?

I will stay tuned to ELAN updates, hopefully it is possible to change the behavior as you described in a future release.

Best, ASE

9

I don’t think there is an option to not load the audio at all; certainly in ELAN, when a media player is created for a media file which contains both video and audio tracks, the audio will be accessible and the only way to not hear it at all, is to keep it always muted.

I had another look into the VLCJ documentation and this issue of multiple players sharing the same volume controls is mentioned there. It applies to so-called in-process players (the type of players ELAN uses, the video(s) are embedded in ELAN).

Maybe the next versions of VLC and VLCJ will allow different approaches.

Best,
Han

10

Thanks again for clarifying. It sounds like VLCJ-Pro achieves the desired behavior but is shipped with a commercial license. I’m not sure which exemptions may apply for educational purpose and whether it would make sense to add an adapter for VLCJ-Pro and let users obtain a license for it independently. There are probably more urgent requirements for ELAN than adding another media framework option…

To find a solution for my initial problem, I had following idea:
If the volume control behavior is framework specific, I may find another framework that is able to playback the media and also implements correct volume control behavior. With that idea, I 1) double-checked which player frameworks recognize the media I would like to playback and then 2) repeated the volume control test for them.

1. Double-checked which player frameworks recognize the media I would like to playback - results:

  • Java Direct Show recognizes the media - but only if “Microsoft Media Foundation” checkbox is disabled
  • JavaFX Media Framework does not recognize the media
  • VLC Player Library recognizes the media

For the non-working options (Java Direct Show with Microsoft Media Foundation enabled, JavaFX Media Framework), a blue window appears instead of the video, playback is not possible and the application log shows messages such as:

Oct 19, 2023 1:51:20 PM mpi.eudico.client.annotator.recognizer.api.RecogAvailabilityDetector createBundle
SCHWERWIEGEND: Cannot load the recognizer class: nl.mpi.recognizer.local.hecate.HecateRecognizer - Class not found
Oct 19, 2023 1:51:20 PM nl.mpi.jni.NativeLogger nlog
NATIVE: N_JMMFPlayer.getAspectRatio: cannot calculate the aspect ratio, height is 0, id 1845344653808
Oct 19, 2023 1:51:20 PM nl.mpi.jni.NativeLogger nlog
NATIVE: N_JMMFPlayer.getAspectRatio: cannot calculate the aspect ratio, height is 0, id 1845344653808
Oct 19, 2023 1:51:20 PM mpi.eudico.client.annotator.player.JMMFMediaPlayer getAspectRatio
WARNUNG: The aspect ratio is not initialized yet: 0.0
Oct 19, 2023 1:51:20 PM mpi.eudico.client.annotator.player.JMMFMediaPlayer getSourceWidth
WARNUNG: Width not initialized yet: 0

If you’re interested in a full log output, let me know.
According to older posts (1, 2) in the forum, there is currently no solution for playback failures associated with the aspect ratio message. Is that still correct?

2. Repeat volume control test with alternative working media framework:

I went on with the Java Direct Show (without “Microsoft Media Foundation”) option, hoping to have full control over the media volumes. I just reran the 8 steps with that framework. This time, results of steps 6-9 are similar to those I reported earlier for VLCJ except for playback - at playback, the behavior described by you (for step 6) occurs in all of these steps:

hitting the Play button in ELAN sets the system volume to the volume of the first player

Confusingly, after resetting the media framework to VLCJ and restarting ELAN, the behavior (including playback) does not differ from the one with Java Direct Show, that is, I can no more reproduce that the last volume set on any player will be used at playback time. I’m not sure why this is the case.

Besides this reproducibility issue, I would like to know:
Can you reproduce that volume control behavior also for the other media frameworks (not only VLCJ)? That is, the volume of the first track will be used at playback time for all tracks, effectively playing back audio of all tracks or of no track at all? Or is it more likely that for some reason, the switch between the frameworks in my local ELAN installation has not been applied correctly?

PS: I’m aware that my reply is mixing different issues (compatibility of media frameworks with some video files vs. audio playback of different media frameworks). If it is easier to proceed with each of them in separate threads in the forum or if more details are required, let me know.

Best, ASE

11

We could have a look at (an adapter for) VLCJ-Pro, to see what it would take to support users with a license.

Concerning your tests with different media frameworks: I cannot get the JDSPlayer to play .webm files. Disabling the Microsoft Media Foundation doesn’t make a difference: the JDSPlayer reports "Cannot play back the file. The format is not supported." in the log and the Player Info context menu of the video reveals it is actually a VLCJ player). DirectShow is a rather old media framework and I assume that without additional codecs installed, it doesn’t support webm.

With other media frameworks on Windows, I don’t see similar interactions with the system’s volume mixer (ELAN’s volume remains 100% there) or interference between different players (per player volume levels are maintained and applied). So, based on my tests I think that other behaviour is unique to VLCJ.

The problem of the video not showing up (instead a blue rectangle is shown) is not solved and the cause is not clear. Because there seems or seemed to be a connection with the aspect ratio not being recognized some additional logging was added. Because the creation of a JMMFPlayer is an asynchronous process, there will now almost always be one or two messages about the aspect ratio not being initialized, even if the player is created correctly in the end. And it can happen that the width and height are correctly read from the file (and all other functions return a "success" value), but the video still doesn’t show up.
On (my) Windows 11 I can now see something similar with the JDSPlayer (while this doesn’t seem to happen on (my) Windows 10, with the same version of ELAN).

Best, Han

12

Thank you for the update! If you come to the conclusion that such an adapter is feasible, kindly let me know.

Regarding the media frameworks: The .webm files were only intended as sample which is available online, my actual recordings are .mp4 and .avi. Your hint to look at the Player Info was very helpful anyways - I see that my local mp4 files are played back using VLCJ even after switching to another media framework. This in turn explains that the volume control behavior has not changed after switching between the frameworks and supports your conclusion.

Thanks also for the clarification regarding the blue rectangle. In my case the aspect ratio is never initialized (even though the display aspect ratio can be read by ffprobe). If I can help with data (e.g., sample videos or log files) to investigate the cause, feel free to contact me.

I really appreciate your dedication to provide support ELAN users. Even if there is not a perfect solution for me, your responses helped me to understand better what ELAN is capable of, how it works and which implications this has for my use case.

Best,
ASE

13

Thanks, ASE!

Yes, if you have a sample video which consistently doesn’t show up (blue rectangle, aspect ratio never initialized, maybe with logs), you can maybe send a download link to me (han.sloetjes AT mpi.nl). Although the problem can probably not (only) be explained by the encoding of the file (we have reports of the same files, in the same lab, show up on some of the pc’s and not on other pc’s), it’s good to try to reproduce and investigate the problem further.

Best,
Han

14

I have sent you a mail with a download link containing an example video and log files. Hopefully it is useful for reproducing the behavior.

Best,
ASE

PS: When creating the logs, I realized that the use of the JavaFX Media Framework option fails and ELAN resorts to Java DirectShow/JMMF:

Preferred media framework 'JFX' can not handle the file: The JavaFX based player requires Java 8 or higher