Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't reproduce confusion matrix of attention weights for TvSum video 7, test split 2 #24

Open
tihmels opened this issue May 21, 2021 · 0 comments

Comments

@tihmels
Copy link

tihmels commented May 21, 2021

Hi,

I thought I would try my luck here...

I am working on your VASNet paper as part of a Deep Learning seminar. Since a few days I am trying to reproduce the Confusion Matrix of Attention Weights for TVSum video 7 from split 2, which is exactly the matrix you have shown in the paper.

Unfortunately I get a completely different figure, I'm running out of ideas. Here is my code:

self_attention = SelfAttention() # your SelfAttention(nn.Module)
self_attention.load_state_dict(torch.load(SELFATT_MODEL_FILE)) # SelfAttention model from split 2

with h5py.File(TVSUM_DATASET_FILE, 'r') as f:

    video_7 = f['video_7']
    features = video_7['features'][...]

    features = torch.from_numpy(features)

    y, weights = self_attention(features)

    weights = weights.detach().cpu().numpy()

    # Values were normalized to range 0-1 across the matrix.
    weights = preprocessing.minmax_scale(weights, feature_range=(0, 1), axis=0, copy=True)

    fig, ax = plt.subplots()
    ax.xaxis.tick_top()

    heatmap = sn.heatmap(
        weights_df,
        xticklabels=50,
        yticklabels=50,
        cmap="YlGnBu")

    plt.show()

And this is what the Confusion Matrix looks like:

confusion_matrix

It definitely loads the correct model for tvsum split 2 and the Attention Weights produced are definitely the same too, I checked it against your VASNet.

Does anyone have any idea why this looks like this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant