[chore] Add readonly matrix test. Remove invalid mutability assertions

[chore] Add readonly matrix test. Remove invalid mutability assertions #20006