Create and publish an official Trilinos User Support Policy

[maherou]

The user support policy for Trilinos has been implicit from project inception. We need to make an explicit policy. Basic elements are:

• Expectations for the general community.
• Expectations for funded projects.
• Roles and accountability.

CC: @trilinos/framework

[maherou]

CC: @bathmatt, @rppawlo

[maherou]

I have completed a draft Trilinos Support Policy document. It is available for review. Please put comments in this issue.

[bartlettroscoe]

Referring to the draft at:

• https://github.com/trilinos/Trilinos/wiki/Trilinos-Support-Policy

in the sentence:

Any useful software product requires corrective and perfective maintenance

should "prefective" be "preventative"?

For the references in the sentence:

They are expected to follow Trilinos policies for Issue Management, Testing and Pre-push Testing.

I think we are making some progress in all of these areas but more improvement is needed (we can discuss the details of these three areas at future Trilinos Leaders Meetings and in other conversations).

The note:

This policy formally applies to any Production Growth package development. It is encouraged for packages in other phases (Research Stable or Production Maintenance).

Trilinos has not yet adopted a lifecycle model that assigns or uses these maturity levels in any detectable way. That suggests that we need to implement the plan at:

• http://trac.trilinos.org/wiki/TribitsDevelopmentPractices#maturity_level_practices

Should we create a Trilinos GitHub issue for Trilinos to adopt this lifecylce model?

Other than that, these general policies seem very reasonable to me. If a major funded project has Trilinos support needs, it needs to provide some funding for those needs and create a plan for doing so involving Trilinos Framework staff and funded Trilinos developers.

But there are a lot of issues that are not covered here that will need to be addressed such as:

1. How to provide support for (orphaned) Trilinos packages with no active development and where the previous developers for those packages are long gone or no longer available to help with their support?

2. How to the new Trilinos Product Leads map or relate into this support policy? What roles should they play (if any)? If they plan no role, how can they effectively lead their product areas (and the packages that fall under their area)?

Also, note that an important aspect for support is even being able to map between Trilinos GitHub issues and CDash builds and funded customers. Therefore, I think that as part of this support story, we need to complete:

• https://sems.sandia.gov/jira/browse/TRIL-174

Otherwise we are going to have a hard time tracking support issues for important funded customers on this Trilinos Github site and implement this support policy.

[maherou]

cc: @wfspotz, @dridzal,

[maherou]

Responses to some of Ross' comments:

• I meant to write perfective maintenance (as it is written). This term refers to improvements that come from user experience.
• Yes, the referenced policy documents need further development, but everything is a work in progress. We can mature all documents in parallel.
• While we may not have formal adoption of a lifecycle, we have a working definition of the phases. In fact, the Trilinos Package Classification spreadsheet (in the Trilinos Project Google Drive folder) lists the phase for each Trilinos package.
• I think we should wait on a lifecycle model initiative. There are other higher priority needs right now.
• I can add statements for packages in other phases, and for product leads. These are good observations.

[srajama1]

@maherou : I am worried about this statement. "First level support will come from funded project team members who are part of the Trilinos team". May be by first level support is just PoC ?

[bartlettroscoe]

I am worried about this statement. "First level support will come from funded project team members who are part of the Trilinos team". May be by first level support is just PoC ?

Just being a POC is fine as long as there are people with the time and funding exist to pass off issues to.

[gflofst]

I am a bit worried about these statements:
"Any improvement in support will have significant impact on quality, cost and time.

Therefore, support is best done by the team members developing capabilities. Support and support planning should be an integral part of development efforts."

It is hard enough to convince everyone that they should follow some process even though it is a research rather than production software effort. These statements push it farther into the production realm, from what I read. I would push in a bit of a different direction with the same goals. Can this be restated towards the sustainable software/reproducibility goal to offer reasonably documented intent to make it easier for more than the original author to support the code? I have had people I've worked with over the years that have quit once they realized that they were the sole support source for what they had written years earlier. I'd like to see this as more of a carrot approach encouraging good practices to reduce the support incidents rather than declaring that, "you wrote it, you are the front line of support for it--forever".

[maherou]

Addressing Siva, Ross and Jay comments on the support role:

• Yes, the Trilinos team members who are members of the funded project are the points-of-contact. Whether they are the people who actually fix issues is a separate decision. I can clarify this in the policy statement.
• There is no implication that, if you wrote it, you will forever support it. Again, I will clarify.
• Having said this, it is best for the developer of a new piece of software to be the person who directly supports fixes. There is no one who is in a better position to do this work. Trilinos developers should (and already do) fix their own errors when actively working on a piece of software.

Please let me know if I missed any point.

[bathmatt]

Questoin on this statement "Trilinos support will be an explicitly planned and funded activity within the funded project: While the Trilinos project has dedicated support staff, staffing and funding for support must come from aggregate resources provided by funded projects. Specific project budget amounts should be set aside for support."

Are you saying that as someone develops a package as part of their LDRD project, they should budget in support and bug fixing? I don't really know what a funded project is, I read the description and it sounds like only if the funding is tied to an application then it is direct funded? I see an issue if this doesn't apply to the full trilinos stack. Here is an instance. Oh, that feature wasn't developed under direct funding.

Also, where does QoS fit into all of this? Is there a response time? One of the reason we don't want to use STK is that if an application other than Sierra has a bug or a feature gets deleted, it can take months/years to address.

Maybe all this is a different issue, but it needs to be addressed.

[jmgate]

Question on this bit:

Therefore, support is best done by the team members developing capabilities. Support and support planning should be an integral part of development efforts.

Do Trilinos developers have any resources/training available to them on best practices in this area? Something along the lines of "What might I not be doing now that I should be to make everyone's lives easier down the road?" Guidelines on code documentation, user manuals, issue prioritization/juggling, etc.?

[bartlettroscoe]

Trilinos also needs some explicit policies about support for newer compilers. Trilinos frequently gets bug reports for problems with Trilinos for newer compilers for which Trilinos lacks automated testing (e.g. #1272 (comment)). Here are some examples of the language that might exist in such a support policy:

• Trilinos developers will do their best they can to address issues with newer compilers but such issues are not going to be given very high priority until an important Trilinos customer requires that compiler.

• Trilinos will try to get automated ("Experimental" or "Specialized") builds for Trilinos set up in a timely way as new versions of major compilers are released (e.g. Intel, GCC, Clang, etc.). Then issues with Trilinos and those compilers will be addressed and cleaned up in a background process over a period of time until they become clean. Once they are clean, then they can be moved to the "Nightly" track (and eventually to the "Clean" track if that compiler becomes critical for an important Trilinos customer).

• Trilinos has no official support policy for compilers where there are no automated builds for Trilinos. (e.g. if there are no GCC 5.4.0+ builds, then we put off fixing any issues for that compiler until we are compelled to address issues for that compiler in some way.) However, if a user submits a complete PR (with tests, etc.) to address an issue for one of these compilers, then Trilinos might consider adopting them.

The absences of such a policy is bad for developers and users of Trilinos.

[ibaned]

It would be good if compiler support were aligned with the default compiler used in Linux distributions in which Trilinos is a package.

[bartlettroscoe]

It would be good if compiler support were aligned with the default compiler used in Linux distributions in which Trilinos is a package.

I think that is a better trigger for when to start trying to test the free compilers like GCC and Clang than just trying to select the newest compiler releases as they come out. That way, at least we know that these compilers are passing the Linux and Mac OSX distributions own tests.

But we would need to come up with a policy for new compiler versions of commercial (non-free) compilers like Intel, IBM, NVCC, etc. Those will not be as much directly attached to any particular OS distributions as much. It think it will be harder to come up with reasonable policies for those. For example, I know that SIERRA has had problems with some certain releases of the Intel compiler so just because a new Intel compiler is released does not mean that we necessarily should jump on it. So we need some other reasonable trigger for when to try out a newer version of a commercial compiler.

I think having some explicit policies about support for newer GCC, Clang, and Intel compilers would likely address the concerns of 95% of the Trilinos users so that is where we should start with such a policy.

[bartlettroscoe]

I think we need to create detailed policies for specific customers to show how the rubber meets the road.

[maherou]

I have updated the Trilinos support policy based on discussions at a planning meeting last week. The primary change is that we are asking co-funded users who encounter support issues to send a message to [email protected] if it is not clear who a direct contact point person should be. This approach should provide a low barrier to getting support. We guarantee that issues sent to trilinos-help will be entered into an issue tracker and assigned as quickly as possible, and will be tracked.

Hopefully this change, along with clarification of other statements, will be acceptable.

Note that one outcome from this policy is that the Trilinos team work with funded app projects to have a customized support policy. It can be short, but should spell out details specific to the project.

@bathmatt , @mwglass please let us know what you think.

[bartlettroscoe]

FYI: The changes the @maherou made to the support policy can be seen here.

Some of the blocks of text mostly got moved around but that is not hard to see.

[hcedwar]

Packages in the "Production Maintenance" state (using as a term of vocabulary as the life cycle model has not been officially adopted) typically do not have funded projects from which a "support tax" can be leveraged. Support is problematic when applications rely on such unfunded "orphaned" packages. What support policies can be defined for these? Should the owning org commit to either supporting such packages out of overhead or officially retire the package?

[maherou]

I have added text about how to support Production Maintenance code. We have received feedback from select users that the current policy seems reasonable. So we will close this issue and move forward with implementing the policy.