2023-07-21 - Testing #1 - The different types of testing - Chewing the Fat 📈

[lukecookssw]

Hey testers!

Let's talk about this rule

[leahy268]

I have seen automated testing from a distance, so it is definitely good to get a bit of an understanding.
Spent a lot of time manually testing things, though it was always very informal for me.

[brydeno]

To me the reason we test is to reduce risk of "failure". So testing effort should be commensurate with the risk.

[leehawkinsssw]

It's good to know that listening to me saying exactly this for 20 years has made it stick in your brain, Bryden!

[piers-sinclair]

Tests are critical and developers often don't have a great understanding of it. It is easy to make a mistake and introduce bugs into already functioning software.

From the Product Owners perspective it can be frustrating to see a developer introducing regressions. However, it's important to note that if you notice a pattern of regressions being implemented across multiple developers each time a problem is solved, then it may be caused by technical debt or poor testing practices. In that case, it would be wise to investigate if that is the case and what can be done about it.

There is also the team culture at play. The entire team needs to be bought in on the idea of your testing strategy and implementing it. Writing tests take a long time so if only 1 or 2 people are bought in then the rest of the team will pump out a lot more work and will look like they are doing a better job even though there is a higher chance for bugs.

[jakebayliss]

We should do another CTF series on the different kinds of tests :) Rule is a big wall of text and I think could do with some examples for each testing type

[lukecookssw]

Raise a PR :)

[MarcusIrmscherSSW]

I love the Glastonbury example in the rule 👍It shows how important load/stress testing is.

[sethdaily]

Good introduction to testing. It seems like there is lots of overlap between the types of tests, so I shall put my faith in the many people much better than me to know what to do and when!

[matt-goldman]

The rule is great! Although this comment:

Don't confuse test approaches & techniques (focused on the "how") with types of testing (the "what"). For example, exploratory testing - as an approach - applies well to several of the types of testing outlined above.

Is a huge statement for a throwaway box. This needs a rule and/or video itself!

[matt-goldman]

Also I am so bad at testing. I think it would be great if SSW ran a course on this, there's definitely a market for it and I would attend (even if I wasn't an employee).

[ozdemir-mehmet]

1 line of code, feels like 15 lines of tests.

[calumjs]

If you use GPT to generate all your tests, should you have meta-tests that ensure the generated tests are appropriate? 🤔

[Hona]

As a dev you're responsible for your usage of AI. If you get it to write a test - then break the case it is testing and it should go red. fix it & it should go green. Blindly letting it write a test that goes green & commit is just unprofessional

[wicksipedia]

Have a look at mutation testing - its a form of testing your unit tests to make sure you don't have checks that result in false positives

[Hona]

I am aware of it - is there a nice way to integrate it into a typical dev workflow? I'm curious maybe for a 'live unit test' but a 'live mutation test'

[calumjs]

@Hona Agreed that blindly using GPT generated content is obviously not a good idea...

My point though was whether there should be another layer of the testing pyramid for meta-testing - testing your tests to make sure that when it indicates all tests are passing, that is actually meaningful.

The GPT angle was just to spark some debate, but it could apply to human-written tests just as much.

[stephanfakossw]

Nice intro into testing! Love the stress test example 😊

[SamBWagner]

I think tests fall on a spectrum of delivery <-> resilience where the more you test the more resilient your code is, but the less you deliver. It's a balance that needs to be evaluated on a case by case basis. There is no 1 size fits all solution, as with most things.

[AlexRothwell]

With the huge variety of types of tests that occur for client projects, this is a great introduction

[vladislav-kir]

I would love to see a CTF series about detailed exploration of different kinds of testing :)

[UlyssesMaclaren]

Nice explanation that I could understand :)
I'm actually interested in what the answer to that reaction button question is... is it ALL THE TESTS!!?

[PennyWalker]

Me too, I looked and thought, well you need to know it's working -> smoke test, but it has integrations, definitely 'component', but you need to know your code works... 'unit testing'... then you need to know the over all thing works, is this where 'manual/auto tests' come in... Am I now doing too many tests to get any work done? 😁

[jernejk]

It depends on how deep the client's wallets are and how much time they are willing to give for a well-tested solution. 😂
Usually, we need to go with what gives us the best value in building it reliably and the best value in keeping it working in the long run.

[leehawkinsssw]

Am I now doing too many tests to get any work done?

The mental switch is realizing that creating tests is just another part of the work to be done and not some separate thing.

[tkapa]

I like the idea of testing but have trouble applying the knowledge, it's an almost entirely separate art to actually making the app. Mocking is one of the biggest downsides to testing, and it's been the biggest pain point for me every single time I've had to interact with it.

I think that the rule is highly detailed, but there is some extraneous stuff that could be left out or separated.

[Aibono1225]

Love the rule because it just gave me more insights about testing. I often have the feeling that implementing tests took too much time, but I also understand its importance in all cases. As we said, it's crucial to discover any bugs at the beginning rather than later on.

[johnxu16]

Spending much time writing unit tests to reach a high percentage of code coverage is unrealistic sometimes.

1. Not all functions worth writing a unit test
2. Devs cannot think of many cases thoroughly as testers do

Based on my working experience, the smoking test is extremely important since it ensures every deployment is functional. E2E is hard to implement and only be used on the mobile end. Regression-test will be required much more often than your expectation since the complexity of projects grows fast nowadays.

[tino-liu]

Just yesterday I wrote an unit test that covers a method for checking data payload that covers all the bad data I could think of. What it should do is expect an argument exception and pass the test. However, the test failed and the bad data went through. I found out later that there were cases I didn't initially thought of when developing the feature. So now I have a bug to fix. Just maybe this could be avoided if it was done in TDD style.
Testing in action!

[KristenHu]

I like the video, I didn't know testing well before, but it makes me know it more systematically.

[tombui99]

Testing is highly crucial in software engineering yet only few clients I have worked with spend time to write tests because of budget. With testing, we can easily identify bugs, ensure software quality, validate requirements and so much more. It guarantees a reliable and secure product, meeting user expectations and industry standards

[leehawkinsssw]

Just a bit of advice, Tom: use safer language than "guarantees" when discussing the benefits of testing with clients. No amount of testing - of any type - can guarantee a bug-free product https://www.ssw.com.au/rules/complete-testing-is-impossible/ Making stronger claims about your testing than you can possibly back up immediately puts you in the prime spot for blame if issues are found in production.

[zl24622]

Nice video. The first time heard about the Testing Pyramid, a few other knowledge learned from the rule and video.

I remember reading a blog in which the author says he is not doing Test-Driven-Development, instead, he is doing Testbility-Driven-Development. If he notices he has to write many tests or do a lot of mocks, probably means his class or function or even the whole architecture is too complicated or has too much dependence on each other. He would spend more time refactoring his code.

I feel his approach is more realistic because, in the real world, it's always a balance between time, and $ and quality. Most of the time we probably cannot write all tests, but can use tests to help keep a mindset for developing simple, clean applications.

[PothieuG]

Great video and nice summary in the rule !
I never hearded about Testing Pyramid/
I'm not a testing expert and it's still difficult for me to appreciate how far I should test the code or not.
I really want to see and learn all the testing strategies I saw today.

[drwharris]

I find people either

1. Ignore testing all together,
2. Do the bare minimum just to ✔️ a box
3. Go overboard and over complicate things leading to unreliable testing

People often forget the importance of keeping tests simple and isolated.

[william-liebenberg]

Looking forward to seeing when most of us write tests. Before, during, or after building a feature.

Are we a group of TDD purists (before) or do we think a bit differently and write code towards the user requirement first, then write tests to cover what we have done.?

[BrookJeynes]

I think testing is important but it's heavily dependent on what type of work is being done. Sometimes tests are too difficult to mock or are just simply not needed. Other times, having a strong suite of unit tests can ensure that over time as the application changes, the intended functionality remains the same.

[AttackOnMorty]

Unit test is essential for refactoring. It is a minimal guarantee that the behaviour has not been modified.

[Levijj22]

Great video! Simple explanation of what seems to be a vast and complicated topic.
The pyramid illustrates it well.

[Freego1783]

Nice to see a CTF on Testing, I know I lack some skills there.

I have often encountered clients who explicitly requested not to write any tests because they didn't want the devs to spend time on them and wanted the devs to focus on the features. Those client considered it was a waste of money, and obviously, they were impossible to convince.

[TobyChurches]

Great CTF! I love the breakdown of the different types of testing! This better summarised the information from a entire course at University 😁

[ozairashfaqueSSW]

Awesome CTF! I would like to go deeper in it