Great post about testing, it's pretty much all you need to develop a robust app.
BUT TeamCity UX is the worst UI from all CI systems that I have used. From first version of Jenkis to managed stuff like CircleCI. People hate is so much that someone added a Slack icon :teamshitty: at work.
I recently had to switch from Jenkins to TeamCity, and while my very first thought fiddling with TC was that it's unnecessarily complex for my needs, I found myself creating the build jobs much faster and more straightforward than with Jenkins.
Jenkins plugins are very often deprecated/unmaintained with tons of security vulnerabilites and a bad UI/UX which results in inconsistent UX when using several plugins.
Also, with Jenkins you're forced to use groovy for pipeline code :/...
Groovy seems to get an awfully bad rap from Jenkins. Seems about 95% of people encounter it there in a broken state and assume the language itself is that bad. I really wonder why they have deployed it in such a way as presumably a build server can do pretty much anything so why it would be necessary to constrain groovy scripts behavior I can't understand.
I've used TeamCity (the UI has changed many times), Jenkins (including blue ocean), Bamboo, a few others... TeamCity is by far the best I've experienced.
Jenkins seems to suffer from too many cooks/anything goes mentality. It's clearly not well focused. It's feels like a handful of pet-projects lumped into one umbrella.
When it comes to functionality, TeamCity is constantly improving and innovating.
Based on the OP, I suspect their experience is with an old version or maybe misused or misconfigured.
At first, I thought TeamCity's UI was batshit crazy. In retrospect, I think it was just that it's not organized at all like Jenkins is, and I needed to re-adjust my mental model for how the build server should work. Or something like that. All I know is that, at some point along the way, I came to use TeamCity to automate more things, while still spending less time futzing with it than I did with Jenkins.
That said, I feel like TeamCity is also distressingly full of partially implemented features and flaky UX. It's just that its warts tend to accumulate on the periphery, rather than directly between me and the thing I'm trying to get accomplished before I can go home.
I hated Jenkins on first sight but loved Teamcity. Its model is exactly how I imagined an ideal CI system to work. We use it in a minimal way though - a common git+ssh+make workflow that gets inherited to all project builds with only 3 params set in case a default doesn't work.
So, I think it's about mental model as you wrote, plus where your techstack is in terms of unification (containerization helps there tremendously).
Maybe not worse than the "first version of Jenkins" but it's quite bad and I have never ever heard "I like TC UI" from anyone in any job, they all disliked it or outright hated it (myself included). And after so many years, it's still bad and complicated. The base is solid though, it'd just be so much better with proper UX.
I like the TeamCity UI. There I said it and mean it.
Sure, TC has had issues (especially when it came to scaling), but they fixed a lot of those and honestly, it's doing well now.
Issues usually came from people who didn't know how to make robust pipelines or leverage TC's functionality and felt like they had to reinvent everything. And that was in a company with thousands of employees, thousands of TC configurations and hundreds of agents.
To be fair I expect that somebody outside my circle would like it at least! Now that being said:
>Issues usually came from people who didn't know how to make robust pipelines or leverage TC's functionality and felt like they had to reinvent everything
Blaming the end user is almost never the answer, even less when the average TC user is somewhat technical or very technical. If a tool throws hundreds of things to your face and call it a day, then people will end up reinventing everything since they didn't have a chance to discover how to make sense out of it (without tons of free time and guessing that is). I haven't used the very latest version though so things may have improved further but I believe it'd require a major overhaul to put things in order.
True, but all CI systems allow you to run arbitrary shell scripts. This is often the source of e.g. hard-to-debug broken builds, because you can’t necessarily run the bits locally, or at least it’s inconvenient.
I think you should consider that groupthink gets established very easily inside companies. It only takes one brash but senior engineer (even if only in the "been there a long time" sense) to loudly proclaim a tool sucks and how much they hate that tool, and pretty quickly you can get a chorus of "me too" and people who disagree just decide they have better things to do than pick a fight.
I've seen this happen a bunch of times, and when investigated the opinions are often not based on much thought, or are actually flat out wrong.
My company has just migrated from TC to Jenkins except for the project I run, which is still on TC. Why? Because I've actually read the TC user guide and so understand how it works, what it can do. Plus I like the GUI.
The justification for migration was idiotic, something like "Jenkins has a big community, look at how many plugins there are". Well guess what - Jenkins has more plugins than TC because it needs more to match the feature set. A lot of things that are built in with TC are unmaintained crappy plugins in Jenkins that you have to assemble yourself.
The other day I saw one of the instigators of this migration complaining on Slack that the build queue priorities plugin they were using had gone unmaintained and had serious bugs. Build queue priorities are built in to TC for a long time and JetBrains doesn't ship broken features very often, so my guess is it works. I had to resist engaging in a told-you-so moment ... why bother, the damage was done already.
In another case another senior guy decided on nearly his first week that a major piece of software we use sucks. For years he has constantly made claims of the form "$PRODUCT can't do X" or "feature Y is broken" and when the user guide is checked, in fact it can do X and when asked what is broken about feature Y he can't point to any actual concrete bug reports he can or will file. But nothing will shake his view on this and other junior devs pick it up and echo it to make themselves seem discerning and informed.
Time and time again I've seen stupid product choices in companies done for no better reason than someone formed a snap opinion without bothering to read the user guide. TC works great. It's one of the best CI systems out there. Spend an afternoon reading the user manual from cover to cover and it will pay huge dividends - also note that it's free if you only need 3 agents.
If you don't read the documentation, sure, it's hard to guess how things work, but it's hardly the responsibility of the tool maker to change that.
If people try to reimplement things in arbitrary broken shells scripts (the user issues) that can't be run locally, then it's not the CI tool's responsibility either.
That's surprising to me. I used TeamCity for ~4 years at my last job and thought it was pretty decent (compared to experiences with Jenkins, Circle, and Travis).
It's because they never mention doing any actual usability testing or user research. Yes, their software can "perform" across all of these tests, but if the information architecture isn't intuitive or doesn't match what people want, it's not going to be good.
But TC is in the middle of replacing their GUI with a totally new one, the 'experimental' UI, which each release fleshes out in more screens. It's a full redesign.
So yeah perhaps they'd have done the new experimental UI in one fell swoop if they had less testing, who knows. But I'd rather they have a well tested product than ultra-fast UI replacement.
Personally, prefer it over Gitlab much easier to use. Find the documentation much clearer compared to Gitlab. Do like the pipeline yaml but something similar is possible with Teamciy but then its Kotlin I think.
I didn’t get to use TC for long, but I can say that none of them has pissed me off as much as Bamboo. Have you used it much and if so, how does it compare for you?
As somebody who does like TeamCity - it can be a bit clunky to navigate around. It’s much like Zabbix in that once you figure it out it’s not a problem, but the learning curve for anything beyond basic use can be steep.
I pretty much use Gitlab CI these days for anything outside my day job though, and the stuff at work is all Azure Pipelines (via Azure DevOps) now.
Variable scope and templates can get a little weird.
At work we're experiencing several issues with triggers that don't kickoff when there are changes in the repo.
Builds stuck for 2 hours or more on "Collecting changes".
Builds that build from the default branch, even though they were called with a specific branch to build from.
I could go on.
I personally don't have any problems with the TeamCity UI. I've used it in anger over extended periods of time. It's appropriately complex given what it does, and presents things efficiently.
The snapshots part is something that I completely disagree.
> Here, we change the engine type from turbofan to propfan. Just to test how it works. Since the new engine no longer matches our snapshot, the test fails. We’ve got a report, and our engineers are on their way to investigate the problem.
I would be really happy if I know that the onboard computer has tests that checks if the engine being used is compatible before it takes off.
IMO, tests in general should be as close as possible to the real use-case ( kudos for the screenshot testing ) and as far as possible from the implementation. What happens with snapshots is that you are bound to the implementation. Maybe I have a logic somewhere in my code that does s/propfan/turbofan/. Would a user be interested in that or the functionality itself?
We ditched all JEST's snapshots statements in our code-base, because of code-coverage abuse and false sense of "being safe" just a couple of months ago. It was a good decision that helped us be closer to "what the user sees" rather than "what the markup should be"
It's more akin to file diffing. Developers are notified of any differences in snapshots. The developer can they either accept the new snapshot, or investigate and fix.
Sort of - It ensures that your logic still generates the same output as before, so perhaps a step after the schema and a little closer to the user. I think people take it to mean the output will still look the same for the end user, though. That's rarely the case since markup is typically styled by CSS. CSS changes will largely be missed in the snapshots unless a class name is added, removed, or updated in the markup.
For E2E testing, is the browser still totally necessary? I feel the React rendering test libraries (React Testing Library/Enzyme) manage quite well? I have used Selenium and that ilk before but find you get odd failures every now and then (i.e. browser problems but also resources problems when you run parallel tests).
Also, I am sure this will be pointed out several times, but snapshot testing doesn't seem like a great idea. Your test is wrapped totally around your implementation and it is kind of unclear what you are testing i.e. change something, run test, snapshot fails because of UI update, update snapshot, passes...if you introduce some kind of unexpected bug with that change, how do you know? It works if you are testing for the same result but if you are making changes to the snapshot, you lose all coverage...tests are just as important then too.
Tbf, you usually end up wrapped around your implementation anyway (i.e. checking to see if there is a button that has certain text) but I feel that snapshots are a bit shortcut-y and give false confidence.
Surely the point of UI tests is to really test the functionality from the point of view of the user...I don't think snapshots achieve this (and I have always got in trouble when I strayed away from this principle...personally).
Tests are a trade - you spend some dev time writing and maintaining tests, and in return you might save some time and money down the line, if the test prevents bugs from doing damage. Tests aren't always good. There are tests that are so difficult to write and maintain, that they basically never repay their dev cost.
All that is to say, I like snapshot tests. Snapshot tests can be created and updated automatically, basically making their dev cost close to 0. They are so cheap that they don't actually have to catch a lot of bugs for them to be a net positive.
Anyone have experience with successfully doing UI tests (especially screenshot tests) in a very young software product? We try to keep our "plumbing" code and backend code well tested, but the UI is changing all of the time and I'm quite worried it'd be a nightmare to maintain UI tests at this stage.
(Selenium project founder here.) Test maintenance cost is a sane thing to be worried about! Early on, there can be lots of code thrashing and UI changes. Better to keep the UI tests to a minimum at that stage - have them act more like smoke alarms. You only need a few alarms in the right places to tell you that something is wrong.
Other general advice is that in a young product, it's often more important to make sure you're building the right thing. ("Right" is subjective, but could be defined as "has users" or "makes money".) Having a well-tested product that has no users and/or makes no money is an even bigger nightmare than UI test maintenance. Once you know you've built the "right" thing (hopefully sooner than later), you can afford to invest more in all kinds of test automation (not just UI tests).
It's not quite the same as screenshot testing, but I've had success using snapshot testing in React. The idea is outlined here:
https://jestjs.io/docs/en/snapshot-testing
Snapshots of the intermediate representation (e.g. JSX in React) will be less brittle than e.g. a screenshot of pixels. It also works best when your UI is a function of some easily-constructed state. Other frameworks sharing these features would also work well with snapshot testing.
I use webdriverio, which has pretty straightforward screenshot plugins. Honestly early on it's better to focus on API testing. My advice is start small for the UI and focus on a few critical components like page routing or features that would be a show stopper if they broke. I don't think screen shot tests will be of great value if your product is not very mature. When I first started I made the mistake of blindly following my predecessors user stories that weren't particularly well thought out.
> For example, a huge amount of UI problems we catch belong to the Screenshot testing stage. Fewer problems belong to the Linters / Unit / Render tests. That doesn’t make those tests meaningless. On the contrary, it could mean that we work in these areas well enough to prevent tons of issues.
They don't say how they measure "issues" in their chart, but if it's through CI failures, it seems likely that "Linters / Unit / Render tests" catch fewer issues because developers run them locally before pushing code.
I'm curious what kind of practices people have put in place to minimize the need for UI tests? I personally believe a UI test should only be written as a last result if it's impossible to test through other more backend-ish means. When you combine concurrent test runs with shared mutable state, you get flaky unreliable tests, and it's almost impossible to avoid with UI tests. So it seems the better solution is to restructure your code as much as possible to get as close as you can to the UI layer as possible with unit tests. But how far can you actually push this? I'm currently sad because my team is in the middle of a push to integrate Cypress into their react frontend, and I'm dreading the future where we have a billion qa-written UI tests that are redundant compared to our unit test coverage, and flaky besides.
A billion tests most certainly won't be useful. Good tests target very specific things that are critical to your product. In my experience UI tests are only slightly more flaky than other testing methods as long as they are written well.
As a mobile developer I read this eagerly looking for any useful ideas to bring UI testing into my client projects. I’m digesting still but so far, no.
I’m intrigued by screenshot testing but can it work across all screen sizes, from iPhone 6 to XR to iPad, and work in all orientations?
Otherwise my problems are that our designer made tap pinned too small in a couple places without anyone noticing. Or that something isn’t perfectly aligned or a line separator isn’t long enough, etc. they always seem to require a human to use it and say, I don’t like this or this could be better.
These screenshots are generated from the test suite for all my UI tests. I have puppeteer navigate a series of steps and save each screenshot, resizing to mobile resolutions. This is done in Chrome, Puppeteer also supports Firefox. The screenshots are then integrated with my documentation both as sitemaps like above and to demonstrate usage:
In addition to being user-friendly for documentation there is tremendous QA opportunity to being able to observe every page at every resolution. Everything required better code too, you cannot forget to link to a page or have some part of the navigated route not working or have anything going wrong anywhere.
I am finalizing localization so in the next week or so the documentation will also be able to switch between any combination of language and device.
Note: some of my documentation/screenshots hasn't generated yet
The Screenshots Diff section was amazing! Learned a ton from this section alone as I wasn't even aware you could do something like this for UI testing.
Meanwhile the TeamCity classic GUI has frequent freezes that lock up the entire Browser tab, and they claimed that they fixed it but it's still happening in the latest release. Nice blog post but fix your shit please.
BUT TeamCity UX is the worst UI from all CI systems that I have used. From first version of Jenkis to managed stuff like CircleCI. People hate is so much that someone added a Slack icon :teamshitty: at work.