Cross-browser testing, without the browser.

Reading time:- 10 mins

Follow along @ https://github.com/YOU54F/react-ts-testing-example

Consider the following statements that you might come across as a tester on a web app project.

  • Must be compatible with latest browser versions
  • Must be compatible with mobile

Pretty vague right? There are lots of browsers, and lots of versions. My definition of latest might be different to yours. Reminds me of this gem.

A small list of of some of the top browsers, but there are plenty more.

  • Safari
  • Safari iOS
  • Opera
  • Internet Explorer
  • Internet Explorer Mobile
  • Edge
  • Firefox
  • Chrome

When they say mobile, that’s a wide range to cover. They didn’t even mention latest there. There are quite a few operating systems, and a multitude of versions, by dozens of manufacturers. The stakeholder could expect their dusty old iPad 1 & Windows 8 touchscreen tablet’s to work.

  • Apple iOS
  • Google Android.
  • Windows Phone
  • Palm OS
  • Symbian OS

Let’s see if we can help scope that requirement down a bit, to something a bit more manageable.

Know Your Audience

Some of the most popular sites for checking the worlds browser stat usage are

https://www.w3counter.com/globalstats.php

http://gs.statcounter.com/

Both are excellent for noticing trends in usage but if you base your testing strategy for cross-browser testing on this alone, you will find that you may be targeting the wrong market.

Chrome’s market-share dominance won’t ring true in an office full of locked-down corporate boxes with Internet Explorer.

If your company doesn’t already have analytics which records site visitors, Insist that they do. It will be pivotal in ensuring your product will work, for the people who want to use it by providing tailored facts about what devices are in use for your particular product.

The challenges for web developers and testers alike

With the advent of HTML5, SCSS, Javascript ES5 & 6 with their snazzy new tool-belts, many old browsers (and not so old) are left in the dust. A promise here, and object.assign there and before you know it, you’ve just stopped anyone accessing your website in IE11 and earlier versions.

Oops.

With so many browsers out there, and more importantly so many versions each supporting different feature-sets, it can be hard to keep track, and even harder to test.

We could buy a load of real devices (logistical nightmare) or test in the cloud with a device provider such as BrowserStack / SauceLabs / TestingBot (pricey).

Both of those approaches, even if we use automation, give us slow feedback. What if we could test in 800+ browsers when our developers are writing their code. It is no substitute to UI testing, but augments our dev/testing process nicely.

We can give our developers a helping hand, and help mitigate risk as testers, so we don’t have to have such a reliance of cross-browser testing via traditional methods.

Can I use it? Yes, you can!

Some great people have compiled maintained matrices of supported HTML/CSS & JavaScript features against browsers split by version. These sites should be bread-and-butter to your front end developers, but if not, please point them in the direction of them.

ECMAScript 5/6/7 compatibility tables

http://kangax.github.io/compat-table

Browser support tables for modern web technologies – HTML5/CSS3

https://caniuse.com

Gotta catch ’em all, gotta catch them early.

Software defects cost money. How much money usually depends on where it is found in the software life-cycle. An ambiguous requirement that could otherwise be firmed up into a set of agreed testable criteria before any code is written, may become something entirely different from what was envisioned, when unveiled.

I said before, we aren’t going to use any browsers for our testing, and although we want to test early, we can’t test before any code is written, as the technique we are going to apply is static-analysis.

Static analysis, also called static code analysis, is a method of computer program debugging that is done by examining the code without executing the program. The process provides an understanding of the code structure, and can help to ensure that the code adheres to industry standards.

Wikipedia

We can utilise some tools, to cross-reference our code against the compatibility databases, targeting only the browsers and versions we are interested in.

Our developer can run these locally to keep them on track, and we can add these checks to our CI pipeline, to ensure that any new code is validated against these databases.

So hopefully by now, we have compiled a list of browsers we want to support, either by analytics, or by discussion and agreement with your stakeholders.

We are going to use a tool called browserslist to define our list of browsers to our tools.

https://github.com/browserslist/browserslist

Add it into your project as a dev dependency

$ yarn add browserslist --dev 
or
$ npm install browserslist --save-dev

Add browserslist config to your package.json

"browserslist": [
">0.2%",
"not dead",
"not ie <= 11",
"not op_mini all"
]

This command will list the currently targeted browsers in your project, based in your defined list above

$ npx browserslist

CSS Checking with StyleLint

Stylelint-no-unsupported-browser-features – This plugin checks if the CSS you’re using is supported by the browsers you’re targeting. It uses doiuse to detect browser support.

https://github.com/ismay/stylelint-no-unsupported-browser-features

Add it to your project (along with stylelint and the standard-config)

$ yarn add stylelint stylelint-config-standard stylelint-no-unsupported-browser-features --dev

or

$ npm install stylelint stylelint-config-standard stylelint-no-unsupported-browser-features --save-dev

Setup your stylelint config

$ touch .stylelintrc
{
"extends": "stylelint-config-standard",
"plugins": [
"stylelint-no-unsupported-browser-features"
],
"rules": {
"plugin/no-unsupported-browser-features": [true, {
"severity": "warning"
}]
}
}

Run it with

$ npx stylelint 

JavaScript Checking with esLint against caniuse db

eslint-plugin-compat – Lint the browser compatibility of your compiled JS code

https://github.com/amilajack/eslint-plugin-compat

Add it to your project

$ yarn add eslint-plugin-compat --dev
or
$ npm install eslint-plugin-compat --save-dev

Add the eslint config to your package.json

"eslintConfig": {
"parser": "babel-eslint",
"plugins": [
"compat"
],
"rules": {
"compat/compat": "warn"
},
"settings": {
"polyfills": [
""
]
}
}

Run it as follows

$ npx eslint 

Javascript Checking against ECMAScript 5/6/7 compatibility tables – 2 options

Compat.js – Static analysis tool for detecting browser compatibility issues in JavaScript and HTML.

https://github.com/jgardella/compat

Add it to your project

$ yarn add compat-cli --dev
or
$ npm install compat-cli --save-dev

setup your config file but note this tool does not use the browserslist configuration, hence listed versions, replacing <path/to/js> to where your bundled javascript lives.

$ touch .compatrc.json
{
"target": "",
"ignoreFeatures": ["Object static methods"],
"jsEnvs": ["ie11","chrome74",
"edge16","firefox67","safari12_1",
"ios12","samsung8_2"]
}

Run compat test with options defined in above config

$ npx compat

Passing the supportedEnvs flag will show available browsers

$ npx compat --supportedEnvs

Compat-CLI – ECMAScript 5/6/7 compatibility tables CLI

https://github.com/kamilogorek/compat-cli

Another CLI client that does the same as the above too, I haven’t tried this yet.

So now we’ve found some issues? What can we do?

Polyfill.io – A polyfill, or polyfiller, is a piece of code (or plugin) that provides the technology that you, the developer, expect the browser to provide natively. Flattening the API landscape if you will. The following website will list what polyfill’s are required for a particular feature.

https://polyfill.io/v3/

Autofixer PostCSS plugin to parse CSS and add vendor prefixes to CSS rules using values from Can I Use. It is recommended by Google and used in Twitter and Alibaba.

https://github.com/postcss/autoprefixer

babel-preset-env – a smart preset that allows you to use the latest JavaScript without needing to micromanage which syntax transforms (and optionally, browser polyfills) are needed by your target environment(s). This both makes your life easier and JavaScript bundles smaller!

https://babeljs.io/docs/en/next/babel-preset-env.html

Modernizr – A JavaScript library that detects HTML5 and CSS3 features in the user’s browser

https://modernizr.com/

Ok, so I don’t need to test on an actual browser, just mash polyfill into everything?

Not quite, trying to support every browser with lots of polyfills will leave you with a bloated web-app and no-one likes a slow site, you will have non-functional requirements to meet as well.

We haven’t performed any functional testing here on our UI, and there is no substitute to performing automated cross-browser testing of key journeys for confidence.

However you won’t need to rely on them so heavily, to keep you informed of browser compatibility and version inconsistency, so they remain lean, quick and useful rather than the painful entanglement that they can become.

Cool story bro, but wheres the code?

I have created a simple react website in typescript with some features not supported in older browsers, it incorporates the tools discussed in the article with a working example you can experiment with and build upon.

It also showcases some other UI testing tools which will be discussed in further articles.

  • Unit-testing React Components with Jest & Enzyme
  • Code-coverage with Istanbul
  • Unit test reporting with Jest-Junit & Jest-Stare
  • UI browser testing with Cypress.io
  • UI test reporting with MochaAwesome
  • Alerting via Slack
  • CI integration with CircleCI

It is available on Github – https://github.com/YOU54F/react-ts-testing-example

Protecting your API development workflows with Swagger/OpenAPI & Pact.io

Reading Time:- Get a cup of tea and pull up a beanbag, I’m going to say 15 minutes

Background

API’s (Application Programming Interfaces) have been around since 2000. I remember utilising my first one back at Uni, based in Java to modify a Lego Mindstorms NXT robotics kit to run an Artificial-Neural Network.

I dropped into the world of micro-services 5 years ago, along with a team of great engineers helping to define a service blueprint that could be used as a framework for new service providers.

I took this new found knowledge onto my next employer, who were setting up a new micro-service architecture to serve a web front-end, whilst communicating to a legacy backend through an abstraction layer, which was just another API.

We employed consumer-driven contract testing but with little collaboration from dependencies, we ended with CDC tests solely maintained by consumers which only served to protect us from changes of which we had little visibility, but it works better in collaboration and offers more benefits, than hinderances.

The aim of this article is to provide you with some tools and techniques to aid and promote collaboration with your API development workflows in order to increase the quality of your product. (After all, I am a Software Tester!)

Using Swagger to document your API

I am a big advocate of using hand-crafted Swagger definitions of each service as one of the first outputs of our developement.

Why?

  • You are possibly in an Agile world and you want to start rapid prototyping.
  • Documentation is lean – Everyone has probably laughed at you when you suggest that we write the specifications for the API right off the bat rather than 100 discussions that always have little output (see next point), let’s make it cheap and easy to change so it’s not as onerous.
  • Getting multiple teams together is difficult and often non-productive – your team and the teams you integrate with don’t want to sit in long meetings about boring things like what data-type a field should be.
  • Contracts are key – If we can agree what the request/response will look like, we don’t care about the underlying implementation, as long as it honours them. It also doesn’t mean they are set in stone, as a fixed artefact. They can change as the project develops. It also makes testers happy, as we can plan for integration testing early.
  • Swagger is cheap to write, anyone can do it, for free. – https://editor.swagger.io/

So your team want’s to start writing code and the provider hasn’t come up with an API contract yet? ( They are probably in long boring data-field style meetings ).

Fine. Let’s document the API we want to see with https://editor.swagger.io/ and save the JSON or YAML version to your machine, stick it in version control and give your provider access. Swagger will provide validation and auto-completion as you type to ensure that the swagger you have written, can be developed. ( A big bonus, I can assure you! )

They are chuffed now as they have a view as to what one of their consumers’ wants, documented in a validated specification. They can make changes as necessary, and via a pull requests, collaboratively develop the specification, and ensure that everyone is in the loop. This collaboration relationship is great to build up, as it will help us immensely in testing our services can integrate together.

So now we have our API, defined in swagger. Either written by us, our provider, or collaboratively. Whichever way you’ve got it, don’t trust it. Even though the Swagger editor provides validation whilst writing it, you can save the file with errors. Now I know you won’t do that, but someone else might. Don’t worry, we got it.

npm install --global swagger-cli

swagger-cli validate /path/to/swagger.json

Hopefully this will report no errors, but if it does, it will tell you what needs resolving. If you add this into your CI process, when someone checks in a change to the specifications, you can catch it on a pull-request, ensuring that anyone working from the specification in master, has a valid copy.

Consumer Driven Contract testing with Pact

Now we have an API specification, we can start developing our application however there are a couple of things we want to consider.

  • What happens if our provider isn’t ready for several months – how can we integration test it?

We can use the notion of CDC testing, also known as consumer driven contract testing, although I prefer the term collaborative driven contract testing. The former almost implies the consumer has free-reign to drive their own requirements, however it requires agreements from both parties, and participation, hence collaboration.

To perform our contract testing, we will use Pact.io (https://pact.io)

Pact (noun):

A formal agreement between individuals or parties.

Synonyms: agreement, protocol, deal, contract

Pacts are synonymous with API design, but how often do these get broken.

We might find that after developing our consumer or provider in isolation, when we come to integrate the systems in a test environment, that expectations are not being met, resulting in systems failing to communicate.

We can use Pact and its toolset, to enable us to generate contracts (pairs of request/responses) saved as a JSON file to use for component integration testing, in complete isolation from a service dependency whilst publishing these contracts to a central broker which can be queried at a later date by when the service is available. 

Utilising Pact in a consumer development flow
  1. Using the Pact DSL, the expected request and response are registered with the mock service.
  2. The consumer test code fires a real request to a mock provider (created by the Pact framework).
  3. The mock provider compares the actual request with the expected request, and emits the expected response if the comparison is correct.
  4. The consumer test code confirms that the response was correctly understood
  5. Tests pass!
  6. Results published to a broker as a JSON file
  7. Results are tagged with a branch tag for later querying

Always read the manual

In the above example, after the tests pass (step 5), we publish the results to the broker. This presents a problem, thankfully an avoidable one. We might find that the PACT contract tests have drifted from the Swagger specification. It may be that a required field is missing from the request, or the tests expected an array, but the swagger specification defined it as a string.

npm install -g swagger-mock-validator

swagger-mock-validator path/to/swagger.yml path/to/pact.json

This tool will tell us where that drift occurs, and you can fail your CI step, correct any errors and then re-run your CI build, ensuring that you publish post successful specification validation.

This gives us a high level of confidence, that as long as our provider sticks to the swagger specification, we should be in a good position come pact validation when our provider has their first build. We will talk about their part of the PACT testing pipeline shortly.

Fake it till you make it with Pact & Swagger Tools

So I know, trying to get your developers to write more unit tests, might seem like a hard sell, but what if we could re-use those interactions we had generated in a previous test to drive our higher level component integration & UI tests.

To help reduce the number of interactions that need verifying, you will want to use flexible matching on both requests and responses. (https://github.com/pact-foundation/pact-js#matching)

The mock service, is part of PACT’s ruby standalone package – https://github.com/pact-foundation/pact-mock_service

Usage:
  pact-stub-service PACT_URI ...

We create a docker image containing the standalone executable and copy in the consumer-provider JSON contract, for each service we need to mock. We can then run these locally, but CI will publish these to AWS in order to perform e2e tests, whilst we await our providers first build.

So we are now using our PACT contracts, that we generated, to perform our component integration and e2e testing, with mocks. Everything is green on your dashboard. It looks great right, time to relax.

Provider validation process with PACT

Utilising Pact in a provider development flow
  1. The provider retrieves its clients pacts from the broker
  2. Each request is sent to the provider, and the actual response it generates is compared with the minimal expected response described in the consumer test
  3. Provider verification passes if each request generates a response, that contains at least, the data described in the minimal expected result
  4. Tests passed and results published to the broker

Using the PACT broker to protect deployments

Each application version should be tagged in the broker with the name of the stage (eg. test, staging, production) as it is deployed.

This enables you to use the following very simple command to check if the application version you are about to deploy is compatible with every other application version already deployed in that environment

$ pact-broker can-i-deploy --pacticipant PACTICIPANT --version VERSION --to TAG --broker-base-url BROKER_BASE_URL

Take home points

  • Write your API specifications in Swagger
  • Store them in version control and give access to any providers/consumers for collaboration
  • Validate the swagger specifications are correct with swagger-cli
  • Write pact tests in a unit-testing framework of your choice, using one of the many different language implementations of Pact. (We use pact-js & Jest, written in TypeScript)
  • Run the tests during CI to generate the contract
  • Validate the generated pact contract against the swagger specification during CI
  • If it passes, publish the pact contract to the pact broker, tag it with the branch name.
  • If it is part of a development/staging/production additionally tag it with an identifier
  • Consumers can generate mock providers from the pact contract to use in integration / UI / e2e testing
  • Providers can read from the pact broker and test that they meet consumer expectations, as pact will mock the clients requests specified in the contracts.
  • All participants can use the can-i-deploy tool at CI time, to check if its compatible with other consumer/providers in a specific environment.

For a later blog post

  • Follow up from this blog post, with real code based examples on a Github repository you can clone, fork and play with for real.
  • How pact can help you avoid supporting multiple versions of API’s and deprecate features/endpoints gracefully .
  • Validating your developed service against your hand-crafted swagger specification that your tester won’t stop banging on about.

This week I have mostly been…

Starting this blog to chart some of my adventures in tech.

Here are some of the things I have covered over the last months, and will be taking the time to write up over the next few weeks :-

  • Building an i7-5920k 64gb Custom Mac Pro
  • Building a touch-screen macbook air using an Asus transformer TX300CA
  • Running MacOS in Windows with VMWare
  • Cross-platform development in Windows with MacOS & Linux VM’s, utilising RSync to allow the ability to provide code-completion in an IDE on the host-machine, but allow changes on the guest OS’s to propagate back to the host machine (and it’s editor)
  • Utilising PACT/Swagger to support Consumer Driving Contract testing flows
  • Evaluating Cypress.io as a UI testing tool against traditional Selenium based approaches.
  • Decompiling Digifant ECU’s namely Polo G40’s in order to better understand how the engine management works and allow me to tune my engine.
  • How stress control / management techniques can aid individuals and teams in software delivery