Testfully CLI

Testfully CLI is a multi-platform CLI tool that allows you to work with requests from the command line, and integrate with your CI/CD pipeline.

Pre-requisites

  • Node.js version 16 or higher
  • A Node Package Manager such as npm or yarn.

Installation

Install Testfully CLI globally using npm or yarn.

# Install using npm
npm install -g @testfully/cli

# or install using yarn
yarn global add @testfully/cli

How It Works

To use Testfully CLI, you need to export your requests via the Testfully App. Once exported, you can source the exported files to the Testfully CLI, and run your requests from the command line. To learn more about exporting requests, please refer to the Export Workspace Data page.

run command

The testfully run command executes one or more requests. By default, it runs all requests in the supplied file(s). You can specify one or more requests or folder IDs to run specific requests. Use the following parameters to customize the command.

--request-id

By default, the testfully run command runs all requests in the supplied file(s). You can specify one or more request IDs to run specific requests.

# Run requests with ID 1 and 2 only
testfully run --request-id 1 --request-id 2  rick-and-morty.json

--folder-id

By default, the testfully run command runs all requests in the supplied file(s). You can specify one or more folder IDs to run specific requests.

# Run requests in folder with ID 1 and 2 only
testfully run --folder-id 1 --folder-id 2 rick-and-morty.json

--logs

Display logs in the console.

# Display logs in the console
testfully run --logs  rick-and-morty.json

--verbose

Display raw requests and responses in the console.

testfully run --verbose  rick-and-morty.json

--environment-id

Specify the environment ID to use for the requests.

testfully run --environment-id 1  rick-and-morty.json

--environment or -e

Use the --environment or -e option to provide the environment file to use for the requests. If the provided environment file does not exist, Testfully CLI will throw an error. When you provide an environment file that contains more than one environment, Testfully CLI will use the first environment and ignore the rest. When both --environment-id and --environment are provided, --environment takes precedence.

testfully run --environment pre_prod_env.json -- payment_tests.json

Did you notice the -- before the requests file? It is a separator that tells the CLI that the options have ended and the positional arguments have started. You can use it to separate options from positional arguments.

--globals or -g

Use the --globals or -g option to provide the globals file to use for the requests. If the provided globals file does not exist, Testfully CLI will throw an error.

testfully run --globals workspace_globals.json -- payment_tests.json

To provide multiple globals files, use the --globals option multiple times or pass multiple files as value for a --globals or -g option.

testfully run --globals shared_globals.json ci_globals.json -- payment_tests.json

Did you notice the -- before the requests file? It is a separator that tells the CLI that the options have ended and the positional arguments have started. You can use it to separate options from positional arguments.

--delay-request

You can use the delay-request option to specify the delay between requests in milliseconds. Alternatively, you can use the Delay setting for each folder for a more granular control. In the absence of both, requests are executed without any delay. In the presence of both, the delay-request option takes precedence. Set the delay to 0 to disable it.

testfully run --delay-request 1000 rick-and-morty.json

--reporters

The reporters option allows you to specify the type of test report format to generate. Currently, Testfully CLI supports junit and cli reporters.

testfully run rick-and-morty.json --reporters junit

--bail

The --bail argument stops the execution of requests when the first request fails. By default, Testfully CLI continues to execute requests even if a request fails.

--silent

The --silent argument suppresses all logs, it only displays any error messages.

--global-var name=value

You can use the --global-var option to set value of a global variable. To set multiple global variables, use the --global-var option multiple times. To pass a value with spaces, enclose the value in double quotes.

testfully run --global-var api_key=12345 --global-var base_url=https://api.example.com rick-and-morty.json

--memory-size

By default, Testfully CLI allocates 1MB of memory for each script execution. You can use the --memory-size option to specify the memory size in bytes. There is no limit on the memory size, but it is recommended to allocate a reasonable amount of memory to avoid performance issues.

--insecure

The --insecure option allows you to disable SSL certificate verification. This is useful when you are using a self-signed certificate or a certificate from a private CA. Use this option with caution, as it can expose you to security risks.

--ssl-extra-ca-certs

The --ssl-extra-ca-certs option allows you to specify the path to a file containing additional CA certificates. This is useful when you are using a self-signed certificate or a certificate from a private CA. You can specify multiple files by providing multiple --ssl-extra-ca-certs options. The file should be in PEM or CRT format. For PEM format, you can combine multiple certificates into a single file.

--ssl-client-cert

The --ssl-client-cert option allows you to specify the path to a file containing the client certificate. This is useful when you are using mutual TLS authentication. The file should be in PEM or CRT format. If you need to provide client certificates for multiple hosts, you can use the --ssl-client-cert option multiple times.

ssl-client-key

The --ssl-client-key option allows you to specify the path to a file containing the client key. This is useful when you are using mutual TLS authentication. The file should be in PEM, KEY, PFX or P12 format. If you need to provide client keys for multiple hosts, you can use the --ssl-client-key option multiple times.

--ssl-client-key-passphrase

The --ssl-client-key-passphrase option allows you to specify the passphrase for the client key. This is useful when you are using a password-protected client key. If you need to provide passphrases for multiple keys, you can use the --ssl-client-key-passphrase option multiple times.

--ssl-client-host

The --ssl-client-host option allows you to specify the host for the client certificate. This is useful when you are using mutual TLS authentication and need to provide different client certificates for different hosts. If you need to provide client certificates for multiple hosts, you can use the --ssl-client-host option multiple times.

--ssl-client-cert-list

The --ssl-client-cert-list option allows you to specify a list of client certificates in JSON format. This is useful when you are using mutual TLS authentication and need to provide different client certificates for different hosts. The JSON format should follow the below structure:

[
  {
    // a user-friendly name for the client certificate
    "name": "dummy-api",

    // a list of hostnames or IP addresses that the client certificate is valid for
    "matches": ["localhost:9443"],

    // the path to the client key file
    "key": { "src": "./tests/mtls/client.key" },

    // the path to the client certificate file
    "cert": { "src": "./tests/mtls/client.crt" },

    // passphrase for the client key, if required
    "passphrase": ""
  }
]

You can specify multiple client certificates by adding multile objects to the array.

positionals

You can specify one or more files to run requests from by passing them as the last arguments.

testfully run --folder-id 1 rick-and-mort-api.json pre-prod.json

Reporters

Reporters are a way to generate test reports in different formats. Testfully CLI currently supports the following reporters:

  • junit: Generates a JUnit XML report. This is useful for integrating with CI/CD tools that support JUnit reports.
  • cli: Generates a human-readable report in the console.

You can specify the reporters using the --reporters option. You can specify multiple reporters by providing multiple --reporters options. The order of the reporters does not matter, as Testfully CLI will generate the reports in the specified order.

testfully run --reporters junit --repo rick-and-morty.json

junit reporter

The junit reporter generates a JUnit XML report. This is useful for integrating with CI/CD tools that support JUnit reports. The report is generated in the current working directory with the name junit.xml. You can use the following arguments to customize the report:

--reporter-junit-export

The --reporter-junit-export option allows you to specify the directory/file path to export the JUnit report. If the file path does not exist, Testfully CLI tries to create it. If the file path exists, Testfully CLI overwrites the file. If the file path is a directory, Testfully CLI creates a file with the name junit.xml in that directory.

testfully run rick-and-morty.json --reporters junit --reporter-junit-export ~/Downlods/junit-report.xml

cli reporter

The cli reporter generates a human-readable report in the console. This is useful for debugging and understanding the test results. By default, this reporter is enabled. If you want to disable it, you can use the --reporters option to specify the reporters you want to enable, or --silent option to suppress all logs.

The cli reporter supports the following options:

--reporter-cli-silent

The --reporter-cli-silent is an alias for the --silent option. It suppresses all logs.

--reporter-cli-show-timestamp

The --reporter-cli-show-timestamps option allows you to show timestamps of each request in the report. By default, this option is disabled.

--reporter-cli-no-summary

Turns off the statistics summary at the end of the report. By default, this option is enabled.

--reporter-cli-no-failures

Turns off the failures summary at the end of the report. By default, this option is enabled.

--reporter-cli-no-assertions

Turns off all of the assertions in the report. By default, this option is disabled.

--reporter-cli-no-success-assertions

Turns off the success assertions in the report. By default, this option is disabled.

Common Errors

This section lists common errors that you may encounter while using Testfully CLI.

Error: No input file specified

You receive this error when you do not provide any input file to the testfully run command. Sometimes, you may have provided an input file but you are missing the -- separator. Make sure to provide the input file(s) after the -- separator. The -- separator tells the CLI that the options have ended and the positional arguments have started. You would need this separator when you have --globals, -g, --environment or -e in your command. Here is an example that throws this error:

testfully run --globals workspace_globals.json  payment_tests.json

To fix this error, add the -- separator before the input file(s). Here is the corrected command:

testfully run --globals workspace_globals.json  -- payment_tests.json

The -- separator tells the CLI that the options have ended and the positional arguments have started.

FAQ

How does Testfully CLI execute requests?

Testfully CLI uses the host computer to execute requests. It does not use any cloud resources. It uses the same underlying engine as the Testfully App to execute requests so that you can get the same results.

What features are supported in Testfully CLI?

When it comes to executing requests, Testfully CLI supports all features that are supported in the Testfully App except for the following:

  • OAuth2 Authorization using the Authorization Code or PKCE grant types.

Does Testfully CLI store any data on my computer?

Testfully CLI does not store any data. Any computed data is stored in memory and is discarded once the request is completed.

Does Testfully CLI require an internet connection?

Testfully CLI does not require an internet connection to execute requests. However, it may require an internet connection to send requests to API endpoints if they are not available locally.

Can I use Testfully CLI in my CI/CD pipeline?

Yes, Testfully CLI is shipped as a standard Node.js package. You can use it in your CI/CD pipeline to run your requests. We have tested it against Windows 10, Windows 11, Ubutnu 20.04, and macOS 13 and 14.

How can I update Testfully CLI?

You can update Testfully CLI using npm or yarn.

# Update using npm
npm update -g @testfully/cli

# or update using yarn
yarn global upgrade @testfully/cli

How can I uninstall Testfully CLI?

You can uninstall Testfully CLI using npm or yarn.

# Uninstall using npm
npm uninstall -g @testfully/cli

# or uninstall using yarn
yarn global remove @testfully/cli
Next
Getting started