.. _testing:

Testing Guidelines
==================

Verify current Lithops status
-----------------------------

Run ``lithops test`` to run an extensive inspection, which simulates your lithops installation on a variety of tests.

* Get all available options by running ``lithops test --help``.

* Get all available test functions and their parent group by running ``lithops test -t help``.

* Run all test instances named ``<test name>``, via ``lithops test -t <test name>``.
  Run a test from a specific group by prefixing the test name with group name, e.g. : ``lithops test -t <test group>.<test name>``.
  Run multiple tests by separating them with a comma, e.g. ``lithops test -t <test name1>,<test name2>``.

* To get all available groups, run ``lithops test -g help``.

* Run a single group of tests via ``lithops test -g <group name>``.
  Run multiple tests by separating them with a comma, e.g. ``lithops test -g <test group1>,<test group2>``.

* To stop the test procedure upon first encountering a failed test, add the -f flag, e.g. ``lithops test -f``.
* To remove datasets, uploaded during the test procedure, use the -r flag, ``lithops test -r``.
  WARNING - do not use this flag on a github workflow, due to race condition issues.

* Get a complete list of the available flags by running ``lithops test --help``.

* A summarizing example: ``lithops test -t test_map,storage.test_cloudobject -g call_async -f``.
   
Alternatively, you may run the tests via ``python3 -m lithops.tests.tests_main``, followed by aforementioned flags.

Contribute
----------

Add a test to an existing test group
....................................

Locate the matching test group in ``lithops/tests`` (indicated by file name) and add your test function appropriately:
* Add your test function as a method inside the class inheriting from ``unittest.testcase``.

* Use existing util functions from ``lithops/tests/util_func`` or add new ones to that package.

* A test that's simultaneously testing any two of the following functions: {async, map, map_reduce} will be placed in the proper file by complying with the following hierarchy: map_reduce > map > async.

Add a new test group
....................

Before adding a test function that aims to test an untested feature:

* Create a new file in ``lithops/tests`` using the template file as your guide:

  * Create a new copy of the template file and name it "test_feature_name", thus automatically creating a new test group named "feature_name".

  * Figure out which rows are necessary for your new test group, by following documentation beside the rows, then, proceed to un-comment said rows.

* Continue to add the function by adhering to the instructions in the clause above.


Additional information
......................

Whether you're adding a new function, or a new test group you'd be better off knowing the available resources at your disposal:

* ``lithops/tests/util_func`` contains many functions, divided into categories, that you may find helpful.

* ``lithops/tests`` contains a template_file called "test_template" which contains documentation regarding each common resource/import that may interest you.

* Many examples utilizing said resources may be found across the "test_*" files of ``lithops/tests``.

* For a variety of evaluation functions ("assert*") belonging to unittest, browse this [documentation page](https://docs.python.org/3/library/unittest.html).

Examples
........

* Access the functions in your chosen backend storage class via "STORAGE", e.g. `STORAGE.put_object`.

* Access your bucket via `STORAGE_CONFIG['bucket']`.

* Pass on "CONFIG" to your function executor instance, to allow your test function to work with users that provided a path to the config file via a flag, e.g. `fexec = lithops.FunctionExecutor(config=CONFIG)`.