| |
| |
| \ Initial design decisions |
| ------------------------- |
| |
| Before I started working on Piglit, I asked around for OpenGL testing methods. |
| There were basically two kinds of tests: |
| |
| 1. Glean, which is fully automatic and intends to test the letter of the |
| OpenGL specification (at least partially). |
| |
| 2. Manual tests using Mesa samples or third-party software. |
| |
| The weakness of Glean is that it is not flexible, not pragmatic enough for |
| driver development. For example, it tests for precision requirements in |
| blending, where a driver just cannot reasonably improve on what the hardware |
| delivers. As a driver developer, one wants to consider a test successful |
| when it reaches the optimal results that the hardware can give, even when |
| these results may be non-compliant. |
| |
| Manual tests are not well repeatable. They require a considerable amount of |
| work on the part of the developer, so most of the time they aren't done at all. |
| On the other hand, those manual tests have sometimes been created to test for |
| a particular weakness in implementations, so they may be very suitable to |
| detect future, similar weaknesses. |
| |
| Due to these weaknesses, the test coverage of open source OpenGL drivers |
| is suboptimal at best. My goal for Piglit is to create a useful test system |
| that helps driver developers in improving driver quality. |
| |
| With that in mind, my sub-goals are: |
| |
| 1. Combine the strengths of the two kinds of tests (Glean, manual tests) |
| into a single framework. |
| |
| 2. Provide concise, human readable summaries of the test results, with the |
| option to increase the detail of the report when desired. |
| |
| 3. Allow easy visualization of regressions. |
| |
| 4. Allow easy detection of performance regressions. |
| |
| I briefly considered extending Glean, but then decided for creating an |
| entirely new project. The most important reasons are: |
| |
| 1. I do not want to pollute the very clean philosophy behind Glean. |
| |
| 2. The model behind Glean is that one process runs all the tests. |
| During driver development, one often gets bugs that cause tests to crash. |
| This means that one failed test can disrupt the entire test batch. |
| I want to use a more robust model, where each test runs in its own process. |
| This model does not easily fit onto Glean. |
| |
| 3. The amount of code duplication and forking overhead is minimal because |
| a) I can use Glean as a "subroutine", so no code is actually duplicated, |
| there's just a tiny amount of additional Python glue code. |
| b) It's unlikely that this project makes significant changes to Glean |
| that need to be merged upstream. |
| |
| 4. While it remains reasonable to use C++ for the actual OpenGL tests, |
| it is easier to use a higher level language like Python for the framework |
| (summary processing etc.) |
| |
| |
| |
| \ Coding style |
| ------------- |
| |
| Basic formatting: |
| |
| * Indent with 8-column tabs |
| * Limit lines to 78 characters or less |
| * Function return type and name go on successive lines |
| * Opening function brace goes on line by itself |
| * Opening statement braces go on same line as the 'for' or 'else' |
| * Use /* C style comments */, not // C++ style |
| * Don't write 'if (condition) statement;' on one line - put the statement on |
| a separate line. Braces around a single statement are optional. |
| |
| The following indent command will generally format your code for piglit's |
| style: |
| |
| indent -br -i8 -npcs -ce input.c -o output.c |
| |
| Though, that doesn't give perfect results. It messes up the |
| PIGLIT_GL_TEST_CONFIG_BEGIN/END section. And array initializers sometimes |
| come out funny. |
| |
| When in doubt see other recently added piglit tests for coding style. |
| |
| |
| Code conventions: |
| |
| * Use "const" qualifiers whenever possible on array declarations, pointers |
| and global variables. |
| * Use "static const" for initialized arrays whenever possible. |
| * Preprocessor macros should be UPPER_CASE |
| * Enumeration tokens should be UPPER_CASE |
| * Most other identifiers are lower_case_with_underscores |
| * Library, executable, and source file names are '<base>_<api>.' |
| e.g. libpiglitutil_gles2 |
| * Test names are '<lowercasegroupname>-<testname>.' e.g. glsl-novertexdata |
| * Use int, float, bool except when GL types (GLint, GLfloat) are really needed |
| * Always declare GL entrypoint pointer type with APIENTRY, or use piglit |
| dispatch typedef |
| |
| Test conventions: |
| |
| * The goal is to find bugs and demonstrate them as simply as possible, not |
| to measure performance or demonstrate features. |
| * Write tests that are easily read, understood and debugged. Long, complicated |
| functions are frowned upon. |
| * Don't try to test too much in a single test program. Most piglit programs |
| are less than 300 lines long. |
| * If a test doesn't render anything, it doesn't need to set the window size. |
| * Avoid changing an existing testing such that it might fail where it |
| previously passed. Break it into subtests and add a new subtest, or add |
| a new test which supersedes the old one. |
| * Things that should be seen are drawn in green (or blue as a second color) |
| while red is for things that shouldn't be seen. |
| * Calculate drawing coordinates from piglit_width/piglit_height as needed, |
| instead of hard coding. |
| * If a test can safely run at the same time as other tests, add it as a |
| concurrent test in 'all.tests' (or wherever you add it). |
| |
| |
| Utility code: |
| |
| Piglit has a rich set of utility functions for basic drawing, setting |
| up shaders, probing pixels, error checking, etc. Try to use them before |
| rolling your own. |
| |
| Python framework: |
| |
| Piglit uses python's PEP8 standard for formatting of python code; using only |
| spaces with no tabs for indenting. See |
| http://www.python.org/dev/peps/pep-0008/ for more information. |
| |
| |
| |
| \ Release Philosophy |
| ------------------- |
| |
| Since Piglit is a test suite, it is "production software" at all times. |
| Test case might be incorrect, but despite that it is not useful to speak of |
| "stable" and "unstable" versions of a test suite, especially one that sees |
| a relatively small rate of change like Piglit. |
| |
| For this reason, developers of OpenGL drivers and related software, and even |
| testers are encouraged to follow the git repository on freedesktop.org at all |
| times. A web interface to this repository can be found here: |
| |
| https://gitlab.freedesktop.org/mesa/piglit |
| |
| Nevertheless, for purposes of marking a specific point in time for packaging |
| in an environment where non-developers do tests on a wide range of hardware, |
| it has been pointed out that it would be useful to have official releases. |
| |
| For this reason, we will occasionally bump the version number in the file |
| RELEASE and create an appropriate tag in the git repository. |
| |
| This tag is the official way of marking a release, so the tarballs provided |
| automatically by the cgit frontend are official release tarballs. |
| |
| |
| \ Contributing Patches |
| --------------------- |
| |
| If you want to contribute patches, please subscribe to the piglit |
| mailing list (http://lists.freedesktop.org/mailman/listinfo/piglit) |
| and then send them to piglit@lists.freedesktop.org using "git |
| send-email". One of the core piglit developers should respond with |
| comments and suggested improvements. The piglit mailing list is also |
| a good place for general discussion about piglit development, such as |
| future plans for the project, and coordinating work between |
| developers. |
| |
| Note that Piglit patches use the terms "Reviewed-by", "Tested-by", and |
| "Acked-by" in the same way as they are used for Linux kernel patches |
| (see https://www.kernel.org/doc/Documentation/SubmittingPatches). You |
| are also welcome to add a "Signed-off-by" line to your patch, but it |
| is not required. |
| |
| For developers who are new to piglit: when submitting a patch, it is |
| helpful to add a note (after the "---" line in the patch file) |
| indicating that you are new to the project and don't have commit |
| access; that way once your patch has been revised to meet our |
| standards of correctness and coding style, we will know that we should |
| commit it for you. If we forget, please remind us! Once you have |
| successfully contributed a handful of patches, feel free to apply for |
| commit access usind the process described here: |
| http://www.freedesktop.org/wiki/AccountRequests/ |
| |
| Please be patient--most of us develop graphics drivers (such as Mesa) |
| as our primary job, so we have limited time to respond to your patches |
| on the piglit mailing list. If your patch hasn't received a reply in |
| a week, send a follow-up email to make sure we haven't missed it. If |
| you have questions that are better discussed in real time, many piglit |
| developers can also be found in the #dri-devel channel on Freenode |
| IRC. |