GET Experience from Yandex. How to make your own report for autotests / Yandex Blog / Sudo Null IT News FREE

I privation to share my experience on how to make up good reports on car tests and simultaneously invite you to the first of all Yandex event specifically most testing.

First, a couple of words about the event. Along November 30 in St. Petersburg we will hold a Examination Surroundings - our first upshot specifically for testers. There we will tell you how our testing works, what we did to automate it, how we work with errors, data and graphs, and much more than. Involution is free, but there are exclusively 100 places, so you need to register soon enough .

The test environment for us is primarily a platform for communication. We neediness to not only talk about ourselves, but also to blab ou with the participants all but how they work, share knowledge, answer some questions. We think there bequeath beryllium many grassroots topics, just for you to start intellection about them right away, we are starting a series of publications nigh testing in Yandex.

Several reports leave be devoted to test automation on the Psychometric test Surroundings, including mine. So, I will begin. There are unit tests, but there are high-level ones. And when their total begins to arise, analyzing the results of launches becomes a problem. Tell Pine Tree State honestly, which of you did not think to make your report?

With detailed logs, screenshots, quest / response dumps and former extra information (which, by the bye, greatly facilitates the sensing of specific causes of the error). I am sure that some have even succeeded in this matter to. The problem is that it is unruly to make one universal describe for whol types of tests, and it takes a long time to induce a separate report for a specific task. Unless, of course, you accidentally use jUnit and Mavin . In that case, you can make a simple report for a ad hoc type of test in a hardly a hours. Let's see why we need a test report differently xUnit ?

High-level tests take issue from unit tests and have a telephone number of features:

1. They touch much more functionality, which makes it difficult to localize the problem. So, for example, a test through and through a web user interface affects the functionality of the API, which successively affects the functionality of the database, which in turn ... well, you understand.
2. So much tests act on the organization done intermediaries. Information technology hindquarters atomic number 4 a web browser, http-server, proxy, third-party systems, which also have their own logic.
3. Such tests are unremarkably quite a lot and often have to introduce additive categorization. These May be components, areas of functionality, criticality.

All these factors significantly slow down the plac of problem localization principle. E.g., here is what the error in the test for the web interface "Can not click on element" Search Button "" May mean:

• the page did non load away timeout;
• there is no Search Button element on the page;
• the Search Button element is introduce, but it is unacceptable to pawl on it;
• a meteorite fell along the data center in which the service is spinning.

If you attention deficit disorder a screenshot, Sri Frederick Handley Page source, mesh logarithm and a summary of space activity in the area of ​​the data kernel to the results of this run, then it leave be much easier to indicate a specific problem, which means we will spend less time. Therein case, there is a need for a specific report with additional information.

In that respect once was a test

As a test matter for our experiments, we take a completely ordinary test:

          national class ScreenShotDifferTest {     private final long Divagation = 20L;     private WebDriver driver = new FirefoxDriver();     public ScreenShooter screenShooter = new ScreenShooter();     @Test     public void originPageShouldBeSameAsModifiedPage() throws Exception {         BufferedImage originScreenShot = screenShooter.takeScreenShot("http://www.yandex.ru", device driver);         BufferedImage modifiedScreenShot = screenShooter.takeScreenShot("http://beta.yandex.ru", driver);         unsound diffPixels = screenShooter.diff(originScreenShot, modifiedScreenShot);         assertThat(diffPixels, lessThan(Divergence);     }     @After     public void closeDriver() {         driver.quit();     } }        

Let's go through the encipher:

• format device driver;
• initialize screenShooter;
• take a screenshot of the first Thomas Nelson Page;
• take a screenshot of the candidate page;
• count the numerate of different pixels;
• check that the issue of variant pixels does not overstep the tolerance;
• close the driver.

In this form, the test can be used without a beautiful report, since it always compares the same page with itself. But this tryout leave be more more effective if you impart the textbook jUnit parameterization to it :

                      @RunWith(Parameterized.class) public class ScreenShotDifferTest {     ...     private String originPageUrl;     private String modifiedPageUrl;     public ScreenShotDifferTest (String originPageUrl, String modifiedPageUrl) {         this.modifiedPageUrl = modifiedPageUrl;         this.originPageUrl = originPageUrl;     }     @Parameterized.Parameters(name = "{0}")     public static Collection              readUrlPairs () {         getting even Arrays.asList(                 new Object[]{"Yandex Main Page", "http://www.yandex.ruthenium/", "http://beta.yandex.ru/"},                 new Object[]{"Yandex.Market Main Pageboy", "http://market.yandex.atomic number 44/", "http://beta.food market.yandex.ru/"}         );     }     ... }                  

It is ameliorate to pull data from the storage to which the someone using the quiz has access. But for lucidity, the above method acting fits dead.

So, let's think that we deliver non 2 parameters, but 20, or better 200. A standard try out report will attend like this :

What conclusion can personify careworn from the examination report?

Let's call up together what information we need in order to quickly decide connected the presence of errors:

1. Screenshots of the original page and the prospect.
2. Dif screenshots (you pot, for example, mark all the different pixels in Red)
3. Sources of the original Sri Frederick Handley Page and the candidate.

With such information, it testament be overmuch easier to make conclusions about problems, which means cheaper.

Report Implementation

Systematic to build an extended test report, we involve to put through three stages:

1. Model . It will contain totally the information necessary for displaying in the report.
2. Adaptor . He must collect all the indispensable information from the test to the poser.
3. Report Generation . Founded on the collected data, we generate a report based on templates.

So, in prescribe.

Model

To solve this job, we leave use xsd schemes for the subsequent generation of java classes using Java JAXB . As luck would have it, our manakin contains trivial data and is easily delineated by a schema.

The scheme is ready! Now it remains to generate classes according to this scheme. To do this, apply the powerful ace-jaxb2-plugin . The advantage of this plugin is that classes are generated at each compilation. Thus, you tail end live 100% sure that the generated code matches the dodging and relieve yourself errors like "Oh, I forgot to regenerate ..." The result of the plugin will be the generated classes (cautiously, they are huge):

TestCaseReport

              /**  *                
Java grade for TestCaseResult complex type.  *   *                
                
The following schema sherd specifies the expected content contained inside this class.  *   *                
                
                  *                                      *                                          *                                              *                                                  *                                                      *                                                          *                                                              *                                                                  *                                                                      *                                                                                                                                                                                                      *                                                  *                                                      *                                                          *                                                                                                                          *                                        *                                    *                
                *   *   */ @XmlAccessorType(XmlAccessType.Field of operation) @XmlType(identify = "TestCaseResult", propOrder = {     "subject matter",     "verbal description",     "origin",     "modified",     "diff" }) public separate TestCaseResult {     @XmlElement(required = true)     moated String message;     @XmlElement(required = true)     protected Draw description;     @XmlElement(required = true)     protected ScreenShotData origin;     @XmlElement(required = true)     protected ScreenShotData modified;     @XmlElement(required = true)     protected DiffData diff;     @XmlAttribute(name = "uid")     protected Drawing string uid;     @XmlAttribute(name = "title")     protected String title;     @XmlAttribute(name = "position")     weatherproof Status status;     /**      * Gets the value of the message property.      *       * @return key      *     possible object is      *     {@link String }      *           */     public String getMessage() {         return message;     }     /**      * Sets the rate of the message attribute.      *       * @param esteem      *     allowed object is      *     {@link Strand }      *           */     public void setMessage(Chain value) {         this.message = value;     }     /**      * Gets the apprais of the verbal description property.      *       * @return      *     possible object is      *     {@link String }      *           */     overt String getDescription() {         reappearance verbal description;     }     /**      * Sets the value of the description holding.      *       * @param value      *     allowed object is      *     {@link Drawing string }      *           */     public void setDescription(String along value) {         this.verbal description = value;     }     /**      * Gets the value of the origin dimension.      *       * @counte      *     possible physical object is      *     {@relate ScreenShotData }      *           */     public ScreenShotData getOrigin() {         return origin;     }     /**      * Sets the value of the origin property.      *       * @param value      *     allowed object is      *     {@inter-group communication ScreenShotData }      *           */     public void setOrigin(ScreenShotData esteem) {         this.root = measure;     }     /**      * Gets the economic value of the modified holding.      *       * @return      *     possible object is      *     {@inter-group communication ScreenShotData }      *           */     public ScreenShotData getModified() {         return modified;     }     /**      * Sets the value of the modified material possession.      *       * @param value      *     allowed object is      *     {@connect ScreenShotData }      *           */     public void setModified(ScreenShotData note value) {         this.modified = assess;     }     /**      * Gets the value of the diff property.      *       * @return      *     possible object is      *     {@link DiffData }      *           */     overt DiffData getDiff() {         recall diff;     }     /**      * Sets the value of the diff dimension.      *       * @param value      *     allowed object is      *     {@link DiffData }      *           */     public void setDiff(DiffData esteem) {         this.diff = value;     }     /**      * Gets the value of the uid place.      *       * @revert      *     possible targe is      *     {@link String }      *           */     semipublic String getUid() {         hark back uid;     }     /**      * Sets the valuate of the uid property.      *       * @param value      *     allowed object is      *     {@connec String }      *           */     public void setUid(String value) {         this.uid = prize;     }     /**      * Gets the value of the title property.      *       * @return      *     possible object is      *     {@link String }      *           */     public Chain getTitle() {         return title;     }     /**      * Sets the value of the championship attribute.      *       * @param measure      *     allowed object is      *     {@link String }      *           */     public void setTitle(String value) {         this.title = valuate;     }     /**      * Gets the value of the status holding.      *       * @come back      *     possible object is      *     {@contact Status }      *           */     semipublic Condition getStatus() {         return status;     }     /**      * Sets the measure of the status property.      *       * @param value      *     allowed aim is      *     {@link Status }      *           */     public void setStatus(Status value) {         this.position = value;     } }            

Classes are ready besides. Right away you stern easily and simply serialize objects into xml files:

          TestCaseResult testCaseResult = ... JAXB.marshal(testCaseResult, filing cabinet);        

And read objects from the xml file

          TestCaseResult testCaseResult = JAXB.unmarshal(Indian file, TestCaseResult.class)        

Arranger

Lashkar-e-Tayyiba me remind you that we need an arranger systematic to fill the model with data from the test during its execution. To implement the adapter, we bequeath economic consumption the jUnit Rules mechanism , or to be more precise, TestWatcher Rule:

          public abstract class TestWatcher implements org.junit.rules.TestRule {     //обязательно вызывается перед началом теста     protected void starting(org.junit.runner.Verbal description description) {...}     //этот метод вызывается в случае успешного завершения теста     protected void succeeded(org.junit.base runner.Description description) {...}     //этот метод вызывается, если //вы используете// !!(сработает)!! assumeThat()     protected void skipped(org.junit.internal.AssumptionViolatedException e, org.junit.runner.Description description) {...}     //этот метод будет вызван в случае возникновения ошибки в тесте     protected empty failed(java.lang.Throwable e, org.junit.runner.Verbal description description) {...}     //обязательно вызывается после завершения теста     protected void finished(org.junit.runner.Description description) {...} }        

Let's take a look at each method in succession and toy with where you can collect the necessary data.

•               moated void starting(org.junit.runner.Description description)            

  - tote up to it the initialization of the TestCaseResult model and the creation of all the necessary files.

•               burglarproof void succeeded(org.junit.runner.Description verbal description)            

  - in it we cast down the condition OK of the execution of our test.

•               protected void skipped(org.junit.internal.AssumptionViolatedException e, org.junit.runner.Description description)            

  - we are not interested in this method acting. It can be left unchanged.

•               protected void unsuccessful(java.lang.Throwable e, org.junit.blue runner.Description description)            

  - here we testament birth conditional logic. If

                e instanceOf AssertionViolatedException            

  , then an error occurred in the test (FAIL), in any other case the test is broken (ERROR).

•               protected invalidate finished(org.junit.moon-curser.Description description)                          

  - here we serialize the TestCaseResult targe in xml.

In addition to all of the above, our steering wheel should be competent to take and save screenshots, which is delineate in the methods:

•               public BufferedImage takeOriginScreenShot(String url)            

  - fill a screenshot of the original page by url, save a screenshot to the data file system of rules, link to the data and return BufferedImage.

•               public BufferedImage takeModifiedScreenShot(Drawing string url)            

  - the same operations, only for the candidate page.

•               public DiffData diff(BufferedImage original, BufferedImage restricted)            

  - we come the diff of two screenshots, save to the file organisation, link to the data and issue an object with information about the differences.

We will put across completely the files in a directory mark/site/custom, since it is default for reports.

Afterward using the 'ScreenShotDifferRule', our test will scarce change:

          @RunWith(Parameterized.class) public class ScreenShotDifferTest {     snobby String originPageUrl;     private Cosmic string modifiedPageUrl;     ...     @Rule     public ScreenShotDifferRule screenShotDiffer = new ScreenShotDifferRule(driver);     public ScreenShotDifferTest(String rubric, String originPageUrl, String modifiedPageUrl) {         this.modifiedPageUrl = modifiedPageUrl;         this.originPageUrl = originPageUrl;     }     ...     @Test     public void originShouldBeSameAsModified() throws Exception {         BufferedImage originScreenShot = screenShotDiffer.takeOriginScreenShot(originPageUrl);         BufferedImage modifiedScreenShot = screenShotDiffer.takeModifiedScreenShot(modifiedPageUrl);         extendible diffPixels = screenShotDiffer.diff(originScreenShot, modifiedScreenShot);         assertThat(diffPixels, lessThan((long) 20));     }     ... }        

Now, with the assistant of a simple ScreenShotDifferRule after each trial, we will receive structured data in the following cast:

1. {uid} -testcase.xml

                                                    http://www.yandex.ru/                {uid}-blood line.png                                            http://www.yandex.ru/                {uid}-modified.png                                            0                {uid}-diff.png                                            

2. {uid} -origin.png

3. {uid} -diff.png

Report Generation

We need to implement the Maven Study Plugin, which volition take in all the {{uid}} - testcase.xml-ki in one and supported information technology will give an hypertext markup language-page. To do this, ADHD the TestSuiteResult collector object of totally TestCaseResult s to our model . I will non dig wide into the field of creating plugins for Maven - this is a topic for a severalize article. Instead, I propose to consider a ready-made plugin that solves our job.

Sol, we have ScreenShotDifferReport Plugin . The heart of the plugin is the method unexclusive null White House (). In our case, he should:

1. Find all test data files.
   

                 File in[] testCasesFiles = listOfFiles(reportDirectory, ".*-testcase\\.xml");            

2. Read them and change over to objects.
   

                 List                  testCases = convert(testCasesFile, new Converter(){    public TestCaseResult convert (File file away) {       reappearance JAXB.unmarshall(file, TestCaseResult.xml);    }   });                                          

3. Generate index.html based on the data. You can employment freemarker and this template as a template locomotive engine .
   

                 Draw source = processTemplate(TEMPLATE_NAME, testCases);            

4. Attention deficit disorder information about this report to a grouping maven report.
   

                 Sink sink in = new-sprung Sink(); sink.setHtml(source); sink.close();((            

To get a fix report we pauperism to do a command mvn clean install. For simplicity, you tin can deflate the github.com/yandex-qatools/tests-report-example propose and run the bidding for it. As a lead of functional the command in the tests-report-example module in the target / site / directory, you will see a send off report .

Check result

At present you postulate to complete the installing of the entire stick out. To bash this, run the command at the antecedent of the project mvn clean install. Later its implementation, we will pick up artifacts ready for exercise. We connect our fresh made plug-in to the project of autotests conjointly with the standard surefire-plugin.

                                    org.apache.maven.plugins              maven-site-plugin              3.2                                                                    atomic number 44.yandex.qatools.examples                    usage-report-plugin                    ${project.version}                                                        org.Apache.maven.plugins                    maven-surefire-report-plugin                    2.14.1                                                                              

And action the command mvn clean site.

Voila! After passing the tests, the site phase will be executed, within which deuce reports will be generated: SureFire Report and Custom Composition .

"Wherefore physical body two reports?" You ask. The fact is that the jUnit Rules mechanics is non perfect. If an exception throws an exception in the test constructor or in the parameterization method, then the rudder will not be created, which agency that the data for building the report will not be collected. Which in turn means that the test leave non get into the report. You privy improve the data collection process with RunListener Oregon Runner , but this seems like unneeded logic. All information regarding broken tests is in the SureFire news report.

Total

So, we learned how to build unpretentious reports using the jUnit and Wi framework extensions.

pros

1. We get gratis entirely the features of the jUnit framework for running and organizing tests (parallel launch, parameterization, categories).
2. Clearly part information and presentation. You can make the adapter in another speech (for good example, in python), but use the same plugin to generate the view. Surgery use different plugins for the Saami information.
3. We get free reporting logic for delivering reports to the repository (ssh, https, ftp, webdav, etc.) using the Maven Wagon Plugin .
4. We can generate a "partial derivative report". This is achieved by separating the flow of test slaying and reporting. One thread performs tests (which generate information), and the second periodically builds a report.

Minuses

1. Требуется хорошее знание технологий (XSD, JAXB, jUnit Rules, Wi Reporting Plugin). Если что-то пойдет не так, рискуете потерять много времени.
2. Довольно сложно тестировать весь цикл построения сложного отчета (от схемы до html)

Рекомендации

1. Разработка таких систем требует много времени. У нас на разработку первого ушло около 50 литров кофе, двух мешков печенек и 793 нажатий на кнопку Build с учетом анализа технологий и сбора граблей. Сейчас создание отчета под конкретную задачу занимает порядка двух дней. Оцените время, которое вы выиграете, используя этот отчет. Оно должно быть больше.
2. Наибольший эффект достигается, когда вся команда принимает участие в отсмотре подобных отчетов.

In the article I talked about victimization the following technologies:
1. jUnit , jUnit Rules for implementing the Adaptor.
2. JAXB for serializing / deserializing a model in xml.
3. Maven Reporting Plugins to generate a report on the finished data.

The source code for the example is available connected github . Joba, who is construction the report, is available at .

DOWNLOAD HERE

GET Experience from Yandex. How to make your own report for autotests / Yandex Blog / Sudo Null IT News FREE

Posted by: turnerthoonions.blogspot.com

Share this post