Test your service on Sonos

How to read log files to debug a failed test case

Running a test creates a folder in smapi\content_workflow\log_files. If you ran the test as a whole using suite_selftest.py, the name of the folder is the same as your config file. For example, the name is smapiConfig if you used the default config file. If you tested an individual test module the name is the same as that test module. For example, if you tested favorites.py, the name of the log files folder is Favorites Validation. These folders contain the following files:

  1. [Config file name].txt or [TEST_NAME]-Validation.txt – A summary of the test results for each test case.
  2. [Config file name]-DEBUG.log or [TEST_NAME]-validation-DEBUG.log – a detailed log file with step-by-step information of the test run, as well as the SMAPI SOAP request and server response between the test client and service endpoint.
  3. SummaryReport.html – Like the text file, but in an easy-to-read format when viewed in a browser.

These files replace existing log files each time you run a test. For example, running suite_selftest.py will overwrite the log files from the last time you ran it.

Using both DEBUG.log and SummaryReport.html, you can investigate and reproduce any test failure using a SOAP automation tool such as SoapUI.

Note that the results of the ssl_validation tests are only valid if DNS and TCP are working. You should test those before deciding that there are SSL problems.

General steps

  1. Find the test failure in the SummaryReport.html file. This file lists the failed tests above the ones that passed. Failed tests will have a status of Failed or Stopped. This file has the following information:
    • Test case ID – A number. For example, 4216.
    • Test suite name – The name of the test suite. For example, Browse.
    • Test case name – The name of the individual test case that failed. For example, test_combinatorial_pagination_browse_range_overlap.
    • Summary error message – A summary of the error. For example: “(mediaCollection){ id = "populargenre" itemType = "genre" title = "Popular Genres" canSkip = True } Fail: Response index wrong. (expected 0 != actual 100)”.
  2. Find the start and end lines of the failed test case in the DEBUG.log file.
    1. Search for the test case ID (a number) and test suite name. For example, an ID and test suite name may look like this “4216 Browse”.
    2. Search above the test case ID and suite name for the string “Start *test”. Note this line number as the START line.
    3. Find the string “End *test” after (in the down direction) the START line. Note this line number as the END line.
    4. Note that there could be thousands of lines of text for each test case because the detailed log has every SOAP request and response in it.
  3. We recommend copying the section for the failed test case between the START and END lines to a separate file for convenience. Review the log from the bottom up because a test failure usually stops the test case run.

Use the information in the DEBUG.log to replay the SOAP request to your service endpoint by following the steps below.

How to reproduce the test failure by using SOAP tools such as SoapUI

The following example uses SoapUI, but you can use any tool that can send a SOAP request and get a response.

  1. Create an initial project by importing sonos.wsdl from the smapi\content_workflow directory. This creates a list of Sonos SOAP actions such as search, getMediaMetadata, and getMetadata.
  2. Use the log file to identify the SOAP action type of the request in the Body (identified as <ns0:Body>). See the example log below.
  3. Copy the complete XML section starting with <?xml> and ending with </SOAP-ENV:Envelope>.
  4. Locate the SOAP action in SoapUI (or another tool) and paste the XML request from the previous step.
  5. Make sure to enter the same endpoint in SoapUI (or another SOAP tool) that you used in the test.
  6. If the tool you are using supports HTTP headers, set the following header key/value pair:
  7. User-Agent: Linux UPnP/1.0 Sonos/24.0-69180 (Self-Test).
  8. Send the SOAP request and wait for the response.

2016-12-16 18:04:29,040 [DEBUG] SONOS.suds.client - sending to (SERVICE_ENDPOINT) with timeout (30), message:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://www.sonos.com/Services/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.sonos.com/Services/1.1" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">

Known issues

  • The self-test does not test app authentication. If you are implementing this feature, please test it manually. See "how to set up getAppLink authentication" in run the automated self-test for details.
  • The self-test requires a newer version of OpenSSL than most Macs have installed. Please update OpenSSL before performing the self-test.