How I “hacked” into RATP’s API

How I “hacked” into RATP’s API

Jan, 3rd 2017: La RATP (the company in charge of Paris’s underground train network) has finally opened its API for real-time traffic.
Jan, 5th 2017: I email my subscription form in order to be able to use it.
Jan, 17th 2017, 2pm: RATP’s Open Data team eventually answers me and my IP is allowed access to the API.
Jan, 19th 2017, 2am: Finally! After hours of going crazy over the technology used by WSIV (Web Service Information Voyageur), after a few giving up/getting back to it moments, it works. I am now able to display every possible information about the next trains going from the École polytechnique train station (Lozère) to Paris.

Here is the code I ended up with:

<?php
$request = '<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://wsiv.ratp.fr/xsd" xmlns:wsiv="http://wsiv.ratp.fr">
    <soapenv:Header/>
 <soapenv:Body>
     <wsiv:getMissionsNext>
        <wsiv:station>
            <xsd:line>
                <xsd:id>RB</xsd:id>
            </xsd:line>
            <xsd:name>Lozere</xsd:name>
        </wsiv:station>
        <wsiv:direction>
            <xsd:sens>A</xsd:sens>
        </wsiv:direction>
     </wsiv:getMissionsNext>
 </soapenv:Body>
</soapenv:Envelope>';
$location = "http://opendata-tr.ratp.fr/wsiv/services/Wsiv?wsdl=";
$uri      = "http://opendata-tr.ratp.fr/wsiv/services";
$action   = "urn:getMissionsNext";
$version  = 0;

$client = new SoapClient(null, array('location' => $location,
                                    'uri'       => ""));
$xmlstring = $client->__doRequest($request, $location, $action, $version);
$clean_xml = str_ireplace(['SOAPENV:', 'NS1:', 'NS2:'], '', $xmlstring);
$xml       = simplexml_load_string($clean_xml);

$return        = $xml->Body->getMissionsNextResponse->return;
$perturbations = array();
$missions      = array();

$station     = $return->argumentStation->name;
$direction   = $return->argumentDirection->name;
$line        = $return->argumentLine->reseau->name . ' ' . $return->argumentLine->code;

echo "Station $station ($line) en direction de $direction",PHP_EOL;

foreach($return->perturbations as $perturbation) {
    echo $perturbation->message->text,PHP_EOL;
}

foreach($return->missions as $mission) {
    $id = isset($mission->id) ? $mission->id . ' ' : "";
    echo $id, $mission->stations[1]->name, ': ', $mission->stationsMessages;
}

I needed to use PHP for this implementation because one goal was to include this snippet into Frankiz, the school’s students (old) website.

Two days ago, I had no idea what a Web Service was, nor what WSDL stood for, and I thought SOAP was… well you know… soap.

Looking into the API’s documentation

In theory, everything I needed can be found on RATP’s website, where I was able to download the developer’s kit, a .zip file, with the following content :

├── CG-API-RATP.pdf
 ├── FO-inscription.pdf
 ├── ratp-wsiv-opendata
 │   ├── Wsiv.wsdl
 │   ├── doc_wsiv.html
 │   ├── exemple.pdf
 │   ├── index.html
 │   └── ratp.gif

Fast-forward the administrative procedure for allowing my VPS’s IP to request their API, I was thrilled to open the exemple.pdf file.

The examples demonstrated that this API exposed what we would expect from it: lines, stations, timestamps of the next passages. However, the examples were simply unexploitable: instead of raw text, the request was given as an image where we can not even see the full text.

So, back to to the doc_wsiv.html file, the API’s official documentation. Sadly, this is not the documentation you would be used to (especially if you have ever tasted/tested Trello’s API).

I applied my usual strategy an launched up Postman.

Understanding WSDL and SOAP

Postman is the must-have tool when it comes to web requests. I use it to replay API calls, amongst other things.

Eventually, I found this page from Postman’s documentation explaining how I could use Postman to make SOAP requests, too.

The idea is: you simply make a POST request using raw text/xml content, which you define to be the content of your SOAP request.

It worked quite well with the Holiday Service example, so I was quite optimistic.

Using Postman to hit the RATP API

At first, it was quite unconvenient to carry out tests on the RATP API because only the IP of my VPS was allowed to initiate the connection. Hopefully, using my VPN, I was able to send requests to the RATP servers directly from my computer but with the correct IP.

At first, I naively thought that I would hit something by recycling my request from the Holiday Service example, simple changing the xmlns:hs attribute and adapting the URL.

But what URL? Reading several times through the API’s documentation, I did not find any mention of it. It was time to open the Wsiv.wsdl file. Here, I found a few candidates : http://wsiv.ratp.fr/ and http://www.ratp.fr/wsiv/services/Wsiv?wsdl did not work. But I eventually found http://opendata-tr.ratp.fr/wsiv/services/Wsiv, which DID return an error. Hourray.

 

Once I understood what URL I was supposed to request at, I was far from being done. I chose the POST method, the Headers needed to be custom defined: Content-Type=text/xml. In the Body tab, I chose the raw method and indicated that I used XML.

Here are a few error messages I got and how I managed to handle them.

<faultstring>The endpoint reference (EPR) for the Operation not found is http://opendata-tr.ratp.fr/wsiv/services/Wsiv and the WSA Action = null</faultstring>

Solution (from StackOverFlow): Add the header SOAPAction to the request. Adding a wrong action will actually improve the situation but the correct one needs to be determined by reading the .wsdl file.

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <soapenv:Fault>
            <faultcode>soapenv:Server</faultcode>
            <faultstring>unknown</faultstring>
            <detail />
        </soapenv:Fault>
    </soapenv:Body>
</soapenv:Envelope>

Solution: Did not find at the time. This one drove me crazy. The error is unknown. What I wanted was to call the action getLines without parameters (which is the easiest call possible), but that just did not work.
At that point, I eventually managed to display the list of all lines… when I stopped trying to display all lines and display only. My mistake was think my request was correct. It made me lose a lot of time to think that I understood something when I really didn’t.

<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <soapenv:Fault>
            <faultcode>soapenv:Server</faultcode>
            <faultstring>com.ctc.wstx.exc.WstxParsingException: Undeclared namespace prefix "wsiv"
 at [row,col {unknown-source}]: [5,14]</faultstring>
            <detail />
        </soapenv:Fault>
    </soapenv:Body>
</soapenv:Envelope>

Solution: It took me time to understand why and how I was supposed to declare a namespace. In RATP’s documentation examples, it happens that the first line is not totally shown and that is where the namespaces are actually defined. Drove me crazy as well, but forced me to learn about namespace. By trial-and-error, I eventually figured out that i was supposed to add the attribute xmlns:wsiv and xmlns:xsd.

In the end, it was both frustrating and fun to play around with this API. Frustrating because the documentation is very obscure for someone knows nothing about Web services, fun because I like challenges and there were a lot of “Tada” moments.

Here are some requests that probably made me “Tada”.

Jan 17: Getting the API to display lines

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <soapenv:Body>
 <wsiv:getLines>
 <xsd:id>RA</xsd:id>
 </wsiv:getLines>
 </soapenv:Body>
</soapenv:Envelope>

Sadly, it does not do as expected (that is, to display info on line RA). And no, it does not work, if I remove the “RA” line.

Jan 18: Getting the API to display lines with a request that actually makes sense.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://wsiv.ratp.fr/xsd" xmlns:ns1="http://wsiv.ratp.fr">
 <soapenv:Body>
 <wsiv:getLines>
 <xsd:blblblblbl>
 </xsd:blblblblbl>
 </wsiv:getLines>
 </soapenv:Body>
</soapenv:Envelope>

Okay, to be honest, there were initially not that many “bl”, but yes, that would work.

Jan 18: Getting the API to display info about a given line

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://wsiv.ratp.fr/xsd" xmlns:wsiv="http://wsiv.ratp.fr">
<soapenv:Header/>
 <soapenv:Body>
 <wsiv:getLines>
 <wsiv:line>
 <xsd:id>RB</xsd:id>
 </wsiv:line>
 </wsiv:getLines>
 </soapenv:Body>
</soapenv:Envelope>

Yes! I was finally able to give the name of a line, to finally give a variable to the API! The next steps was to fetch info about a given station, and that was easy enough.

The body of the request I really cared about was in the end:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://wsiv.ratp.fr/xsd" xmlns:wsiv="http://wsiv.ratp.fr">
<soapenv:Header/>
 <soapenv:Body>
 <wsiv:getMissionsNext>
 <wsiv:station>
 <xsd:line>
 <xsd:id>RB</xsd:id>
 </xsd:line>
 <xsd:name>Lozere</xsd:name>
 </wsiv:station>
 <wsiv:direction>
 <xsd:sens>R</xsd:sens>
 </wsiv:direction>
 </wsiv:getMissionsNext>
 </soapenv:Body>
</soapenv:Envelope>

With the header SOAPAction=urn:getMissionsNext.

Using CURL to make the request using PHP

Postman has this nice little feature that allows you to generate the curl code for PHP: simply click on the arrow next to the “Save” button.

But the party was far from being over.

Parse the XML

At that point, I thought I was almost done. It only remained to parse the XML. However, SimpleXML would not work out of the box and the XML Object kept being Empty.

Encouraged by this StackOverFlow‘s second answer, I tried to replace everything I had done with PHP5’s SoapClient.

Making the Request using SoapClient

It was even worse. I read the whole SoapClient documentation twice. I played with all kind of combinations of SoapParam and SoapVar. I tried to rewrite my own __doRequest() and __soapCall() methods. I read the C code of this function PHP5’s GitHub repository.

This is where what I call the hacking took place. I am not a hacker, but my friends who are, they tell me that it’s a lot a trial-and-error.

But in the end, nothing seemed to work. I was so frustrated. Good point is, in the process, I understood a lot more about how SOAP worked.

The final hack

What I ignored is the hack suggested by this StackOverFlow‘s first answer.
A one-liner that just does it. Such a beautiful hack.

 

UPDATE: By comparing the results, I figured out that CityMapper already used this API. Not sure when, though? My implementation can now be found on a GitHub repository and a live demo is available here.

One thought on “How I “hacked” into RATP’s API

Leave a Reply

Your email address will not be published. Required fields are marked *