Backing Up SegmentFault Articles with Python, Continued

In the previous article, I simulated the request with requests but did not obtain the cookies. While reading an HTTP article today, I noticed that some response headers contain a set-cookie field. The earlier attempt had clearly failed because it never received the response containing that field.

Manual testing showed that sf_remember was set when a successful login redirected to the home page, not when the POST itself was sent. To reproduce the requests more accurately, I switched to requests.Session, which synchronizes cookies automatically and behaves more like a browser.

def _get_user_cookies(self):
        s = requests.Session()
        s.headers.update(headers)
        rep = s.get(target_url)
        post_url = "%s%s?_=%s" % (target_url, login_api_path,
                                  self.get_req_from_html(rep.text))
        data = {
            'mail': self.username,
            'password': self.passwd,
        }
        s.post(post_url, data=data)

Repeated login attempts showed that the request had to include headers, and the only required header was Referer. With that header, the POST succeeded and returned a 302 redirect to the home page. Following that redirect completed the login. The resulting cookies looked like this:

<RequestsCookieJar[]>

There was no sf_remember field, probably because the POST omitted the remember option. My earlier assumption that this field marked a logged-in session was therefore unreliable. The actual marker was probably just PHPSESSID. The following screenshot shows all cookies after login.

This means that logging in and backing up articles does not require phantomjs; requests alone is enough for SegmentFault. A dynamic site such as Weibo may still require PhantomJS, unless you can analyze and reproduce the entire request flow. That lets you use the simplest possible tool to do what you need.

The code is here. It now prints article information and elapsed time, as shown below.

Summary

When a login link—or any other POST action—redirects directly, regardless of how many redirects are involved, requests can simulate it. Its Session object follows the redirects and carries the session state.

When redirects are controlled by JavaScript in the page, a browser emulator such as PhantomJS makes it easier to complete the whole login flow, though it is slower because browser simulation is more expensive. In theory, requests can reproduce this kind of flow as well, but you must first analyze the JavaScript responsible for the redirects. That is more work and may not succeed; when it does, it can be much faster. For example, a NetEase Cloud Music artist page requests two documents, but only the second is actually needed.

Originally published on SegmentFault.