collecting data from streaming APIs in twitter

21 thoughts on “collecting data from streaming APIs in twitter”

  1. nice , but i was looking for how to collect a group of followers , waiting for that post ,

    Please make it fast .

  2. short and sweet. just too good. can’t wait until they make the streaming live. thanks for the post.

  3. @Ishtiaque – the spritzer, follow and track APIs are open for all🙂 others are not available for everyone.

    The tremendous “Firehose” API is avilable to friendfeed fyi🙂 thats why they got all your tweets

  4. thanks for the info Hasin bhai. That’s really cool. One more question, aren’t they going to use the oauth. I thot twitter has set it as the standard way to access their APIs.

  5. @Ishtiaque yes of course they use oAuth. but i’ve shown the shortcut way by HTTP BASIC AUTH. Because this process will run in background as a shell process and you must use your account for that (which means u know un/pw) – and I dont know (and confused if there is even anything exist like it) how to use oAuth tokens for CLI🙂

    But yes, you can do it with oAuth token. Check my previous blog post to get idea how to implement oAuth using PHP in twitter🙂

  6. Awesome!

    Can you provide an example in case the data stream is interrupted and you have to re-connect?

    Also, how would you do the “track” stream using parameters? Does fopen allow for sending of the paramters?

    Or do I need to use cURL for “track”.

    Thanks

  7. This runs great, and really appreciate the post. Been having problems with it this week though, keeps dropping off, anybody have a solution to auto restart it when it does?

  8. To those asking how to make it re-start should the twitter API fail… use this…
    while(‘1’ == ‘1’) {
    $fp = fopen(“http://username:password@stream.twitter.com/spritzer.json”,”r”);
    while($data = fgets($fp)) {
    $time = date(“YmdH”);
    if ($newTime!=$time) {
    @fclose($fp2);
    $fp2 = fopen(“{$time}.txt”,”a”);
    }
    fputs($fp2,$data);
    $newTime = $time;
    }
    sleep(1000);
    }

    It is a little hackish… but its simple, and it works.
    Yes the sleep is needed or else you could get a temp IP block from twitter…

  9. Thanks for posting this… to give some back — those wondering how to use track and follow, see below:

    $curl = curl_init();

    curl_setopt($curl, CURLOPT_POST, true);
    curl_setopt($curl, CURLOPT_POSTFIELDS, ‘track=#NowPlaying’);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_URL, ‘http://stream.twitter.com/1/statuses/filter.json’);
    curl_setopt($curl, CURLOPT_USERPWD, $_CONFIG[‘twitter’][‘username’] . ‘:’ . $_CONFIG[‘twitter’][‘password’]);
    curl_setopt($curl, CURLOPT_WRITEFUNCTION, ‘progress’);
    curl_exec($curl);
    curl_close($curl);

    function progress($curl, $str)
    {
    print “$str\n\n”;
    return strlen($str);
    }

  10. Anybody who can tell me how to adapt the above code of datacollector.php to work with the new twitter api. Please help sorry if the question is simple but am very new to that. Thanks in advance
    Nagy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s