Author Archives: Yuji Tomita

I'm a human person. I excel in crafting holistic technology solutions that extend beyond a narrow technical scope.

Programmatically log into a website with Python and urllib, urllib2. This one for my django.

"""
Log into Django Site
====================


By Yuji Tomita
2/22/2011
"""

import urllib
import urllib2
import contextlib


def login(login_url, username, password):
    cookies = urllib2.HTTPCookieProcessor()
    opener = urllib2.build_opener(cookies)
    urllib2.install_opener(opener)
    
    opener.open(login_url)
    
    try:
        token = [x.value for x in cookies.cookiejar if x.name == 'csrftoken'][0]
    except IndexError:
        return False, "no csrftoken"
    
    params = dict(username=username, password=password, \
        this_is_the_login_form=True,
        csrfmiddlewaretoken=token,
         )
    encoded_params = urllib.urlencode(params)
    
    with contextlib.closing(opener.open(login_url, encoded_params)) as f:
        html = f.read()
        
        print html
    # we're in.

cProfile Example and iPython Magic Shortcut %prun

Running cProfile is a bit of a pain.

You have to pass your function and all its scope as a string to cProfile, which is.. just a pain. 

cProfile.run('''
def test(n):
    x = 1
    for i in range(n):
        x = x * 2
''')

iPython has a magic function that lets you use the current scope. 

def test(n):
    x = 1
    for i in range(n):
        x = x * 2
 %prun test(100000)

         4 function calls in 0.753 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.752    0.752    0.753    0.753 :1(test)
        1    0.002    0.002    0.002    0.002 {range}
        1    0.000    0.000    0.753    0.753 :1()
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Jquery — Fire event when all images have loaded within element

I needed a smooth fadeIn effect after all images had been loaded in a specific element.

I solved the problem by incrementing a load counter and executing a function when the counter reaches the number of elements to load.

load_counter = 0;
$.each(my_images, function(i, item) {
    $(item).load(function() {
        load_counter ++;
        if (load_counter == my_images.length) {
            // all items have loaded
            $("#my_element").fadeIn();
        }
    })
})

Data Modeling — Free OSX Entity Relationship Diagram Software

I searched for mac ERD software for a while, and couldn’t find anything but crap, crap ports, over-built (for my needs), or $100-$500 a license software.

Of course there was a ton of windows apps, but not for OSX.

I had to turn to stackoverflow for help on this one and thanks to sfinnie from Scotland found an obscure, fantastic piece of software called yEd Graph Editor.

http://www.yworks.com/en/products_yed_about.html

Here’s the current state of the new Grove software that will also share a common ground with upcoming Anuva software.

http://stackoverflow.com/questions/4927511/free-or-inexpensive-simple-table-class-visualization-software-for-osx-er-dia

This gets better and better!

Python — StackOverflow Post Checker / Updater

I’m a little obsessed with constantly refreshing stackoverflow.

I can’t find an API for it (I’ve since found the Stack Exchange API, oops), so I wrote a little python script to parse it for new questions as an exercise.

I built a fetcher class that builds an SQLite database (if one doesn’t exist), parses stackoverflow for questions relating to django, and determines if any are new (based on the URL being stored in the database) and does a Growl notification if found.

It’s extremely straight forward, but this shows you the power of python. From idea to implementation in no time at all, including figuring out how Growl’s python bindings work.

Simply amazing!


"""
"""
Stack Overflow Post Checker
===========================

Parse stackoverflow HTML for questions, store in sqlite database
and send notifications of new questions.

By Yuji Tomita
2/7/2011
"""
import os
import sqlite3
import urllib2
import BeautifulSoup
import Growl


class StackOverflowFetcher:
    def __init__(self):
        self.base_url = 'http://stackoverflow.com/questions/tagged/'
        self.get_or_create_database()
        
        self.growl = Growl.GrowlNotifier(applicationName='StackOverflowChecker', notifications=['new'])
        self.growl.register()
        
        self.tags = [('django', True), ('python', False)]
        self.get_questions()
        self.close_connection()
        
    def get_questions(self):
        """
        Parse target URL for new questions.
        """
        while self.tags:
            tag, sticky = self.tags.pop()
            url = self.base_url + tag
            html = urllib2.urlopen(url).read()
            soup = BeautifulSoup.BeautifulSoup(html)
        
            questions = soup.findAll('h3')
        
            for question in questions:
                element = question.find('a')
                link = element.get('href')
                question = element.text
            
                if self.is_new_link(link):
                    self.growl.notify(noteType='new', title='[%s] StackOverflow Post' % tag, description=question, sticky=sticky)
                    self.record_question(link, question)
                    
                    
    def get_or_create_database(self):
        """
        Check if database file exists. Create if not.
        Open file and send query. 
        If query fails, create tables. 
        """
        path = os.path.join(os.path.dirname(__file__), 'questions.db')
        
        try:
            f = open(path)
        except IOError:
            f = open(path, 'w')
        f.close()
        
        self.conn = sqlite3.connect(path)
        
        try:
            self.conn.execute('SELECT * from questions')
        except sqlite3.OperationalError:
            self.create_database()
            
            
    def create_database(self):
        self.conn.execute('CREATE TABLE questions(link VARCHAR(400), text VARCHAR(300));')
        
        
    def is_new_link(self, link):
        results = self.conn.execute('SELECT * FROM questions WHERE questions.link = "%s";' % link).fetchall()
        if not results:
            return True
        return False
    
    
    def record_question(self, link, question):
        results = self.conn.execute('INSERT INTO questions(link, text) VALUES ("%s", "%s");' % (link, question))
    
    
    def close_connection(self):
        self.conn.commit()
        self.conn.close()


stack = StackOverflowFetcher()

This code is now on GitHub where it will be more updated.
https://github.com/yuchant/StackOverflowNewPostCrawler

iPhone/Objc — Programmatically Create Scrollview Without Interface Builder

I’m new to Objective C in general, and am having trouble finding useful docs with step by step examples like the django project (best docs in the world!).

I could find endless resources on building UIScrollViews through the Interface Builder, but not programmatically.

My biggest hurdle was: How do I subclass a UISCrollView method call (like viewForZoomingInScrollView) if I’m not using a UIScrollView class?

You simply need to set the delegate for the scrollView to point to whichever class HAS those methods.

Example: 

// in my .m file
-(void)loadView {
    CGRect fullScreenRect = [[UIScreen mainScreen] applicationFrame];
    scrollView = [[UIScrollView alloc] initWithFrame:fullScreenRect];
    scrollView.delegate = self;
    self.view = scrollView;
}
// now my scrollView is loaded and ready to be populated with a scrollable subframe in viewDidLoad

// because the delegate has been set to 'self', these methods will be called despite this class not being a UIScrollView subclass.
-(UIView*)viewForZoomingInScrollView:(UIScrollView*) scroll  {
    return imageView;
}